In June 2026, an ambitious design document outlined a locally hosted multimodal agent called the Autonomous Semantic Vector Graphics Engine. The premise envisioned an artificial intelligence acting as a system architect, capable of decomposing a text prompt into named components, rendering them through a continuous see-think-act loop, and returning clean, semantic Scalable Vector Graphics (SVG). However, early peer reviews quickly punctured the illusion, describing the proposal as a well-organized brainstorm rather than a buildable technical specification.
Four months later, a working engine has emerged, though it bears little resemblance to the original conceptual draft. It has evolved into a constraint-solving vector compiler featuring a text front door used primarily as a smoke test and a sketch front door trusted for real-world input. This is the account of that architectural turnaround, detailing the initial miscalculations, the harsh critiques that forced a pivot, and the structural realities of combining artificial intelligence with deterministic geometry.
The Original Pitch and Its Flaws
The initial design document opened by identifying three core problems plaguing generative vector art. The first was context window bloat, where sending a base64-encoded image to the language model after every single line drawn rapidly consumed thousands of visual tokens, leading to out-of-memory errors or the system forgetting its original instructions. The second issue involved mathematical and spatial hallucinations, as large language models chronically struggle with absolute coordinate math, such as calculating the exact intersection of two angled lines, and complex SVG path commands like Bezier curves. The third challenge was the "spaghetti" vectorization problem, where traditional tracing algorithms output thousands of overlapping, unnamed path nodes.
The proposed solution relied on a closed-loop spatial reasoning engine where an LLM acted as an architect, using semantic abstractions like named components and scene graphs to draw logical, mathematically precise assets. The design incorporated decoupled dual-buffer rendering, deep cloning over internal definitions, and a feedback loop where the agent relied on XML snippets and scene graph summaries for micro-corrections. Additionally, a tracing mode dubbed "agentic onion skinning" rendered a target logo at partial opacity for a vision-language model to trace over and critique.
The turning point came when the design was submitted to six different AI models for review. While models like Claude noted that a development team picking up the spec would hit blockers immediately, and Kimi pointed out that the concept conflated semantic understanding with geometric execution, a critique from Minimax laid bare the fundamental flaw of the architecture. The reviewer noted that the design described a closed-loop spatial reasoning engine without defining what the agent was converging toward. For text-to-SVG generation, the agent’s observation step merely confirmed its own mental model with no reference signal to correct against, meaning the loop was closed on a moving target. The critique established that a visual critique loop is only meaningful when working with image-to-SVG workflows where a concrete reference image exists.
The Architectural Pivot
The harsh feedback forced a fundamental division of labor, leading to a pivotal update in the project’s specification. The core realization reframed the system from an LLM drawing vectors to an architecture where the language model plans semantically and a deterministic engine executes geometrically.
This led to the establishment of the zero-coordinate kernel, ensuring that the model never directly places any elements. Instead, it declares sizes and proposes structure while the underlying deterministic engine handles all coordinate calculations and layout mathematics. However, testing this hypothesis revealed further structural hurdles. When running a live test to generate a bicycle, the output remained completely unrecognizable, proving the failure was structural rather than coordinate-related. The architecture lacked a way to express fundamental physical relationships, such as the frame’s rear axle functioning as the center of the wheel. Consequently, inter-component connection was recognized as the actual end goal rather than an optional feature.
Further policy refinements established every tool call as a strict transaction, preventing the language model from attempting direct geometric manipulation. By late August, the project’s product direction was officially updated. DraftsLLMan was redefined as a semantic constraint-solving vector engine designed for icon and technical asset production rather than an autonomous picture generator. Elaborate one-shot art generation was dropped entirely, relegating complex test objects like the bicycle to mere kernel smoke tests. Local language models running on modest hardware were also reconsidered, as heavy reasoning chains that burned thousands of tokens only to emit skeletal structures on later attempts proved to be an unproductive computational tax.
The Shift to Sketch-Based Inputs
The pivot toward accepting sketches as primary inputs resolved the missing components of the text-based workflow by supplying reliable ground truth. Unlike text prompts, which give a feedback loop nothing concrete to compare against, a raster sketch provides an observable baseline where discrepancies become measurable facts.
The system’s three-stage sketch pipeline handles this through stroke extraction, geometric primitive fitting, and semantic grouping. Raw, wobbly contours are processed by a math layer that fits lines, circles, arcs, and corners, populating the scene graph with mathematically perfect nodes while stripping away hand-drawn jitter. Meanwhile, the language model acts purely as an art director, handling semantic naming without performing any arithmetic or tracing operations.
Verifying the efficacy of this approach required moving away from subjective visual assessments toward a rigorous evaluation harness. The system generates a seeded wireframe, rasterizes it into a hand-drawn style using procedural tools or local pipelines, and hands the result to a vision agent to score the recovered scene graph. Scoring relies on semantic alignment where components are counted as successfully recovered when their roles and kinds agree, using bounding boxes only as tie-breakers. A separate relation consistency check verifies spatial relationships like left-of and above, ensuring that layout errors register as genuine failures rather than minor coordinate wobbles.
Current development stands with a verified codebase comprising hundreds of offline tests written in Kotlin alongside test suites running headless rendering via Skiko. While geometric constraint solving powered by the Cassowary algorithm has proven robust and reliable, challenges remain in managing complex component hierarchies and deep model context limits during extensive generation passes.
Ultimately, the journey from an unbuildable conceptual brainstorm to a functioning constraint-solving vector engine highlighted a crucial lesson in AI system design. Building a system that accepts a structured drawing as input and solves underlying geometric constraints provided a viable, gradeable path forward that a purely autonomous text-to-image generator could never achieve.
Comments (0)