[Godot] Add visual verification for LLM code generation

I built an autonomous pipeline that generates playable Godot games from a text prompt. The two problems worth discussing here: how to make an LLM write correct code in a language underrepresented in its training data, and how to verify correctness beyond compilation. This isn't a paper — the code is open-source and the results are reproducible, which I think is more useful for this kind of work. **One-shot coding from context, not training data:** GDScript is Godot's scripting language — \~850 classes, Python-like syntax, but not Python. LLMs have relatively little GDScript in their training data — enough to get the syntax roughly right, not enough to reliably use the engine's 850-class API. Without reference material in context, you get hallucinated methods and invented patterns. Provide the reference material, and the question shifts: can the model actually use it properly? That makes it a real benchmark for how well LLMs use supplied documentation vs. falling back on training priors. The reference system has three layers: * A hand-written language spec — not a tutorial, but a precise reference covering where GDScript diverges from what the model expects (type inference failing on `instantiate()` because it returns Variant, polymorphic builtins needing explicit typing, lambda capture semantics that differ from Python) * Full API docs for all 850+ engine classes, converted from Godot's XML source to compact Markdown * An engine quirks database — behaviors that are hard to discover from docs alone (`MultiMeshInstance3D` silently losing mesh references after serialization, `_ready()` not firing during headless scene building, collision state mutations inside callbacks being silently dropped) **Agentic lazy-loading — the context management problem:** You can't load 850 class docs at once — it would consume the entire context window. But if the agent picks the wrong subset, it writes code against APIs it can't see. The outcome is directly tied to the agent's ability to choose its own context: load too much and you drown reasoning in documentation, load too little and you miss the class you need. The solution is two-tier lazy lookup. A small index (\~128 common classes, one line each) is always loaded. A second index covers the remaining \~730. The agent checks the index, then loads full docs for only the specific class it needs at that moment. Each task runs in a forked context (fresh window, no accumulated state), so context management decisions reset per task rather than degrading over time. This is where the system succeeds or fails — not at code generation, but at context selection. **Three stages of verification:** 1. **Compilation** — Godot headless mode catches syntax errors, type mismatches, missing references. This is the easy filter. 2. **Agentic screenshot verification** — the coding agent (Claude Code) captures screenshots from the running scene and does basic self-assessment: does the scene render, are the expected elements present, is anything obviously broken. This is cheap and catches gross failures. 3. **Dedicated visual quality assurance agent** — a separate Gemini Flash agent receives the screenshots plus a reference image and runs structured verification against task-specific criteria. Operates in static mode (single frame for terrain/UI) or dynamic mode (2 FPS sequence for physics/animation — evaluating temporal consistency, not just a single frame). This catches what the coding agent can't objectively judge about its own output: z-fighting, floating objects, physics explosions, grid-like placement that should be organic, uniform scaling where variation was specified. The separation matters. The coding agent is biased toward its own output. A separate vision agent with no access to the code — only the rendered result — provides independent verification. **What this achieves:** To be clear about the contribution: before these pieces were in place, the pipeline produced games that were consistently unplayable — broken collisions, physics explosions, missing interactions, visual artifacts. Often the agent would find ways to bypass verification entirely, producing garbage output that technically passed checks. Each component described above was necessary to cross that threshold. This isn't an incremental improvement over a working baseline; the baseline didn't work. The contribution is the combination that makes it work at all. **Architecture:** The pipeline decomposes game development into stages (visual target → decomposition → architecture → asset generation → task execution with verification). Stages communicate through structured documents, not conversation. Each task forks a fresh context. The generated GDScript is split into scene builders (headless programs that serialize `.tscn` files) and runtime scripts (game logic), with strict separation of which APIs are available at which phase. Output is a complete Godot 4 project — scenes, scripts, generated 2D/3D assets. This post focuses on the technical findings, but the full story — including a year of wrong turns, four major architecture rewrites, and all the things that didn't work — is coming as a detailed blog post. If you're interested in the "how we got here" rather than just the "what works," keep an eye out for that. Four demos showing prompt → playable game: [https://youtu.be/4\_2Pl07Z7Ac](https://youtu.be/4_2Pl07Z7Ac) The code is on GitHub [https://github.com/htdt/godogen](https://github.com/htdt/godogen) . I'm also on Twitter/X [https://x.com/alex\_erm](https://x.com/alex_erm) where I'll share the blog post when it's out. Happy to answer questions here.