The last essays in this series were about agents rebuilding the renderer. This one is about agents using it. For the past few weeks I’ve been running an experiment inside RISE: agents that work on a scene alongside you. They read the scene file, call tools to add and edit objects, render the result, and iterate.
Early on I noticed a gap. When I wrote a careful prompt that spelled out what I wanted, a memorial courtyard with a weathered copper sphere on a marble plinth, oiled-oak decking, and a reflecting pool, the agent produced six distinct kinds of procedural texture. When I typed what a real user types, “Build me a courtyard,” I got flat gray boxes. Same model, same tools, same renderer. The creativity was in my prompt, not in the agent.
So the question became: can you make an agent creative by default, without asking the user to write a paragraph? What follows is what happened when I tried every obvious answer and measured each one. The short version is in the title. The longer version is a ladder, and each rung taught me something I now believe applies well beyond rendering.
First, build the ruler
A word about the big word in the title before I lean on it. I’m using creativity operationally here: not artistic merit, but whether the agent escapes low-variety defaults and makes nontrivial material and geometry choices the user never specified. That’s what the eval measures, and it’s all this essay claims to measure.
So, before changing any behavior I built the measurement: an eval that runs the bare courtyard prompt against a fresh scene, checks the result for texture variety, geometry variety, and a few honesty guards, and records the agent’s full trajectory. Every phase after this changed exactly one variable and re-ran the eval. This sounds like ceremony. It saved the project in the first week.
Because the baseline embarrassed me. The problem I had set out to fix, agents defaulting to flat colors, was already half solved by some earlier documentation work I hadn’t credited. Median runs were producing two or three texture kinds on their own. The real deficits were narrower and more interesting: in nine baseline runs, not one ever made a surface property like roughness vary across an object (everything came out polished-plastic uniform), and only two ever reached for the advanced modeling tools, the difference between a sculpted fountain and a stack of cylinders. I had almost spent the project fixing yesterday’s problem. If you take one process note from this essay, take that one: let the instrument falsify the premise before you act on it.
One more piece of the ruler mattered. There’s an obvious cheat for “be more creative”: have the agent interrogate the user about style before it builds anything. I’d decided early that a wrong rich guess is a better product than a questionnaire, and the models turned out to agree with the guess part: in 132 earlier runs without an ask-the-user tool, the agent spontaneously asked zero questions. But I didn’t want any creativity gain to quietly push in the other direction either. So a second eval rode along in every batch as a counterweight: one scenario where asking is genuinely the right move and is required to pass. Any change that bought creativity by making the agent chatty would show up as a red mark, not as a vibe. It never fired. Every gain below landed with zero new questions asked.
On rigor, so it’s stated rather than implied: each phase was three runs per model, two models (one hosted, one local, the local chosen because its reasoning gets recorded), one variable at a time, with kill conditions written down before each run so that stopping was a lookup rather than a debate. The eval can be gamed and I know exactly how; it’s a measurement, not a leaderboard, and the known holes are documented next to it. Small numbers, honestly handled, replicated across phases. I’ll take that over a big benchmark I can’t inspect.
Rung one: tell it
Prior work had already measured plain instructions, the “be creative, use rich materials” kind of guidance you’d put in a system prompt. Adherence ran around fifty percent. A coin flip is not a mechanism. Moving on.
Rung two: advise it
Next I built an advisory. After each render, the engine inspected the scene, and if every surface was uniform it said so in the result the agent reads: here’s what’s flat, here’s the feature that fixes it, here’s the doc to read, and an escape clause (“intentional flat styling: ignore”). Truthful, specific, perfectly timed, delivered in-band.
It fired in every run. I verified delivery in the trajectories, because “the agent ignored the advice” and “the advice never arrived” are different findings and most write-ups don’t distinguish them. The agent acted on it zero times.
Maybe the placement was wrong. I moved the same facts into the validation diagnostics list, the thing agents actually check before declaring victory. Zero again.
Here the trajectory gives me something better than behavioral inference, though not ground truth about the model’s internals: the rationale it wrote down as it worked. The local model’s reasoning is recorded, and it dismissed the advisory in writing:
“The full candidate validated clean – no errors. The ‘note’ about scalar_painter is just a design suggestion, not an error.”
And two runs later, my favorite sentence of the whole project:
“…no errors or warnings beyond the expected design note about scalar textures (which is a feature suggestion, not an issue).”
The expected note. By its sixteenth firing the advice had become wallpaper. One run saw it thirty times. Every engineer reading this has skimmed a linter warning in exactly this frame of mind, which is why I trust the finding: it’s not a quirk, it’s triage. In this harness, both models behaved the same way in every run: fix what blocks, skim what suggests.
Final tally for the advisory, across two models, two phrasings, two delivery channels, and up to thirty exposures in a single run: zero for twenty-four. Not low. Zero.
Rung three: show it
Examples did move something, and the shape of what they moved is the most transferable finding here.
I rewrote the code snippets in the agent-facing docs, replacing flat colors with varied procedural textures, a different style in each doc so there was no single example to converge on. Texture variety moved: the floor went from two texture kinds to three. Models copy the example in front of them. So far, so good.
Then I ran the same treatment on the two measured deficits. Examples showing roughness bound to a texture, placed in the exact snippets the agents demonstrably read: zero for twelve. Drop-in recipes for the advanced modeling tools, eleven lines, no assembly required: zero for twelve.
The pattern across all three results, and it took three separate failures to see it cleanly: in these runs, examples moved behavior when they were drop-in edits to something the model already writes. Swapping a flat color for a noise texture in a slot it fills anyway, that copied. Anything that required the model to compose, to restructure its approach, to combine two things it hadn’t combined before, didn’t copy, no matter how prominent the example. That’s a mechanism worth testing beyond my renderer, not a law. But if you’ve ever wondered why your style guide changes some habits and not others, I suspect this is the line it’s splitting on.
Rung four: change what things cost
If composition is the barrier, remove the composition. My first design was the lazy version: preset rich materials, one call, done. I killed it before building it, for a reason worth stating because it’s a product argument, not a technical one. A preset’s ceiling is the preset library. Give agents five gorgeous canned materials and every scene converges on the same five. You haven’t bought creativity, you’ve bought nicer sameness.
What shipped instead was a scaffold: a tool that stamps a small working example directly into the agent’s own scene file, built from ordinary, editable parts, the same parts the agent uses for everything else. Three properties carried the design. The tool has required creative parameters with no defaults, so every call forces a few real decisions (tone, wear, scale). Each expansion is seeded with slight variation, so two scenes never start identical. And because the result is just ordinary scene content, the ceiling isn’t the template library; it’s everything the engine can do, one edit away. That last property is the real distinction, sharper than preset-versus-scaffold: an opaque finished asset versus an editable starting point. The trick is lowering the cost of composing without closing off the representation.
It was the first intervention in forty-two runs to move either of the hard deficits. And the detail that tells me the design was right, not just the mechanism: the agents edited what they got. Provenance naming let the eval separate authored from expanded from expanded-then-edited, so this is traced, not vibes. One run called the geometry scaffold twice for a set of columns, compared the two, and rebound its objects to the version it preferred. Another expanded a vessel, threw the shape away, hand-built a replacement, and kept the material a second scaffold call had given it, the same stone as the walls. Drafts, not assets. The capability was there the whole time. Composition cost was standing in front of it.
[Image pair or triptych: the vessel run. The scaffold’s expansion as inserted (reconstructable from the tool-call payload in that trajectory), then the final scene where the agent’s hand-built replacement wears the scaffolded material. This is the single best illustration in the essay if the reconstruction renders cleanly.]
[Optional image: a courtyard from the geometry-scaffold group, the first eval group to pass every check. The final scene is in that run’s trajectory.]
The mechanism has a measured boundary, and both models found the same one from different directions: coverage. The local model declined the tool with a stated reason (“it only handles five specific material types, whereas my scene needs a broader palette”), the first explicit engagement with any of my interventions in over sixty runs. The hosted model used scaffolds enthusiastically on a courtyard and not at all on a cozy study, because a lamp shade maps to none of the scaffold’s families. A scaffold only lowers the cost of what it covers.
What’s still broken
I want to be careful with the claim here. The scenes are better. They are not rich. A person with taste and an hour will still beat every one of these runs without trying; what’s gone is the embarrassing floor, the flat gray default, and what’s supported is the mechanism that removed it. The distance from here to “rich” is long, and I think I now know which kind of lever shortens it and which kind doesn’t.
The clearest remaining gap isn’t in building at all. It’s in looking. One run rendered a reef with half its corals plainly buried in the seabed, rendered it again at higher resolution, and declared victory. The agents don’t step back, notice that something is visually wrong, and revise. A judging gap, not a building gap.
I know the loop is possible, because one stronger model, given room to work, closed it unprompted: rendered its own full frame, zoomed in, decided the arm and hair weren’t good enough, edited the geometry, and iterated three times before the final render. That’s the behavior everything above is reaching toward.
And the next lever on my list is, of course, not a suggestion. It’s a check that turns “this part isn’t visible in the frame” into a fact the agent has to deal with. That’s the untested experiment, not a result. But in every trajectory I’ve read, hard errors got fixed and suggestions got skimmed, so I know which currency I’d rather pay in.
Harness or model?
Which raises the question I keep turning over: everything above is harness work. I never touched a weight. How far can that go, and where does the model itself become the only remaining lever?
The field’s answer, right now, is that the harness goes surprisingly far. Epoch AI surveyed enhancements applied after training, scaffolding included, and estimated compute-equivalent gains of roughly 5x to 30x on the relevant benchmarks, at a small fraction of the cost of training. The 2026 version of that story is more concrete: LangChain took a fixed frontier coding model from 52.8% to 66.5% on Terminal-Bench through harness changes alone, same weights underneath. My little ladder is a domain-sized replica of that pattern. The lever that worked here, the scaffold, is pure harness; so is the blocking-fact bet I’m making next.
But my own data also shows the two edges of the harness. The first is coverage: a scaffold only cheapens what its author thought to include, and real scenes keep producing objects nobody put on the list, like that lamp shade. However long I make the list, I’m always one scene behind. The second is the prior itself. The reason advice scored zero isn’t a harness defect; it’s that these models were trained, by their makers, to triage. Fix what blocks, skim what suggests. That prior lives in the weights, and no tool result I write can reach it. The harness can only keep routing around it, converting one suggestion after another into a blocking fact, forever.
So what would the model lever buy? I actually have half of it on the table: one of my two eval models is open-weights and runs locally, so fine-tuning it is a real option rather than a thought experiment. The tempting move is obvious. Take the trajectories where richness happened, tune on them, and move the default instead of taxing it. Two things stop me. The first is that the natural training signal is my own eval, and I know precisely how my eval can be gamed; an optimizer would find the holes faster than any agent has. The metric I trust as a measurement would be ruined as a target. The second is quieter: tuning toward my scaffold families risks compiling that preset library into the weights, where I can’t inspect it and the agent can’t edit it. It might generalize instead; I can’t know without running it. But I refused that monoculture as a tool, and I’m not eager to gamble on rebuilding it as a bias.
Where I land, tentatively: the harness is best at changing facts and action costs; the model supplies most of the prior and the judgment. The flat-gray floor was a cost problem, and the harness solved it. “Rich” looks increasingly like a judgment problem, and my results make me suspect that lever sits at the model level. It’s at least suggestive that the strongest model in my logs needed none of my machinery to step back and judge its own render, though that run differed in more than weights, so I hold the point loosely. Some model teams are now explicitly bringing the harness into post-training, so scaffold-shaped behavior becomes signal instead of wrapper; that’s the version of the model lever I’d actually want. For a solo practitioner the honest version of that is humbler: build the harness so each piece can be ripped out the day a model shows up that doesn’t need it, and treat every rung of this ladder as a measurement of what today’s models are, not a monument. The scaffolding is load-bearing now. The goal is for it not to be.
You can’t ask for creativity. You can’t advise it into existence, and you can only show it to someone who already almost does it. What you can do is find the composition cost standing between the agent and the interesting choice, and remove it. Then get out of the way, and watch it throw away your scaffold and keep the material.
Aravind 