Ctrl + Create: Empowering Creative Control in AI-Driven Rapid Level Design

The problem

Early-stage level design is stuck between two costs: a whitebox layout is cheap to change but says nothing about art direction, while a proper paintover is expensive, so it arrives late, often right before the level gets reworked anyway.

The approach

Ctrl + Create keeps the level's geometry as the source of truth and lets a diffusion model paint it. Every object in a Unity whitebox scene carries its own text prompt, plus one shared global style for everything else. A Unity URP plugin extracts segmentation, depth and normal maps per object straight from the whitebox and sends them to a local Node.js + ComfyUI backend running SDXL with ControlNet: segmentation says where, depth says how far, normal says which way. Nothing leaves the machine.

The study

Twenty game and level design students each built the same scene under three conditions: pencil-and-paper drawing, bare Stable Diffusion prompting, and the tool, measuring task efficiency, perceived control, expressive range and open-ended requirements.

What we found

Revising was significantly faster with the pipeline than either baseline. Perceived control matched free drawing and stayed nowhere near the collapse seen under bare prompting. Expression scored highest and most consistently of the three, and unlike bare Stable Diffusion, it didn't track how experienced someone already was with generative AI. Sixteen of twenty participants reported improved spatial control; most of what was still missing was ordinary editor tooling like undo and copy-paste, not the pipeline itself.

Presented at FDG 2026 in Copenhagen, with Fabian Akker (Breda University of Applied Sciences), Angelos Chatzimparmpas and Johannes Pfau (Utrecht University).