Modding Age of Empires II with a Sprite-Diffuser – ENGINE
Final month I got down to create a approach for my mates to make customized civilization sprites for our Age of Empires II lobbies.
Beneath are some ideas and course of on find out how to create a flexible prompt-based picture generator. For rookies I’d suggest Alpaca, and for these snug with coding – Stable Diffusion Web UI and Python.
Particular because of the AoEII modding communities OpenAge, SLX Studio, and Age of Kings Heaven.
Visible Explorations
Screenshot Transfers
This concept took place whereas taking part in with picture switch (img2img) on Age of Empires II screenshots. All of the methods described under use the Steady Diffusion generative “AI” (which I additionally seek advice from because the mannequin or community).
I began in Photoshop with Alpaca and labored on full sport screenshots. Largely what I used to be searching for was how the mannequin behaved, what shapes and textures it selected to protect and what it discarded in its stylistic wanderings. Total it managed to remain in perspective and theme fairly properly.
What was instantly fascinating was how components of the picture began working collectively. Tiled terrain gave strategy to roads and paths – there’s communication between the buildings and setting. For now we are going to solely be working with constructing sprites – however it demonstrates that machine studying may very well be an excellent software for mixing procedural components of a sport.
The Caspar David Friedrich Benchmark
Following this I prompted the community to output photographs right into a strongly isometric perspective, trying to get constant outcomes throughout lighting, shade, form, and texture.
Some actually superior outcomes, because the energy of the picture switch will increase, components of the minimap and UI would develop into different buildings or terrain.
Sprite-Diffuser
Anime Loyalists vs. Moon Colonists vs. Zombie Romans
Now it was time to make some civilizations.
When producing the mannequin requires 512×512 photographs, fortunately all of the Age of Empires II sprites are about half that so we don’t have to fret about resizing. These have been the primary outcomes.
These buildings have been cherrypicked and cleaned in Photoshop and weren’t batched. So as to get numerous outcomes the energy of the picture switch was set very excessive, however because of this the lighting, proportion, and stage of element various enormously. The citadel asset (left) would proceed to be a ache level as it’s 2x bigger most buildings, however the community would insist it was an outsized home.
Refining Outputs with Management Internet and Loopbacks
To get constant outcomes whereas having a excessive switch energy I wanted to make use of Management Internet, a system that guides technology utilizing enter like lineart, depth, or segmentation.
At this level every little thing labored, sprites stayed in perspective and listened to the immediate, however they didn’t be imaginative or sudden. This may’t be solved by making the immediate energy extraordinarily excessive, the lighting or shade will begin to differ an excessive amount of, management internet can not assist there.
As an alternative, there’s a method known as loopback – this runs the generated picture again over itself with the identical immediate. With out it, the community isn’t capable of think about novel modifications to shapes, textures, and shade.
The ultimate method I settled on was Management Internet with Zoe Depth Estimation, two loopbacks, each with about 85% energy (solely 15% of the unique picture is preserved).
Prompting
That is arguably an important a part of the method, however I’ve left it for final. That’s as a result of copying what labored for me will in all probability maintain you again, I didn’t spend that a lot time on prompting! However this can be a good place to start out.
I used easy method for every picture with the next prompts. The one area that modified for every constructing was the Topic (i.e. Archery Vary, Market, Home).
- Form
- Descriptor
- Topic
- Fashion
- Emphasis
- Modifiers
- Isometric exterior of a
- historic Roman
- Barracks
- within the type of Giovanni Paolo Panini
- 3D roman structure, greco-roman stone and pillars with intricate stonework and roofs
- desaturated, 8k, vibrant sunny pure lighting, trending on artstation
Lastly, Steady Diffusion doesn’t create transparencies. I believed object detection would work however surprisingly they struggled to discover a stable masks. As an alternative I compelled a stable background shade within the immediate and created a couple of clear flood fills with ImageMagick
Remaining End result
Different Experiments and Ideas
Customized Buildings & Superb Tuning
It additionally potential to reverse the method. As an alternative of making completely different kinds of present sprites, new sprites within the unique type may very well be made utilizing a fine-tuned Age of Empires II mannequin. Both 3D blockouts or photograph photographs of actual buildings may work as inputs.
Greyboxing to Picture
With certainly one of my present Unity initiatives, I took a greyboxed stage and utilized segmentation to the buildings, floor, and bushes based on the ADE20k dataset.
With Management Internet and object segmentation we will run it by means of the identical Caspar David Friedrich immediate. This may very well be utilized in idea artwork or stage design workflows to shortly block-out setting artwork.
Picture Wealthy Mindset, Seeding, Spawning
With a lot semantic and syntactic info of a picture now out there, every picture within the mission generally is a place to begin for synthesizing new outputs.
By defining 10-20 hero belongings utilizing a standard artwork workflows, be it characters, buildings, or environments, a improvement staff may then package bash new belongings collectively. Studios with a big inner catalogue of idea artwork and belongings could be interested by bringing this to followers to increase their lore and worlds.
Conclusion
These fashions are surprisingly versatile and are a variety of enjoyable to work with. Future video games, ought to they select to, may create a set of base belongings that then seed user-generated lore or inner improvement. The sport world is itself a picture mannequin, video video games are each a spot to be and a fascinating body for future photographs.
The mission is on-line at @neilsonks and engine.study/sprite-diffuser/.