Automatically Creating Prompts! A Workflow Combining the Best of Animagine-XL and Illustrious-XL
- Caption with Florence-2
- Create compositions with Animagine-XL
- Polish with Illustrious-XL
Introduction
Hello, I'm Easygoing!
In this post, I’ll walk you through a workflow that automatically generates prompts from images, leveraging the strengths of both Animagine-XL and Illustrious-XL to create stunning illustrations.
Animagine-XL Series: Exceptional Composition!
The Animagine-XL series is a wildly popular anime model based on SDXL. Its standout feature is its innovative composition.
The Animagine-XL series excels at interpreting simple prompts freely, producing bold and dynamic compositions.
However, compared to the newer Illustrious-XL series, it struggles with fine details, often resulting in hand and finger deformities.
Illustrious-XL Series: Precision in Details!
The Illustrious-XL series, another SDXL-based anime model, may not match Animagine-XL’s bold compositions but shines in generating clean, detailed illustrations.
The brdrXL-v0.3b model, created by BD, is a custom model based on Illustrious-XL.
It stands out for its sharp resolution and high artistic expressiveness, making it particularly strong in detailed rendering.
Automatic Prompt ->Animagine-XL ->brdrXL
flowchart LR
A1(Reference Image)
subgraph Florence-2
B1(Caption<br>- Free Interpretation)
end
subgraph Animagine-XL
C1(Original Illustration<br>- Creative Composition)
end
subgraph brdrXL
D1(Redraw<br>- Detailed Rendering)
end
E1(Final Output)
A1-->B1
B1-->C1
C1-->D1
D1-->E1
Let’s explore a workflow that harnesses AI’s creativity by inputting a reference image to automatically generate prompts, then creating varied illustrations based on those prompts.
The prompts are generated by captioning images using the Florence-2 model, which excels at image recognition.
Generated Illustrations
Let’s dive into the illustrations we created.
Input Image
Generated Prompt
1girl, solo, smile, open mouth, black hair, dress, hat, holding, closed eyes, full body, black background, dress up, witch hat, purple dress, witch, holding basket, halloween
Original Illustration by Animagine-XL 4.0 Opt
Redrawn by brdrXL-v0.3b
Color Adjustment + Upscaling
Using Florence-2 and Animagine-XL to freely interpret the input image, we created a bold composition, which was then redrawn by the brdrXL model to produce a highly polished illustration.
More Examples with the Same Prompt!
Let’s look at other illustrations generated from the same input image.
Trying a Different Input Image!
Now, let’s generate illustrations using a different input image. The top image is the input, and the bottom is the generated illustration.
Woman with a Bouquet
Photographer in a Meadow
Moon Rabbit
The Actual Workflow!
Here’s a breakdown of the actual workflow.
1. Captioning Images to Create Prompts
Models Used
The first part of the workflow involves captioning images to create tag-based prompts using the Florence-2 model, which is adept at image recognition.
The prompt_gen_tags mode in the Florence2Run node outputs tagged captions from input images, ideal for generating prompts for anime illustrations.
These varied prompts are then used with the Animagine-XL 4.0 Opt model to create bold compositions.
2. Full Workflow with Redrawing
The complete workflow includes redrawing with the brdrXL-v0.3b model, followed by color adjustments and AI upscaling for high resolution.
Models Used
Custom Nodes
Here are the custom nodes used, along with their search screens in ComfyUI-Manager.
ComfyUI-Florence2
cg-use-everywhere
ComfyUI-SuperBeasts
ComfyUI-easygoing-nodes
Bonus: Overcoming Illustrious-XL’s “Perfection”!
Both Animagine-XL and Illustrious-XL require quality prompts and negative prompts as inputs.
The Illustrious-XL model excels at detailed rendering, but setting the CFG scale too high can reduce variety, leading to what’s called a “mask-like face”.
To counter this, we introduce the SDXL_Refiner model in the middle to disrupt the illustration and enhance variety.
Models Used
- GhostXL: A derivative of Animagine-XL 3.0 released in February 2024, still highly expressive artistically
- SDXL_Refiner: Disrupts the illustration to enhance completion
- brdrXL_v03b: Used for final polishing
GhostXL
SDXL_Refiner
brdrXL_v03b + Finishing
By using the Refiner to disrupt and then regenerate the illustration, we aim to reduce the overly perfect look typical of AI illustrations.
About Refiner
Bonus 2: Mixing Models in Stable Diffusion webUI
While this post focuses on the ComfyUI workflow, you can also mix models to generate illustrations using default features in Stable Diffusion webUI (A1111, Forge, reForge, Forge Classic).
Example with Stable Diffusion webUI Forge
Stable Diffusion webUI includes a feature to redraw with a Refiner model after the Base model, but you can also use any arbitrary model instead of the Refiner.
Set the total step count 20–30% higher than usual and the switch timing to around 0.7–0.8 for optimal results. Since the same prompt is used for both models, align the quality and negative prompts with the finishing model.
Conclusion: Combining the Best of Animagine-XL and Illustrious-XL!
- Caption with Florence-2
- Create compositions with Animagine-XL
- Polish with Illustrious-XL
In this workflow, we combined Animagine-XL and Illustrious-XL to great effect. While Illustrious-XL excels in detailed rendering, I initially found its glossy skin and clothing textures less appealing.
However, discovering the brdrXL model with its matte, expressive quality has been a game-changer, and I’m excited to use it more.
Animagine-XL and its derivatives produce unexpected compositions from simple prompts, showcasing AI’s creativity and remaining timeless models.
I’ll continue exploring unique expressions through model combinations.
Thank you for reading!
Reference Articles
Color Adjustment Settings
For settings of the HDR Effects with LAB Adjust node for color adjustments, see here.
ComfyUI Custom Nodes
An introduction to standard ComfyUI custom nodes, including those used in this workflow.