Toonflow — The AI Factory That Turns Novels Into Short Dramas

You’ve written a novel with AI. Now what? Toonflow takes the next step — it’s an open-source AI Agent workbench that converts novels into short dramas automatically. Text goes in, video comes out. No manual storyboarding, no frame-by-frame prompting. The AI handles character extraction, script writing, visual generation, and video synthesis in one pipeline.

Why This Matters Beyond Video

If you care about AI Agent orchestration — how to coordinate multiple specialized AI models to complete a complex creative task — Toonflow’s architecture is worth studying:

The Pipeline

Novel Text / Script Draft
       │
       ▼
┌─────────────────────────────────────────┐
│  [1] Event Extraction (LLM)             │
│      Characters → profiles (appearance, │
│      personality, relationships)         │
│      Plot → structured event graph       │
├─────────────────────────────────────────┤
│  [2] Script Generation (ScriptAgent)    │
│      Dialogue + scene descriptions +     │
│      stage directions                    │
├─────────────────────────────────────────┤
│  [3] Storyboard (LLM + Image Gen)       │
│      Visual prompts → AI frames          │
│      Camera angles, composition, props   │
├─────────────────────────────────────────┤
│  [4] Video Synthesis (Sora / Doubao)     │
│      5-20 second clips per scene         │
├─────────────────────────────────────────┤
│  [5] Production Assembly                 │
│      Editing + refinement + export       │
└─────────────────────────────────────────┘

Three-Layer Agent Architecture

This is the most interesting part from an engineering perspective:

1. Decision Layer — Plans the overall production: how many scenes, what style, pacing decisions. Think of it as the “director.”
2. Execution Layer — Runs individual tasks: write this script, generate this image, synthesize this clip. Think of it as the “crew.”
3. Supervision Layer — Quality checks: is the character consistent? Does the dialogue match the scene? Is the pacing right? Think of it as the “editor.”

Each layer can use different models. The Decision layer benefits from a strong reasoning model (Claude, GPT-4o), while the Execution layer can use faster/cheaper models for bulk generation.

Persistent Agent Memory: Agents maintain context across sessions using local ONNX vector retrieval — similar to how WebNovel Writer uses RAG-over-chapters.

Supported AI Providers

The Programmable Vendor System lets you write custom vendor logic in settings — no code changes needed to add new providers.

Installation

# Option 1: Docker (recommended)
git clone https://github.com/HBAI-Ltd/Toonflow-app.git
cd Toonflow-app
yarn docker:local
# → http://localhost:10588  |  Login: admin / admin123

# Option 2: Desktop app
# Download from GitHub Releases (Windows/Mac/Linux)

# Option 3: Server deployment
yarn install && yarn build
pm2 start pm2.json

Cost Per Episode

What This Teaches About AI Agent Systems

The same patterns apply to any multi-model AI pipeline:

1. Separate orchestration from execution — The Decision layer doesn’t generate images; it decides what to generate. Same principle as harness engineering.
2. Structured intermediate representations — Character profiles and event graphs are the “glue” between pipeline stages. Without them, each stage operates blind.
3. Vendor abstraction is critical — The Vercel AI SDK layer means swapping from Sora to Doubao is a config change, not a rewrite. Design for model portability.
4. Persistent memory enables iteration — Without cross-session memory, every run starts from scratch. The ONNX vector store is lightweight but sufficient.
5. Quality supervision must be automated — The Supervision layer catches inconsistencies that would otherwise require human review. This is the difference between “demo quality” and “production quality.”

Operational Notes

• Treat Toonflow as a reference architecture for multi-agent creative production, not as a guaranteed push-button studio.
• Tool-calling reliability varies by provider. If a model’s streaming tool calls fail, switch providers or isolate that stage behind a fallback.
• The vendor abstraction is the main engineering lesson: script, image, video, memory, and review stages should be replaceable without rewriting the whole workflow.
• Keep human review in the loop for final storyboards and releases. Automated supervision catches structural issues, but it does not replace editorial judgment.

How LearnAI Team Could Use This

• Study Toonflow as a reference architecture for multi-agent creative pipelines: decision, execution, and supervision layers.
• Use the novel-to-video workflow as a demo path for LearnAI content production when the input material is appropriate to keep.
• Turn provider-switching and supervision into training material on tool-calling reliability and vendor abstraction.

Real-World Use Cases

• Convert AI-written webnovel chapters into short-drama pilots or Douyin/TikTok storyboard packages.
• Prototype visual treatments for IP development before hiring a full production team.
• Teach agent orchestration using a concrete media pipeline with structured intermediate outputs.

Links

• GitHub: HBAI-Ltd/Toonflow-app
• Official Site: toonflow.ai
• Tutorial: Bilibili 8-min quickstart
• Discord: Community
• Version: Latest release on GitHub