Building an AI Agent Orchestrator with Ollama

Why an orchestrator?

A single LLM call works well for simple Q&A, but real-world tasks — debugging a stack trace, writing production code, reviewing for security issues — benefit from specialised agents that each bring a different capability and context window.

We built DevFlare's orchestrator around a simple insight: let one agent plan, then dispatch each subtask to the agent best suited for it, and combine the results into a coherent answer.

The architecture

The orchestrator follows a plan → dispatch → combine pipeline:

1. Planner agent — receives the user's raw request and produces a JSON plan of tasks, each tagged with the agent that should handle it (research, code, debug, writer, QA, security, devops).
2. Dispatch loop — runs each task through its assigned subagent, collecting outputs. A hard cap of 8 total steps prevents runaway plans.
3. Final synthesis — combines all contributor outputs into one markdown answer.

Each subagent has its own system prompt, token limit (600–1200), and runtime budget (60–90s). The planner is the only agent that sees the full request; downstream agents receive only their assigned task plus relevant context from prior outputs.

Safety features

• Prompt injection defence — control characters are stripped, common injection phrases are flagged and logged, and system prompts are never included in API responses.
• Cancellation — the orchestrator accepts an AbortSignal; if the user cancels mid-run, in-progress agents are stopped and the run is marked CANCELLED.
• Per-step timeouts — every agent call has a configurable timeout (default 120s) with retry logic.

Per-agent model selection

Each agent resolves its model in this order: a workspace-level override → per-agent env var (OLLAMA_CODE_MODEL, etc.) → OLLAMA_DEFAULT_MODEL → a hard-coded fallback. This lets you run cheap models for planning and expensive ones for code generation.

What we learned

The planner's output format is the most critical piece. If the planner produces vague tasks, downstream agents produce vague results. We iterate on the planner prompt more than any other single component.

The other lesson: streaming matters. Users will wait 30–90s for a multi-agent run. Streaming each agent's output as it completes turns that wait into a readable narrative.