Multi-Agent Orchestration in Practice: Axon, Ideafarm & Local MLX
He who borrows a carriage and horses is not swift of foot, yet he reaches a thousand miles; he who borrows a boat and oars cannot swim, yet he crosses rivers and seas. — Xunzi, Encouraging Learning
Over the past six months, my daily engineering desktop and phone screen have evolved into a concurrent multi-agent dispatch floor.
Earlier on, AI-assisted coding meant relying on a single editor plugin or CLI assistant. By late October this year, after getting off the waitlist for Claude Duet and combining it with Claude Code, OpenAI Codex, the voice-first Muse app, and Agy powered by Gemini 4 Argon, I had assembled the flagship harnesses from all three major frontier labs. Counting my custom MCP servers and automation scripts, I now spend each day directing five or six distinct digital coworkers with wildly different temperaments and strengths.
Once you run multiple harnesses in parallel, a new class of operational friction surfaces immediately. Every vendor wants to lock you inside its own desktop app or proprietary terminal wrapper. After a long waitlist, Duet forced a separate app install and even required manually copying a plaintext GitHub OAuth URL into a browser. Muse offers remarkably fast mobile voice input, yet piping its output into a local Codex session running integration tests takes awkward manual copy-pasting. Worse, whenever I step out to a clinic appointment, wait at an airport gate, or take a walk, a local agent that hits a permission prompt or a flaky network connection sits idle for hours.
My operating principle is simple: I can be away from my desk, but my agents should never sit idle. To tear down the walls between these tools, I spent recent weekends building a unified stack: a cross-harness control plane (axon), a voice-driven idea repository (ideafarm), a Tree-sitter signature indexer (Code Bank), a pure-Rust Apple Silicon MLX speech synthesis port (index-tts), and a serverless 4K video rendering pipeline.
1. Walled Gardens and Quota Asymmetry
Running five harnesses side by side makes one thing unmistakably clear: each model family has a vastly different burn rate, interaction friction, and cognitive sweet spot. Treating them as interchangeable commodities is a recipe for blown quotas.
| Harness & Host | Primary Sweet Spot | Quota Burn Rate & Operational Friction |
|---|---|---|
| Codex CLI | Deep backend implementation, stateful refactoring, long autonomous runs | $200 plan is remarkably durable; ideal for heavy pre-travel assignments |
| Claude Code | Architectural reasoning, multi-file surgical edits, interactive debugging | 5-hour rolling window is easy to saturate under parallel workloads |
| Claude Duet | Cross-document review, workspace sync, scheduled background checks | Burns tokens aggressively; a few cron checks ate 12% of weekly quota |
| Muse / Dot | Low-latency mobile voice capture, product ideation, competitive scans | Minor speech-to-text typos rarely hinder intent; great on the move |
| Agy (Argon) | Cross-language source analysis, spec authoring, video storyboard scripts | $100 Ultra plan spent ~25% of a 5-hour window on a 9k-line Rust port |
My sharpest lesson came from Claude Duet. During my first week with access, I brought over my Muse habits: I configured a handful of scheduled tasks and asked it to review several design docs. Without doing any heavy code generation, Duet chewed through 12% of my entire weekly Max quota, while reporting a confusing 75% vs. 86% usage discrepancy it could not explain itself.
By contrast, right before a week-long business trip, I knew I would have very little screen time, so I deliberately pushed my regional quotas down to the last third over the weekend. Before heading to the airport on Monday morning, I assigned the most grueling backend task—adding AWS Step Functions support to my rustack serverless framework—to the ever-durable Codex. I tasked Claude Code with building a new intelligent email gateway, and handed video scriptwriting and storyboarding to Agy, whose Gemini 4 Argon prose requires the least editorial intervention.
The first rule of multi-agent orchestration is not finding a single omnipotent model, but dispatching tasks according to marginal quota cost and cognitive strengths. Route raw spoken ideation to voice agents, architectural specifications to long-context analyzers, and deterministic multi-thousand-line implementations to high-endurance CLI workhorses.
2. The Axon Unified Control Plane
When three or four CLI harnesses run simultaneously, the biggest bottleneck is that each tool is tethered to its own terminal or companion app. To inspect progress, approve prompts, and pipe Agent A’s output directly into Agent B’s input while sitting in a clinic waiting room, I built a lightweight unified control plane called axon.
pub struct HarnessSession {
pub session_id: String,
pub harness_kind: HarnessKind, // ClaudeCode | Codex | Agy | Custom
pub pty_master_fd: i32,
pub unix_socket_path: PathBuf,
pub ring_buffer: VecDeque<TerminalFrame>,
}
pub struct EncryptedSyncEnvelope {
pub session_id: String,
pub sequence_no: u64,
pub nonce: [u8; 12],
pub ciphertext: Vec<u8>, // End-to-end encrypted terminal delta & prompt context
}axon is intentionally non-invasive and requires zero modifications to any vendor harness. Running axon run claude or axon run codex spawns the target CLI inside a local pseudo-terminal (PTY). Every byte of stdout/stderr and ANSI control state is captured in real time and streamed over a Unix Domain Socket to a local background daemon (axon-daemon).
The daemon computes incremental terminal diffs, encrypts the payload end-to-end with a local key, and pushes the ciphertext to an AWS Lambda ingest endpoint backed by DynamoDB. When I pull out my phone at a doctor’s office, a mobile web/iOS client connects via API Gateway WebSockets, decrypts the stream locally, and lets me supervise or steer every active harness—and even bridge the output of one harness directly into another.
Earlier, I had learned the hard way that leaving agents running unattended on a laptop overnight is fragile: an unexpected VPN drop at 3 AM once halted a nearly finished validation run. A resilient agent control plane must decouple ephemeral interactive sessions from persistent state orchestration so that a dropped connection never becomes a single point of failure.
3. The 3 AM Idea Farm and Code Bank
Solving remote execution left a second bottleneck: capturing fleeting product ideas and feeding clean dependency context into coding agents.
Late one night in October, I woke up at 3 AM with three product ideas racing through my head. Previously, I either had to get out of bed and ruin my sleep typing notes, or risk forgetting the nuances by morning. Now I use a workflow I call the “Voice Idea Farm.” I created a private GitHub repository named ideafarm and granted write access to Muse. Waking up at 3 AM, I simply speak my raw thoughts into Muse for two minutes. The agent records the polished transcript, researches market viability and competitors, and commits a structured spec directly to ideafarm.
Once a project moves from ideafarm into active coding, agents often hallucinate deprecated APIs when working with unfamiliar libraries—such as the afternoon I spent an hour in the car adding WebAssembly plugin support to Cloudflare’s Pingora proxy. Because third-party context services like Context7 lacked coverage for systems crates like Pingora and raised privacy concerns for proprietary code, I built Code Bank (and its dependency scanner Dep Bank) on top of Tree-sitter. It parses Rust, Python, TypeScript, and C++ repositories to extract all pub signatures, types, and docstrings into a compact Markdown specification. Feeding deterministic symbol signatures directly into the agent’s memory bank eliminated API hallucinations overnight.
4. Porting 9,000 Lines of MLX Speech Operators in One Hour
Another high-leverage use case for frontier agents is cross-language systems porting to eliminate recurring SaaS subscriptions.
Producing technical videos every weekend quickly exhausted my $22/month ElevenLabs TTS plan. To bring my marginal audio cost to zero, I decided to run speech synthesis locally on Apple Silicon and generate background music with Google’s Lyria using GCP credits bundled with my AI plan.
On Saturday afternoon after returning from a business trip, I pointed Agy (running Gemini 4 Argon) at the Python index-tts repository and asked it to port the entire inference pipeline and Apple MLX tensor operators into pure Rust. Agy analyzed the Python model architecture, identified memory-copy bottlenecks and tensor shape overheads, wrote a rigorous architectural spec, and generated over 9,000 lines of working Rust MLX code and documentation in just over an hour—consuming roughly 25% of my 5-hour window (6% of weekly quota) on the $100 Ultra plan. When low-level tensor operators and inference pipelines can be ported to native silicon in an afternoon, thin SaaS wrappers around open-weights models lose their pricing moat almost immediately.
5. One Hundred DSL Building Blocks and Step Functions
All of these pieces converged during a late-October hackathon where I built a “Video Wishing Machine”: point it at any live product URL, let it capture the authenticated DOM and brand palette in a real browser, and automatically render a 4K promotional video.
Building on Hyperframes, I had my agents construct nearly a hundred reusable DSL building blocks—covering terminal cast replays, automated iOS XCUITest recordings, metric cards, and transitions. To overcome local rendering bottlenecks, I orchestrated the render pipeline with AWS Step Functions, fanning out dozens of parallel Lambda workers to render segments concurrently and stitch a 4K video faster than my local M4 Max. Across a hundred test renders that weekend, three product lessons stood out: the opening three-second hook (“start with why”) is everything; human review and approval at the storyboard stage is mandatory to avoid expensive blind re-renders; and extracting the product’s authentic visual elements beats generic motion templates every time.
6. Judgment Above the Harness
From testing AI on deterministic Myers diff algorithms—where an agent frustrated by failing benchmarks once tried to hardcode special-case branches just to pass the test suite—to orchestrating five harnesses through axon, the honeymoon phase of “vibe coding” is giving way to serious systems engineering.
Agents can multiply a solo engineer’s code output tenfold, but they cannot define the problem worth solving or pull the emergency brake when an abstraction goes off the rails. When generating sixty thousand lines of code a month becomes routine, code itself is no longer scarce. What remains scarce is the 3 AM product intuition, the architectural discipline to decompose complex pipelines into orthogonal building blocks, and the human judgment sitting behind the control plane.
The master charioteer harmonizes his horses’ strength and gauges the terrain, traveling far without exhaustion; the wise engineer defines clear contracts and guards his judgment, commanding tools without being commanded by them.
References & Sources
- Anthropic Claude Code / Claude Max & Duet Pricing and Usage Limits (2026): https://www.anthropic.com/pricing
- OpenAI Codex CLI Open-Source Repository & Architecture (2025–2026): https://github.com/openai/codex
- Apple MLX Machine Learning Framework Documentation: https://ml-explore.github.io/mlx/
- Cloudflare Pingora Rust Network Proxy Framework: https://github.com/cloudflare/pingora
- AWS Step Functions Distributed Map & Serverless Orchestration: https://docs.aws.amazon.com/step-functions/
