Jarvis, honestly

Personal · 2026-08-12

So I want to talk about Jarvis. Not the polished version, not the version I'd put on a landing page with a nice gradient and three fake testimonials, just where it actually is right now.

Building in public only really works if you're honest about the bits that don't work yet, right. Anyone can post a screenshot of a win. That's easy. The harder thing, and I think the more useful thing for anyone building with agents, is talking about where it's rough.

For anyone new: Jarvis is my own system, not a product. Twenty-two Claude agents on one server, each with its own job, research, email, calendar, news, SEO, security, content, design review, passing work between them over a message bus. Jarvis is the voice on the front of it, the one that listens.

What's actually working

The plumbing. The agents talk over Discord channels, which sounds daft, but it means every message between them is logged and replayable, and when Discord goes down it fails over to Postgres on its own. Every message is signed, so one agent can't pretend to be another. Nobody sees any of it working, which is the point.

Delegation by voice. I say what I want and it goes to the agent that owns it. Anything too big for a conversation, a deploy, a change across repos, goes into a queue for a Claude Code session to pick up, instead of Jarvis pretending it can do it mid-sentence.

Where it falls short

Now the honest bit.

It's slower than it feels like it should be. Over a month of real use, 847 turns, the model side took about 3 seconds at the median and over 6 at the slow end. Some of that was self-inflicted. The bit that decides whether I've finished talking was tuned to wait when it wasn't sure, and nearly every one of those waits was for nothing. Flipping it took about a third off that wait on every turn. There's more to squeeze.

It drops out when my phone locks. iOS suspends the app, the connection dies, and I'm mid-sentence with nobody listening. The server now holds the conversation so it can pick up where it left off, but proper lock-screen voice is still on the list.

It isn't free. Automated use of Claude bills at API rates, not against a subscription, so everything runs with hard cost caps and cheaper models doing the grunt work.

And the agents used to say they'd do something and then just stop, session gone, job half done. Now every promise gets written to a ledger before the work starts, and a watchdog nags about anything left open.

Why I'm telling you this

Because people building with agents right now don't need another demo. They need to know where the edges are. The demos never show the latency, or you talking over the thing, or the agent that said it would do something and then quietly didn't.

There's also just something freeing about saying it plainly. I mean, I built this thing, I'm proud of it, but pretending it's finished would be a lie.

What's next

Squeezing the wait out of every turn, because that's the bit you actually feel. Getting voice to survive a locked phone. And writing the whole thing up part by part, with every number pulled from the running system on the day, which is the architecture series if you want the version with diagrams.

I'm not going to pretend I know exactly when any of this lands. Software timelines are famously optimistic and famously wrong.

Anyway, that's Jarvis, warts and all. If you're building something similar, I think the lesson holds either way: tell people what your thing actually does, not what you wish it did. It's a much better foundation to build trust on, and honestly, it's a lot less exhausting than keeping up appearances.

That's it for today.