TL;DR
- ·Small teams building voice AI had two options: pay per-minute to closed APIs like Deepgram or ElevenLabs, or dig through research code that never ships.
- ·I built Mithivoices from scratch and open-sourced it - a modular stack with STT, LLM routing, TTS, and telephony adapters.
- ·A 5-minute quickstart that puts a working voice agent on a phone number. Production-ready and open for contributors.
Overview
Mithivoices is an OSS opinionated stack for real-time voice agents. It optimises for latency and swap-ability over completeness.
Problem Statement
The gap between a voice demo and a production voice agent is measured in months. Latency, interruption handling, and cost per minute kill most builds.
Market Need
The closed-API problem was getting worse, not better. Builders who could afford hosted providers were fine. Everyone else was building on top of things that could change terms at any time. An open, swappable stack felt like infrastructure that was missing.
Target Users
Indie hackers, small SaaS teams, and researchers who want to ship, not train.
Research
Went deep on a stack of GitHub repos that had stalled - looking for where they broke. Talked to builders who had abandoned voice projects mid-way. Interruption handling and latency were the two things that killed otherwise-working demos, across almost every conversation.
Insights
- ·Builders don't want more models - they want opinionated defaults that let them skip the research phase entirely.
- ·Sub-500ms end-to-end latency is the difference between 'demo' and 'usable.'
- ·Swap-ability of STT/LLM/TTS is a hard requirement. Projects built on single-provider stacks break the moment pricing changes.
Opportunity
Ship the fastest, most swappable OSS voice stack - and let community modules cover the long tail.
Scope
- ·Latency-first pipeline architecture
- ·Pluggable STT / LLM / TTS providers
- ·Interruption + barge-in handling out of the box
Out of Scope
- ·Hosted SaaS layer
- ·Voice cloning
- ·Custom model training
Assumptions
The biggest bet was that contributors would show up if the core was opinionated enough. Open-source projects that try to please everyone often end up serving no one. I picked a default stack and was prepared to be wrong about the choices.
Success Criteria
- ·Target: first voice agent shipped by a stranger within 30 days of v1 - if someone outside the project can install and run it, the quickstart is good enough.
- ·Target: median end-to-end latency under 500ms in the reference agent.
- ·Kill condition: if the quickstart takes longer than 15 minutes for a developer who reads the README once, the defaults are not sane enough.
Expected Behaviour
Clone repo → run the reference agent → swap one provider → ship a branded agent in a weekend.
North Star Metric
Voice Agents Shipped by External Users
Supporting Metrics
- ·Median E2E Latency
- ·External Contributors
- ·GitHub Stars-to-Fork Ratio
Prioritization
Every roadmap item was scored against 'does this reduce time-to-first-agent?' Anything that didn't was closed without ceremony.
Wireframes
The README was the prototype. I wrote it before I wrote any code - describing how a developer would install and run a voice agent in a weekend. If the README made it sound easy, the API needed to make it easy. Everything I built was measured against whether the quickstart stayed honest.
PRD
Module-level RFCs in the repo rather than a top-level PRD. Each module - STT adapter, interruption handler, TTS wrapper - had its own one-pager: the interface it exposed, what it didn't do, and what the open questions were.
Prototype
A live agent on a real phone number. Every architectural decision had to survive an actual call. If it felt wrong on a call, it didn't matter that it looked right in a spec.
Final Solution
A clean pipeline, provider adapters, and a 5-minute quickstart that puts a working agent on a phone number.
Reflection
I built Mithivoices myself and then open-sourced it - which changed the design from day one. When you open-source something, every part has to work on its own. You can't have a messy monolith if strangers are going to use it. That's a product decision as much as an engineering one. The thing I didn't expect about OSS PM: contributors take the project in directions you didn't plan, and some of those directions are better than yours. I ran the module backlog through a RICE score and thought I had the reach estimates right. A contributor working on a deprioritised STT adapter pointed out I'd undercounted how many downstream modules depended on it - I'd scored from the roadmap, not the dependency graph, and he was right. I revised the score, shipped v0.2.0 on the adjusted priorities, and he stayed active on the project afterward. The framework didn't make me right. It made my reasoning visible enough for someone else to catch what I'd missed.
Lessons Learned
- ·The quickstart is the marketing site for developer tools. If setup takes more than a few minutes, most people don't get to the product.
- ·I was tempted to build a hosted tier early. Not doing it was the right call - but it was a harder call than I expected, because the hosted tier would have made the demo much easier to show.