Writing · AI Product
LLM Orchestration: What I Learned Building a Chain and an Agent
Two small builds, two shapes: Mithivoices is a plain three-step chain, the AI-First CRM prototype is a LangGraph agent with five tools. What each shape was good for, and where the agent needed guard rails.
PM hiring question
How do you sequence LLM calls without it breaking?
Recruiter takeaway
"He picks the simplest orchestration that works."

01 · The question I keep getting
What exactly is LLM orchestration, and why should a product manager care about it? Simply put, orchestration is how you string together language model calls, tools and state so a task finishes without the system losing the thread. It is the difference between a chat wrapper and a product.
I have built two small pipelines with very different shapes. Neither has real users yet, so this is about design choices, not scale.
02 · Mithivoices: a chain is enough
Mithivoices takes a voice recording and answers out loud. One request runs three steps in order: Whisper turns speech into text, a language model writes the reply, and Piper turns the reply back into speech. The model can be Gemini in the cloud or Ollama on your own machine, behind one interface, so swapping providers is a setting rather than a rewrite.
There is no framework and no graph, because nothing needs to branch. The one judgement call sits in plain code: a language check decides whether to translate before speaking, and it does not translate when it is unsure which language the text is in. Wrong-language speech is worse than none.
The backend keeps no conversation history. That is the next real decision: the moment the assistant has to remember what you said three turns ago, the chain needs state, and that is when I would reach for something like LangGraph.
03 · The CRM: when a graph earns its keep
The AI-First CRM prototype is different. A field rep types something like "met Dr Rao today, discussed the new inhaler, she was positive", and the assistant has to decide what that is: a new visit to log, a correction to an earlier one, a question about the doctor, a compliance question or a request for a next step. Those are five tools, and choosing between them is the model's job. That is where LangGraph helped: one node calls the model, one runs the chosen tool, and the state carries the result back to the form.
The state is not the chat transcript. Each message starts a fresh run, and the memory is the current form: its values and the ID of the visit being edited go in with every turn. The form is the source of truth, so the model never has to remember it.
04 · Three patterns from the CRM
- Rules before the model: a compliance question ("can I visit Dr Rao again this month?") is caught by a simple pattern and answered straight from the database. It is faster, and the answer cannot be talked into being wrong.
- Typed tool results: each tool returns structured fields for the form, not prose. The interface applies the fields and highlights them for a moment, so the rep sees exactly what changed.
- Clean the reply: the model sometimes leaks JSON or tool names into its answer. Those lines are stripped before the rep sees them, and if nothing readable is left, a plain fallback message is shown instead.
05 · Where it broke
After a tool ran, the hosted model would sometimes try to call another tool, including one that does not exist. The fix was blunt: once a tool result is in, the next model call is not allowed to call tools at all, only to summarise.
The bigger flaw is about state, not the model. The chat saves a visit as soon as it understands it, and the form has its own Save button that creates a new record. Use both and you get a duplicate. Two paths that write the same thing need one owner, and in a pilot that owner would be the rep, confirming a draft.
06 · What I'd pick today
For a fixed sequence, plain function calls. For anything where the model chooses between tools or loops, a graph framework, because the control flow stays visible. Either way, the work that decides whether it holds up is around the model: the rules that skip it, the schema that constrains it, and the cleanup after it.
The SEO-GEO Optimizer uses a related idea at a larger scale: chained phases with a human approval gate between them, so nothing moves on without a review.