Building AI Features Users Actually Trust

The AI-First CRM lets pharma reps log physician interactions by talking to an assistant instead of filling a form. The assistant is powered by a LangGraph pipeline. It works. Reps did not use it at first, because they did not trust it.

Reps in a regulated industry cannot afford to log wrong information - a mis-classified interaction can breach compliance. The model was performing well on our eval. That number is a great engineering result and a terrible product argument. 'Mostly right' is not a promise a rep can carry into an audit.

Adoption in the first two weeks was low. Reps used the assistant to draft, then re-typed the same information into the form manually. We had shipped a very expensive typewriter.

I sat with a few reps for an afternoon each and watched them use the tool. One rep asked the same question out loud every time the assistant responded: 'how do I know it heard me right?' He wasn't asking for better accuracy. He was asking for a receipt. I wrote that word in my notes and didn't fully understand it until I watched him re-type the assistant's output by hand.

I also read every help ticket from the first month. Zero were about accuracy. All were about two things: auditability ('can I see what got logged?') and reversibility ('how do I fix it if it's wrong?').

We did not retrain the model. We added four affordances.

• Read-back: after each turn, the assistant echoes what it heard in structured form before saving. One extra tap. Zero ambiguity.
• Source pinning: every logged field shows the exact rep utterance it came from. Trust is easier when the receipt is visible.
• Uncertainty as UI: confidence below a threshold surfaces as a yellow badge with "confirm?" - not a hidden score.
• Reversible commit: every save has a 30-second undo without leaving the screen. Small, but it changes the emotional cost of trying.

Adoption moved substantially - from a small fraction of reps to most of them - in about a month. The model accuracy did not move. The rate of manual re-typing dropped to near zero.

The support tickets shifted topic entirely: from 'how do I check what got saved?' to feature requests for new interaction types. When the complaint changes, the product has crossed something.

The mistake was optimising the wrong number. I had spent two weeks squeezing a few more accuracy points out of the model. Those points were invisible to the user. The read-back UI took a day and moved adoption far more.

The trade-off I still watch: friction added for trust can become friction added for its own sake. The read-back is one extra tap, worth it. If I added a second confirmation on top, I'd be paying for safety I didn't need with adoption I couldn't afford. That balance point has to be checked with real users, not felt.

The same failure mode - optimising the wrong thing because the metric isn't defined - is what led me to build SEO-GEO Optimizer. AI IDEs were optimising for SEO signals that Google deprecated in March 2024, because nobody had defined the right success criterion for them.