It Was Never the Pipeline

Over the past eighteen months, I've put together five speech pipelines that take audio, turn it into text, and score it. Every time, I thought the model would be the hard part, but it never was. The problem was different each time, so I didn't see what kept repeating until I compared all five projects side by side. That comparison is where what I'm building now came from.

One: changing the model

The first was for a call center. It started with a speech-to-text API, but halfway through we moved transcription to a prompt for a general model, then to a newer model, and then to another one after that. The scoring and storage stayed the same, and the people reading the results didn't notice the changes.

I had built the whole pipeline around the model, which made changing the model easy, while most of what I'd built around it kept working.

That didn't convince me right away, and without the next four projects, it probably wouldn't have. I still thought everything else depended on which model I picked.

Two: preparing the audio

The second covered only the audio processing before paying for tokens. It cut silences longer than 3 seconds, with a conservative threshold to keep real pauses. It detected hold music, but waited 7 seconds before classifying it that way: a false positive would delete a customer's words, whereas a false negative only cost a few cents, so I'd rather keep some music. Then downsample, convert to mono, and keep it lossless.

Those thresholds mattered a lot, but I'd left them as constants in a file only I could change. A quality analyst knows what a long pause means on a collections call far better than I do, so those settings should be theirs to change, with safe ranges so they can't break the processing by accident.

Three: the health network

The third was for a health network. It had three jobs: 50 workers processing audio, 50 processing text, and one process to put the results together. Because nobody had asked for a whole product, it didn't need the screens, endpoints, and auth of the first two. The work underneath was the same, but building only the processing recipe let it ship faster than any of the others.

Four: checking the result

The fourth processed field audio from a utility: transcribe, summarise, and get 12 attributes. Each one came back with a confidence value and the line of transcript behind it when there was one. Because supervisors could check an answer if it looked wrong, rather than having to accept or reject all 12 together, they actually used it.

I had left the output format until the end. After this project, I wanted that decided before running anything.

By then, more than one version had its own cost calculator added late, so we'd only get a report after the money was spent. I wanted the cost per minute to be a limit in the recipe from the start, so the pipeline couldn't just go over it.

Five: a phone call

The fifth runs in realtime, talking to a person on the phone, where a lot of the earlier choices don't work. Silence helps you know whose turn it is, so you can't cut it out. The slow, accurate model takes too long, and you have to handle retries while someone is waiting on the line.

The first serious bug was the voice losing its Chilean accent after an interruption. Three places asked for that voice: the note, the audition, and the call. We changed the sentence that sets the accent, but the same instruction also said which voice to use after an interruption, and that part still said neutral Spanish. So we'd tell it to keep the accent, but drop it whenever someone cut in. The model wasn't the problem. The instruction was telling it two different things.

The fix was deciding who owns what before the call. We put the accent line in one place, so the note, the audition, and the call all read it from there. Now a profile sets what the voice speaks and what the transcriber expects, so neither side has to guess. Once we had that contract, the accent bug went away, and so did the other bugs like it.

What I'm working on now

I want an app to ask for a result by setting its limits (budget, latency, privacy, and quality) without having to say too much about how to get there. It shouldn't have to say "use this model, this endpoint, this prompt". It should be able to ask for "I need this result, under these constraints."

Something turns that request into an exact recipe, fixed for that run. A specialised processor runs just that recipe, and sends back the result and a receipt with the steps, the cost, and the confidence.

For the fourth pipeline, I'd ask for 12 attributes per recording, each with a confidence and the evidence behind it, agreeing on a cost per minute before running it. (This example doesn't need realtime. That comes in the fifth). The recipe names the model, so we could change it next month without changing the work after it. The thresholds go there too, with who's responsible for each one, and the supervisor gets the cost and confidence in the receipt.

So the recipe has the model, settings owned by someone outside engineering, a budget it can't go over, and the output format, while saying whether each step can run twice. I didn't plan all of this at the start: these needs kept coming up across the five projects, and now I'm trying to deal with them together.