Speech AI: How STT and TTS Models Work

Speech AI covers two mirror-image tasks: recognising what was said and producing natural-sounding speech. Both rest on the same idea as text models, turning a signal into a sequence a neural network can predict, but audio brings its own problems: noise, accents, overlapping speakers and a very tight latency budget in conversation.

Audio as data

A microphone measures changes in air pressure. Digitising them means taking a sample many times per second. The sample rate sets the highest frequency you can capture (half the rate), and different uses settle on different values: 8 kHz for traditional telephony, 16 kHz for most speech models, and 44.1 or 48 kHz for music and video. Speech is intelligible with frequencies below 8 kHz, which is why 16 kHz is the usual compromise.

Raw samples are a poor input: one second at 16 kHz is 16,000 numbers, and the same word looks very different as a waveform depending on pitch and loudness. Most models therefore start from a spectrogram. The audio is cut into short overlapping windows, typically 25 ms long every 10 ms, and each window is converted into energy per frequency band. A mel scale spaces those bands the way human hearing does, finely at low frequencies and coarsely at high ones, and taking the logarithm compresses loud and quiet sounds into a similar range. The result, a log-mel spectrogram, is effectively a narrow image: around 80 rows of frequency bands and 100 columns per second.

import librosa

# Load as 16 kHz mono, the format most speech models expect
y, sr = librosa.load("speech.wav", sr=16000, mono=True)

# 25 ms windows (400 samples) every 10 ms (160 samples), 80 mel bands
mel = librosa.feature.melspectrogram(y=y, sr=sr, n_fft=400, hop_length=160, n_mels=80)
log_mel = librosa.power_to_db(mel)

print(y.shape, log_mel.shape)   # for 10 s of audio: (160000,) (80, 1001)

Not every model uses hand-designed features. wav2vec 2.0, for example, learns its own representation directly from the waveform with a convolutional encoder.

Speech-to-text: from waveform to words

Speech-to-text (STT, also called automatic speech recognition or ASR) usually runs as a short pipeline. Voice activity detection (VAD) finds the stretches that contain speech, so the recogniser does not waste time on silence and long recordings can be split at pauses. Features are extracted, the model produces text, and post-processing adds punctuation and casing and turns "twenty pounds" into "£20" (inverse text normalisation). Two families of recognition model dominate.

Encoder–decoder models such as Whisper

OpenAI's Whisper is a transformer with two halves. The encoder reads a 30-second log-mel spectrogram. The decoder then generates text tokens one at a time, exactly like an LLM, while attending to the encoded audio. Special tokens at the start of the output select the language, the task (transcribe, or translate into English) and whether to emit timestamps.

Whisper was trained on about 680,000 hours of audio paired with transcripts collected from the web. That breadth made it robust to accents, background noise and technical vocabulary, and it is multilingual out of the box. Its open weights made it the default starting point for self-hosted transcription. The design also has costs: it works on 30-second windows rather than a live stream, and because the decoder is a language model, it can write fluent text that was never spoken.

CTC and transducer models for streaming

The other family predicts a symbol for every audio frame. With connectionist temporal classification (CTC), the encoder outputs, for each frame, a probability over characters or subword pieces plus a special "blank". Decoding merges repeated symbols and drops the blanks:

frame outputs:     h  h  _  e  _  l  l  _  l  o  o  _
merge repeats:     h  _  e  _  l  _  l  o  _
drop blanks:       h  e  l  l  o

The blank between the two l runs is what keeps "hello" from collapsing into "helo". CTC models are fast and need no autoregressive decoder. wav2vec 2.0 showed how to pretrain such an encoder on unlabelled audio and then fine-tune it with CTC on a small amount of transcribed speech. Transducer models (RNN-T and its successors) add a small prediction network that conditions on the tokens emitted so far, which improves accuracy while still emitting text as audio arrives. They are widely used for live captions and on-device dictation, where results must stream: interim words appear immediately and are revised until the segment is final.

Transcribing a file locally

The faster-whisper package runs Whisper models with an optimised inference engine on CPU or GPU and includes a VAD filter:

# pip install faster-whisper
from faster_whisper import WhisperModel

model = WhisperModel("small", device="cpu", compute_type="int8")

segments, info = model.transcribe(
    "meeting.wav",
    beam_size=5,
    vad_filter=True,        # skip silence; reduces invented text
    word_timestamps=True,
)
print(f"language: {info.language} (p={info.language_probability:.2f})")

for seg in segments:        # a generator: transcription runs as you iterate
    print(f"[{seg.start:7.2f} -> {seg.end:7.2f}] {seg.text}")

Larger checkpoints are more accurate and slower. For a fuller guide to hardware and model sizes, see running models locally.

Measuring recognition quality

The standard metric is word error rate (WER): align the model's output with a reference transcript and count substitutions (S), deletions (D) and insertions (I), divided by the number of reference words (N). WER = (S + D + I) / N. Because insertions count, WER can exceed 100%.

import jiwer

reference  = "send the invoice to anna by friday"
hypothesis = "send the invoices to anna friday"

print(jiwer.wer(reference, hypothesis))   # 0.2857...: 1 substitution + 1 deletion over 7 words

A single number hides a lot:

• All words weigh the same. Dropping "by" costs as much as getting a drug name or an account number wrong.
• Normalisation changes the score. Whether "10" equals "ten", and whether casing and punctuation count, can move WER by several points. The Whisper authors built a dedicated text normaliser for this reason.
• Averages hide the tails. A good overall WER can conceal poor results for particular accents, noisy phone calls or your domain vocabulary.
• It ignores everything except the words. Punctuation, speaker labels and timestamps need their own checks.

Evaluate on a sample of your own audio, and track errors on the terms that matter to you. For languages written without spaces, such as Chinese or Japanese, character error rate (CER) is used instead.

Who spoke when: diarization and timestamps

A transcript of a meeting is far more useful with speaker labels. Speaker diarization answers "who spoke when" without knowing who the speakers are. A typical pipeline segments the audio into speaker turns, computes a speaker embedding (a voice fingerprint vector) for each segment, and clusters the embeddings so that each cluster becomes SPEAKER_00, SPEAKER_01 and so on. The open-source pyannote.audio toolkit provides pretrained pipelines for this.

To merge diarization with a transcript you need word timestamps. Each word is assigned to the speaker turn it overlaps most. Whisper's native timestamps are per segment, so tools derive word timings from the model's attention or with forced alignment against a CTC model. Overlapping speech and short interjections remain the hardest cases. If you control the recording, one channel per speaker (as in many call-centre and conferencing systems) removes most of the problem.

Text-to-speech: from words to waveform

Text-to-speech (TTS) runs the problem in reverse, and the hard part is everything that is not in the text: pronunciation, rhythm, emphasis and emotion.

The classic pipeline

1. Front end. Text normalisation expands what people write into what people say: "Dr Lee paid £5 on 3/4" becomes "Doctor Lee paid five pounds on…", which may be 3 April or March 4th depending on locale. Grapheme-to-phoneme conversion then turns words into pronunciation symbols. Most audible TTS errors start here.
2. Acoustic model. Predicts a mel spectrogram from the phonemes. Tacotron 2 did this with an attention-based sequence-to-sequence network; later non-autoregressive models predict each sound's duration explicitly, which is faster and avoids skipped or repeated words.
3. Vocoder. Converts the spectrogram into audio samples. Early neural vocoders such as WaveNet generated one sample at a time and were slow; HiFi-GAN, trained adversarially, generates the waveform in parallel, faster than real time even on a CPU.

Audio tokens and language-model TTS

Neural audio codecs compress audio into a short sequence of discrete codes per frame, and a matching decoder turns the codes back into sound. Once audio is a sequence of tokens, TTS can be treated as language modelling. VALL-E did exactly this: given the text and the codec tokens of a three-second recording of a speaker, it generates codec tokens that continue in the same voice, recording conditions included.

The token approach brings natural prosody and zero-shot voice cloning. It also inherits language-model failure modes: occasional skipped or repeated words, less precise control over pronunciation, and higher compute cost. As of 2026, many commercial and open TTS systems use codec tokens or hybrids of the two designs, and most let you correct pronunciation of names and terms with a lexicon or phoneme hints.

Voice cloning, consent and watermarking

If a few seconds of audio are enough to clone a voice, anyone's voicemail greeting is enough too. Cloned voices have been used to impersonate relatives and executives in fraud calls, and voice alone is no longer a safe way to authenticate someone.

Watermarking embeds an inaudible signal in generated audio that a detector can find later; Meta's open-source AudioSeal and Google DeepMind's SynthID are examples. Watermarks help providers trace their own output, but they only exist if the generator adds them and can be weakened by heavy re-encoding. Classifiers that try to detect synthetic speech without a watermark are unreliable. Regulation is moving in the same direction: the EU AI Act, for instance, includes transparency obligations for synthetic audio.

Voice agents: STT, LLM and TTS in one loop

A voice agent listens, decides and speaks. The common cascaded design chains three models: streaming STT produces a transcript, an LLM produces a reply (possibly calling tools, see agents and tool calling), and streaming TTS speaks it. Every stage adds delay, and the gap people tolerate between turns in conversation is short: a few hundred milliseconds feels natural, while a wait well over a second feels like a bad phone line.

Teams shorten the chain by streaming everything: the LLM starts as soon as the transcript is final, its tokens flow into TTS sentence by sentence, and audio plays as soon as the first chunk is ready. A short first sentence in the reply helps too.

Native speech-to-speech models take audio in and produce audio out within one model, using audio tokens on both sides. Examples as of 2026 include the models behind OpenAI's Realtime API and Kyutai's open Moshi. They respond faster, hear tone and hesitation, and can sound more natural. In exchange you get less control, fewer model choices and harder debugging, and you still need a transcript for logging, compliance and downstream automation.

Turn-taking and barge-in

Deciding when the user has finished is harder than it looks. A fixed silence threshold either cuts people off mid-thought ("my number is 07700…") or makes the agent sluggish. Better systems combine VAD with a model that judges whether the utterance sounds complete.

Barge-in is the reverse: the user starts talking while the agent is speaking. The agent must stop playback quickly, cancel the LLM and TTS work in flight, and record in the conversation history only the part of the reply the user actually heard. Echo cancellation keeps the agent from hearing its own voice, and the system should distinguish a real interruption from a passing "mm-hm".

How to choose

Whatever you pick, build a small test set from real recordings in your setting, with the accents, microphones and vocabulary your users have, and compare candidates on it before trusting any published score.