Speech recognition, denoising and text-to-speech

Meera’s WhatsApp showed 23 unplayed voice notes. The Baner store manager talked in a quick mix of Marathi and English. Delivery partners recorded from scooters, with horns in the background. A flour supplier left four-minute messages about prices.

“I want to read these,” she said. “In English, in ten seconds each. And a second thing: when an order is ready, someone shouts the number across the shop. Customers can’t hear it over the mixers. Could the store speaker announce it instead?”

Arjun decided to start with the announcement, which sounded easy, and then use it to test the voice-note reader before letting it anywhere near Meera’s private messages.

A clever test that went badly wrong

The announcement took six lines. A text-to-speech model from the Hub turned “Order 1042, two Diwali hampers, is ready for pickup at the counter.” into a five-second WAV file. It sounded slightly robotic, but clear.

Then Arjun had an idea he was proud of: a round trip. Feed that announcement into a speech recognition model and see whether the same sentence comes back. If it does, the recogniser works, and he can trust it with the voice notes.

He opened the WAV file in a free audio editor to trim a click at the end, exported it, loaded it in Python, and handed it to OpenAI’s Whisper model:

import soundfile as sf
from transformers import pipeline

audio, rate = sf.read("announcement_trimmed.wav")   # the editor had exported it at 44,100 Hz
asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
print(asr(audio))

The model replied, with total confidence:

{'text': " You're a fool. Hey, I knew you had this. I want a sip of coke with that paper."}

No error. No warning. Just a different sentence entirely.

Priya listened to the clip, then looked at the code. “What does a record player do if you play a record at the wrong speed?”

“It sounds like a monster. Slow and deep.”

“And if someone who’d never heard the song tried to write down the lyrics?”

Arjun winced. “They’d guess words that roughly fit the noise.” He looked back at the screen. “The model played my audio at the wrong speed?”

What a computer hears

Sound is air pressure wobbling back and forth. A microphone measures the height of that wobble many thousands of times every second and stores each measurement as one number. The number of measurements per second is the sampling rate, in hertz (Hz).

• Most speech models expect 16,000 Hz (16 kHz): plenty for voices.
• CDs and many audio editors use 44,100 Hz. Some audio models expect 48,000 Hz.

A plain list of numbers carries no label saying how fast it was recorded. Arjun’s array had 44,100 numbers per second of real time. Whisper assumed every 16,000 numbers was one second. So it heard each real second as 44,100 ÷ 16,000 ≈ 2.76 seconds: the announcement slowed down nearly three times and pitched down like a monster. The recogniser then did what the lyrics-guesser did. It wrote the most plausible English that roughly fit the sound.

Show the answer

60 × 16,000 = 960,000 numbers, for maybe 150 spoken words. The same words as text are about 200 tokens. Audio is enormously more data per idea than text, which is why audio models first compress it (next section) and why long recordings are processed in chunks.

The fix is to resample, which means recomputing the list at the rate the model expects. librosa does it while loading:

import librosa
audio, rate = librosa.load("announcement_trimmed.wav", sr=16000)   # resampled to 16 kHz, mono
print(asr(audio))

{'text': ' ordered below. Two Deloli hampers is ready for pick up at the counter.'}

Much closer. But “Order 1042” became “ordered below”, and “Diwali” became “Deloli”. The robotic voice had blurred the number, and Whisper’s vocabulary is biased toward common English words. Arjun had just learned, for free, the two places speech recognition fails most for Crumb: numbers and Indian words. That would matter a lot for order numbers and sweets like kaju katli.

Following one clip through Whisper

Speech recognition, also called speech-to-text or ASR (automatic speech recognition), turns spoken audio into written words. Whisper is a family of open models from OpenAI, trained on hundreds of thousands of hours of audio from the internet (about 680,000 hours for the first release, far more for later ones) in about 100 languages. That breadth is why it copes unusually well with accents, background noise and mixed languages. It comes in sizes from tiny (fast, rough) to large (slow, accurate); small is a good laptop default.

Priya traced the five-second announcement through it step by step.

1. The raw audio. 5.26 seconds at 16,000 Hz: 84,224 numbers, each between −1 and 1.

2. The spectrogram. Whisper does not read the raw wave. Its feature extractor (the audio equivalent of a tokenizer, which you met in Module 11) converts the wave into a log-Mel spectrogram: a picture of how much energy the sound has at each pitch, over time. Think of it as sheet music for any sound. Low notes at the bottom, high notes at the top, time running left to right, loudness as darkness. Whisper’s version has 80 pitch bands and one column every 10 milliseconds, padded or cut to exactly 30 seconds. So every input becomes a grid of 80 × 3,000 numbers, regardless of length.

3. The encoder. A transformer reads that grid, like a vision model reads patches, and builds an internal understanding of the whole 30-second window.

4. The decoder. A second transformer writes the transcript one token at a time, exactly like a chat model generating text. It starts with special tokens that set the job: <|startoftranscript|>, then a language token like <|en|> or <|hi|>, then a task token, <|transcribe|> or <|translate|>. Then the words, then <|endoftext|>.

5. Timing. On Arjun’s laptop, the five-second clip took about 2.3 seconds. Longer audio is handled in 30-second windows.

Show the answer

Something. A text generator always produces its most likely next token, and it was trained on internet audio where silence is often followed by phrases like “Thank you for watching” or “you”. Arjun tried it: eight seconds of silence came back as " you". A clip of faint hiss came back as the same unusual character, ლ, repeated over a hundred times. Whisper has no built-in way to say “nobody spoke”. You must remove silence before transcribing, or check for these tell-tale outputs.

That task token in step 4 is Whisper’s best trick for Crumb. With <|translate|>, it transcribes speech in any of its languages and writes the result directly in English. A Marathi voice note goes in; an English paragraph comes out.

The real code: from voice note to English text

First, install the pieces. Hugging Face audio pipelines decode files (MP3, OGG, M4A) using ffmpeg, a free command-line audio and video tool. Install it once: brew install ffmpeg on macOS, sudo apt install ffmpeg on Ubuntu, or winget install ffmpeg on Windows. Without it, passing a file path fails with “ffmpeg was not found”.

uv add transformers torch librosa soundfile noisereduce

Step 1: load and resample, every time.

import librosa

def load_16k(path: str):
    """Load any audio file as 16 kHz mono numbers."""
    audio, _ = librosa.load(path, sr=16000, mono=True)
    return audio

note = load_16k("baner_voice_note.ogg")
print(f"{len(note) / 16000:.1f} seconds")

Doing this yourself, instead of trusting each library’s defaults, means you always know the rate. WhatsApp voice notes are OGG files; librosa uses ffmpeg (or a similar decoder) to open them.

Step 2: transcribe and translate.

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")

result = asr(
    {"raw": note, "sampling_rate": 16000},     # say the rate out loud, so nothing is assumed
    chunk_length_s=30,                         # long notes are processed in 30-second windows
    return_timestamps=True,                    # also return when each phrase was said
    generate_kwargs={"task": "translate"},     # write the result in English
)
print(result["text"])
for chunk in result["chunks"]:
    print(chunk["timestamp"], chunk["text"])

Passing a dictionary with "raw" and "sampling_rate" makes the rate explicit. If you pass a rate other than 16,000, the pipeline will try to resample for you, which needs the extra torchaudio package; resampling yourself with librosa avoids that. Use "task": "transcribe" to keep the original language, and add "language": "marathi" (or "hindi") to generate_kwargs when Whisper guesses the language wrongly on short or mixed clips.

How well does it work on Crumb’s voice notes? For clear English and Hindi, very well. For fast Marathi mixed with English, whisper-small gets the gist but mangles names and numbers. whisper-large-v3-turbo is noticeably better if you have a GPU. Test on thirty real notes before trusting it.

Cleaning up the scooter noise

Audio denoising removes background sound (traffic, mixers, fans) while keeping the voice. Two families of tools exist:

• Classic signal processing. Estimate what the steady noise sounds like, then subtract it at each pitch. The noisereduce library does this in one line. Fast, no AI model.
• Neural enhancement models. Models on the Hub, such as the SpeechBrain enhancement models, learned to separate speech from noise. Better on messy, changing noise, but heavier.

import noisereduce as nr
import soundfile as sf

clean = nr.reduce_noise(y=note, sr=16000)
sf.write("baner_clean.wav", clean, 16000)

Here is an honest surprise. Whisper was trained on noisy internet audio, so it is already robust to noise, and aggressive denoising can add a watery, metallic texture that makes transcription worse. Denoise when a person will listen to the recording, or when you have measured that it improves your transcripts. Do not add it by reflex.

Audio embeddings: sound as meaning-numbers

For text, an embedding is a list of numbers where similar meanings sit close together (Module 07). An audio embedding is the same idea for sound: two recordings of the same voice saying similar things get similar lists; a voice and a beeping oven timer do not.

import torch
from transformers import AutoFeatureExtractor, AutoModel

extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
encoder = AutoModel.from_pretrained("facebook/wav2vec2-base")

def embed(clip):
    inputs = extractor(clip, sampling_rate=16000, return_tensors="pt")
    with torch.no_grad():
        frames = encoder(**inputs).last_hidden_state   # shape (1, frames, 768): one row per ~20 ms
    return frames.mean(dim=1)[0]                         # average the rows into one 768-number vector

similarity = torch.nn.functional.cosine_similarity

Arjun tested it on the announcement. The five-second clip produced 262 frames of 768 numbers, averaged into one vector. Then he compared versions of it using cosine similarity (1 means pointing the same way, 0 means unrelated):

Compared with the original announcement	Similarity
The same clip with loud hiss added	0.58
That noisy clip after noisereduce	0.83
A beeping oven timer	0.11

The numbers tell a story. Noise pushed the clip far from itself; denoising pulled most of the way back; a completely different sound is nowhere near. Uses for audio embeddings at Crumb: spotting the same supplier message forwarded twice, grouping delivery voice notes by type, and, with models trained specially for it, recognising which staff member is speaking. In the next lesson, a model called CLAP puts sounds and text into the same space, so you can search sounds with words.

Text-to-speech: the shop announcement

Text-to-speech (TTS) is Whisper in reverse: text in, audio out.

tts = pipeline("text-to-speech", model="facebook/mms-tts-eng")
speech = tts("Order 1042, two Diwali hampers, is ready for pickup at the counter.")

# speech["audio"] holds the numbers; speech["sampling_rate"] says how fast to play them
sf.write("order_ready.wav", speech["audio"].squeeze(), speech["sampling_rate"])
print(speech["sampling_rate"], speech["audio"].shape)   # 16000 (1, 84224)

squeeze() removes the extra outer dimension so the file has a single channel. Notice that the output carries its own sampling rate: always save it with that rate, or the announcement plays at the wrong speed, the same bug in reverse.

Meta’s MMS models cover over a thousand languages, including facebook/mms-tts-hin (Hindi) and facebook/mms-tts-mar (Marathi). Their licence, CC-BY-NC 4.0, does not allow commercial use, so they are fine for a prototype and not for Crumb’s paid product. Newer open TTS models sound more natural every few months; filter the Hub by the “Text-to-Speech” task, listen to the samples, and check each licence.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks

The voice-note reader ran for a week. Then Meera forwarded a summary that said the Kothrud store manager had written: “you”. Nothing else. The original was a twelve-second note where the manager had pressed record, got distracted by a customer, and said nothing. Whisper, given silence, produced its favourite filler.

Arjun added a simple guard: measure how loud the clip is, and skip anything that is almost all silence. In Module 17 you will use a proper voice activity detector, a small model that marks exactly where speech starts and stops.

import numpy as np

def has_speech(clip, threshold=0.01, min_fraction=0.1):
    """True if at least 10% of 20 ms frames are louder than the threshold."""
    frames = clip[: len(clip) // 320 * 320].reshape(-1, 320)   # 320 samples = 20 ms at 16 kHz
    loud = np.sqrt((frames ** 2).mean(axis=1)) > threshold
    return loud.mean() > min_fraction

The other failures to watch for:

• Wrong sampling rate. Silent and catastrophic. Resample to the model card’s rate, and save TTS output with the rate it returns.
• Numbers, names and local words. “1042” became “below”; “Diwali” became “Deloli”. Never let a transcript trigger an action, such as a refund or an order change, without a confirmation step.
• Hallucinated text on silence or noise. Trim silence; flag suspicious outputs like a lone “you” or long repeated characters.
• Code-switching. Sentences that switch between Marathi and English mid-phrase are hard for every model. Measure on real notes.
• Privacy. Voice is personal data. Transcribing on your own machine keeps it private; sending it to an API means a third party processes it.
• Non-commercial voices. Check TTS licences before a customer hears them.

Try it yourself

Record a 20-second voice note on your phone in a language other than English (or ask a friend to), ideally with some background noise. Then:

1. Load it with load_16k and transcribe it with "task": "transcribe".
2. Run it again with "task": "translate".
3. Turn the English translation into speech with MMS TTS and save it.
4. Round trip: transcribe your TTS file. What changed?

Show the answer

Step 1 should print the words in the original language and script (Devanagari for Hindi or Marathi). Step 2 should print an English version, usually a fair paraphrase rather than a word-for-word translation. Step 3 gives a clear, slightly robotic English voice. Step 4 usually returns most of the sentence, with errors on numbers, names and non-English words, just as “Order 1042” became “ordered below”. If step 1 prints the wrong language, add "language": ... to generate_kwargs. If it prints nonsense, check the sampling rate first.

Common mistakes

• Passing raw arrays without stating the rate. Load with sr=16000 and pass {"raw": ..., "sampling_rate": 16000}.
• Transcribing long audio without chunking. Whisper reads 30 seconds at a time; use chunk_length_s=30.
• Trusting transcripts of silence. Detect speech first; treat a lone “you” as a warning sign.
• Denoising by reflex. Measure whether it helps your transcripts.
• Shipping a non-commercial voice. Read the TTS model’s licence.

Recap

• Audio is a long list of numbers; the sampling rate says how many per second. A mismatch produces confident nonsense, not an error.
• Whisper turns audio into a spectrogram, encodes it, and decodes text with special tokens for language and task; "task": "translate" writes English directly.
• Whisper is a text generator underneath, so silence and noise produce invented text. Check for speech first.
• Denoising helps people listen; it does not always help Whisper. Audio embeddings turn clips into vectors you can compare.
• pipeline("text-to-speech") returns audio plus its own sampling rate; save with that rate and check the licence.

On Friday, Meera’s phone showed something new: each voice note had a two-line English summary underneath it, and the Koregaon Park store speaker said “Order 1050 is ready for pickup” in a calm voice every few minutes.

“It’s like a record player and a stenographer,” Arjun explained. “The recording is just numbers, so we first make sure it plays at the right speed. Then Whisper draws a picture of the sound, reads it, and writes down the words, in English if we ask. It’s good, but it guesses at numbers and Indian words, and if nobody speaks it still writes something, so we check for silence first.”

Then Meera brought up the next problem. The recipe archive held 6,000 photos, and she wanted them tagged: eggless, festive, birthday, savoury. No classifier on the Hub knew any of those labels, and she was not going to pay someone to label 6,000 photos by hand.

“You don’t need a model that already knows your labels,” Priya said. “You need one that can read them.”

Go deeper

• Hugging Face documentation (audio tasks, Whisper, text-to-speech): https://huggingface.co/docs
• Hugging Face LLM course: https://huggingface.co/learn/llm-course