Introduction

Voice activity detection is the task of deciding whether a short chunk of audio contains speech or not. With NumPy and SciPy, you can build a solid baseline detector by loading audio into arrays, cutting it into short frames, and computing simple features such as energy and zero-crossing rate. That will not beat a production speech model, but it is a practical way to understand the signal-processing pipeline.

A good first step is to separate the problem into three stages: read the waveform, compute frame-level features, and classify each frame as speech-like or not. Once that pipeline works, you can improve it with better features or a trained model.

Read and Normalize the Audio

SciPy can read WAV files directly, and NumPy makes it easy to convert the samples into floating-point arrays.

1import numpy as np
2from scipy.io import wavfile
3
4sample_rate, audio = wavfile.read("sample.wav")
5
6if audio.ndim == 2:
7    audio = audio.mean(axis=1)
8
9audio = audio.astype(np.float32)
10audio /= np.max(np.abs(audio)) + 1e-8
11
12print(sample_rate, audio.shape)

This code converts stereo to mono and scales the waveform into a stable range. That normalization step makes feature thresholds easier to reason about.

Split the Signal Into Frames

Speech detection is usually done on short windows rather than on the whole clip at once. A common frame size is 20 to 30 milliseconds with some overlap.

1import numpy as np
2
3frame_size = int(0.025 * sample_rate)
4hop_size = int(0.010 * sample_rate)
5
6frames = []
7for start in range(0, len(audio) - frame_size + 1, hop_size):
8    frame = audio[start:start + frame_size]
9    frames.append(frame)
10
11frames = np.stack(frames)
12print(frames.shape)

Now each row of frames is a short slice of the signal. That is the unit we will classify.

Compute Simple VAD Features

Two lightweight baseline features are short-time energy and zero-crossing rate.

1import numpy as np
2
3def short_time_energy(frame):
4    return np.mean(frame ** 2)
5
6def zero_crossing_rate(frame):
7    signs = np.sign(frame)
8    return np.mean(np.abs(np.diff(signs)) > 0)
9
10energies = np.array([short_time_energy(frame) for frame in frames])
11zcrs = np.array([zero_crossing_rate(frame) for frame in frames])

Energy helps detect whether a frame contains enough signal to matter. Zero-crossing rate helps distinguish speech-like structure from some kinds of low-frequency hum or steady noise.

Build a Simple Speech/Non-Speech Classifier

A baseline rule-based classifier can combine those features into a speech mask.

1energy_threshold = np.percentile(energies, 60)
2zcr_low = 0.02
3zcr_high = 0.25
4
5speech_mask = (
6    (energies > energy_threshold) &
7    (zcrs > zcr_low) &
8    (zcrs < zcr_high)
9)
10
11print(speech_mask[:20])

This is intentionally simple. The thresholds will vary by microphone quality, background noise, and recording level, but it demonstrates the structure of a working detector.

You can also turn the frame labels into time ranges:

1for i, is_speech in enumerate(speech_mask):
2    if is_speech:
3        start_time = i * hop_size / sample_rate
4        end_time = (i * hop_size + frame_size) / sample_rate
5        print(f"speech from {start_time:.2f}s to {end_time:.2f}s")

Add Spectral Features for Better Classification

If you want a slightly richer classifier, SciPy can help compute frequency-domain features.

1from scipy.fft import rfft
2
3def spectral_centroid(frame, sample_rate):
4    spectrum = np.abs(rfft(frame))
5    freqs = np.linspace(0, sample_rate / 2, len(spectrum))
6    return np.sum(freqs * spectrum) / (np.sum(spectrum) + 1e-8)
7
8centroids = np.array([
9    spectral_centroid(frame, sample_rate) for frame in frames
10])

Speech often occupies a different spectral region than silence or simple background noise. Even without a trained machine-learning model, this extra feature can make threshold-based classification more stable.

Know the Limits of a NumPy/SciPy Baseline

This approach is good for learning and for simple clean recordings. It is not robust enough for difficult environments with music, overlapping speakers, strong background noise, or far-field microphones.

For higher accuracy, the usual next steps are:

• smoother post-processing across adjacent frames
• better features such as MFCC-like descriptors
• a trained classifier on labeled audio
• a specialized VAD model instead of static thresholds

Still, the NumPy and SciPy pipeline is valuable because it shows the mechanics clearly and gives you a strong baseline to compare against.

Common Pitfalls

The most common mistake is applying one fixed threshold to raw audio without first normalizing the signal. The same threshold then behaves differently across recordings.

Another issue is using frames that are too large, which makes speech boundaries feel sluggish, or too small, which makes the detector noisy.

A third problem is treating this simple detector as production-grade speech recognition infrastructure. It is a baseline VAD and classifier, not a full speech system.

Summary

• Load audio into NumPy arrays and normalize it before feature extraction.
• Split the waveform into short overlapping frames.
• Use short-time energy and zero-crossing rate as simple baseline VAD features.
• Add spectral features such as centroid when you need a stronger classifier.
• Treat a NumPy/SciPy detector as a baseline pipeline that can later be upgraded with trained models.