User Guide

The Arabic Audio Intelligence System is a deep-learning-powered platform that converts spoken Arabic audio into searchable, analysable, and summarised text. It was built as an academic deliverable addressing the real-world challenge of processing Arabic speech, a language with complex morphology, optional diacritics, and dozens of spoken dialects, entirely with open-source neural networks.

Rather than a single-purpose transcription tool, the system implements a fully integrated pipeline: raw audio goes in, and structured knowledge (transcripts, speaker timelines, emotional states, summaries, indexed notes) comes out. Every component is independently accessible through a module-driven interface. The system supports both fully local inference (no internet required after initial model download) and a cloud-accelerated mode via the OpenAI Whisper API for instant transcription.

The project satisfies every item in the course specification, including all optional advanced tasks (speaker diarization, emotion detection, text summarization, and a voice chat messenger with search capabilities). Four ASR models are available: OpenAI Whisper API (cloud), Whisper Local (HuggingFace), Wav2Vec2 (HuggingFace), and a custom CNN+LSTM trained from scratch.

The architecture is split into two independent layers: a Python FastAPI backend at port 8000 that handles all ML computation, and a Next.js 16 frontend at port 3000 that renders the interface. The two communicate via a stateless REST API over HTTP, meaning the backend can be hosted on a GPU server anywhere while the frontend can be served from a CDN.

Audio Input  (.wav / .mp3 / .flac / .ogg)
       │
       ▼
 ┌─────────────────────────────────────────┐
 │           Audio Preprocessing           │
 │   Resample to 16kHz mono                │
 │   Segment if > 30s (Whisper chunking)   │
 │   MFCC / Log-Mel extraction             │
 └───────────────────┬─────────────────────┘
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
   ┌────────────┐     ┌──────────────────────┐
   │ CNN + LSTM │     │  Whisper / Wav2Vec2  │
   │ (custom)   │     │  (HuggingFace)       │
   └──────┬─────┘     └──────────┬───────────┘
          └──────────┬───────────┘
                     ▼
          ┌──────────────────────┐
          │   Text Transcript    │
          └───────┬──────────────┘
                  │
       ┌──────────┼──────────────┐
       ▼          ▼              ▼
 ┌──────────┐ ┌────────┐ ┌────────────────┐
 │Summarizer│ │Emotion │ │Speaker Diarize │
 │mT5/TF-IDF│ │Wav2Vec2│ │PyAnnote/ECAPA  │
 └──────────┘ └────────┘ └────────────────┘
       │          │              │
       └──────────┴──────────────┘
                  │
                  ▼
        ┌─────────────────────┐
        │    Search Engine    │
        │  FAISS + sentence-  │
        │  transformers       │
        └─────────────────────┘

Backend API surface

Frontend routing

Each feature is a Next.js App Router page under src/app/. All API calls are centralised in src/lib/api.ts. The dynamic side navbar uses Framer Motion layout animations to transition between a horizontal bottom dock (home page) and a vertical sidebar (all other pages), animated as a fluid spring, not a CSS snap.

OpenAI Whisper API (Cloud)

The fastest option: sends audio to OpenAI's servers for transcription using the latest Whisper large model. No GPU or local model download needed. Requires an OPENAI_API_KEY in the .env file. The API handles any audio format and returns Arabic text directly. Ideal for quick demos or machines without GPUs.

OpenAI Whisper (Local)

Whisper is an encoder-decoder Transformer trained on 680,000 hours of multilingual audio. The audio encoder is a convolutional stem followed by a stack of Transformer blocks that produces a context-rich embedding of the log-mel spectrogram. The decoder auto-regressively generates BPE tokens using cross-attention over those encoder outputs. It natively supports Arabic and handles diacritics, code-switching, and noise robustly. We use whisper-small (244M parameters) locally; the model auto-downloads from HuggingFace on first use.

Wav2Vec 2.0: Arabic fine-tune

Wav2Vec 2.0 (Meta AI) learns speech representations directly from raw waveforms. Its convolutional feature extractor downsamples the audio, and a 24-layer Transformer encoder is pre-trained with a masked contrastive loss (like BERT for speech). A linear CTC head is then fine-tuned on labelled data. We load facebook/wav2vec2-large-xlsr-53-arabic, the XLSR-53 model fine-tuned specifically on the Arabic Common Voice split. Because CTC decoding is non-autoregressive it is roughly 2× faster than Whisper at inference.

CNN + LSTM: Custom Academic Model

Built from scratch to satisfy the course requirement for a deep-learning model designed and trained by students. It operates on MFCC features extracted from audio. Two 2D convolutional layers with batch normalisation extract spectro-temporal patterns from the feature map. The output is reshaped and fed into two bidirectional LSTM layers that model temporal context across frames. A fully-connected layer followed by a softmax over the character vocabulary produces per-frame posteriors; CTC loss handles alignment-free training.

Input waveform  →  librosa MFCC  →  (128, T) feature map
     ↓
Conv2D(32, 3×3, padding=1) → BN → ReLU
Conv2D(64, 3×3, padding=1) → BN → ReLU → MaxPool2D(2,2)
     ↓
Reshape  →  (T//2,  64 × 64)   [batch × time × features]
     ↓
BiLSTM(256, num_layers=2, dropout=0.3)
     ↓
Linear(vocab_size)   →   log_softmax
     ↓
CTC Loss  ←→  reference transcript

PyAnnote Audio: Speaker Diarization

PyAnnote is a speaker diarization toolkit built on PyTorch. The pyannote/speaker-diarization-3.1 pipeline chains voice-activity detection (a segmentation model), speaker embedding extraction (ECAPA-TDNN d-vectors), and spectral clustering into a single end-to-end pipeline. It outputs a speaker-labeled timeline with RTTM-compatible start/end timestamps.

Wav2Vec 2.0: Emotion Classification

A Wav2Vec 2.0 base checkpoint fine-tuned for speech emotion recognition. A mean-pooling layer followed by a 4-class linear head is trained on prosodic emotional datasets. The four classes are happy, angry, neutral, sad. The model outputs a probability distribution over all four, making it interpretable beyond a single predicted label.

mT5 / TF-IDF: Summarization

Two modes are offered. Extractive summarization uses TF-IDF sentence scoring: sentences are ranked by weighted term frequency and the top‑N are returned verbatim; fast, no GPU required. Abstractive summarization runs a multilingual sequence-to-sequence model (Helsinki-NLP or an Arabic T5 variant) to generate a free-form condensed paraphrase of the input text.

The CNN+LSTM model is a custom-trained checkpoint that must be present at outputs/checkpoints/best_model.pt before selecting it in the Transcribe page. This section covers three paths: using the pre-trained checkpoint from the repo, training your own, or fixing the backend if you see name 'T' is not defined.

Option 1 — Use the checkpoint already in the repo (recommended)

The checkpoint is committed to the repository via Git LFS. After cloning, it will be at outputs/checkpoints/best_model.pt (~55 MB). If LFS was not installed when you cloned, run the commands below to pull it down:

# 1. Install Git LFS (one-time):
git lfs install

# 2. Pull the checkpoint (if you already cloned without LFS):
git lfs pull

# 3. Verify the file exists and is the correct size:
ls -lh outputs/checkpoints/best_model.pt
# expected: 55MB

Option 2 — Set up from GitHub (full fresh install)

# ── Prerequisites: Python 3.9+, Node 18+, git ──

# 1. Clone the repository
git clone https://github.com/Moaz2010/arabic-asr-project.git
cd arabic-asr-project

# 2. Create and activate a virtual environment
python -m venv venv
venv\Scripts\activate         # Windows
# source venv/bin/activate      # macOS / Linux

# 3. Install Python dependencies
pip install -r requirements.txt
pip install -r backend/requirements.txt

# 4. Start the backend  (auto-finds a free port 8000–8019)
python backend/main.py
# → Uvicorn running on http://0.0.0.0:8001  (or whichever port is free)
# → Copy the NEXT_PUBLIC_API_URL value it prints

# 5. Set the frontend env variable
echo NEXT_PUBLIC_API_URL=http://localhost:8001 > frontend\.env.local

# 6. Install frontend deps and start Next.js
cd frontend
npm install
npm run dev
# → http://localhost:3000

# 7. Open the Transcribe page, select CNN+LSTM, upload a WAV file

Connecting the config

The backend loads the checkpoint path from configs/config.yaml. If your checkpoint is saved elsewhere, update this file before starting the backend:

# configs/config.yaml
models:
  cnn_lstm:
    checkpoint: "outputs/checkpoints/best_model.pt"
    vocab_size: 50
    n_mfcc: 128
    hidden_size: 256
    num_layers: 2

Troubleshooting: "name 'T' is not defined"

This error means the backend is running a stale cached version of audio_utils.py that still uses the old torchaudio.transforms as T alias. The fix has been applied in the current codebase — you just need to make sure Python reloads from disk.

# Quick fix — clear all pycache and restart:

# Windows PowerShell:
Get-ChildItem -Recurse -Filter __pycache__ | Remove-Item -Recurse -Force

# macOS / Linux:
find . -type d -name __pycache__ -exec rm -rf {} +

# Then restart:
python backend/main.py

Option 3 — Train your own checkpoint

See the Training Notebook section of this guide (Training Notebook tab in the sidebar). The short path:

# 1. Open the training notebook in Colab or VS Code:
jupyter lab notebooks/02_cnn_lstm_training.ipynb

# 2. Add your Mozilla Data Collective API key in Colab Secrets:
#    Name: MDC_API_KEY  →  your key from datacollective.mozillafoundation.org

# 3. Run all cells (≈35 min per epoch on a Colab T4 GPU)
#    The notebook saves checkpoints/best_model.pt on every val-WER improvement

# 4. Copy the trained checkpoint to the expected path:
cp checkpoints/best_model.pt outputs/checkpoints/best_model.pt

# 5. Restart the backend — it will auto-load the new checkpoint
python backend/main.py

Arabic ASR is data-starved compared to English. All available high-quality labelled Arabic speech resources total a few hundred hours, versus millions of hours for English. The following datasets are used or referenced in this project:

Data preprocessing

Every audio file is resampled to 16 kHz mono before any model sees it. For the CNN+LSTM training pipeline, 128 MFCC coefficients are extracted per frame (hop = 512, window = 2048). Feature maps are normalised to zero-mean unit-variance per sample. During training, SpecAugment is applied: random time masks (T=50 frames) and frequency masks (F=40 bins) are zeroed out to improve noise robustness.

Neural models cannot consume raw audio bytes directly; they need structured numerical representations. Three representations are used across the different models:

MFCC (CNN+LSTM)

Mel-Frequency Cepstral Coefficients approximate the human auditory system. A Short-Time Fourier Transform (STFT) extracts frequency content over overlapping frames, the frequency axis is warped to the perceptual Mel scale, log energies are computed, and finally a Discrete Cosine Transform produces compact coefficients. The result is a 2D feature map (frequency × time) that the convolutional layers treat like an image.

import librosa, numpy as np

y, sr = librosa.load("audio.wav", sr=16000, mono=True)

mfcc = librosa.feature.mfcc(
    y=y, sr=sr, n_mfcc=128, n_fft=2048, hop_length=512
)  # shape: (128, T)

# Normalise per sample
mfcc = (mfcc - mfcc.mean()) / (mfcc.std() + 1e-8)

Log-Mel Spectrogram (Whisper)

Whisper uses 80 Mel filter-bank channels with 25ms frames shifted by 10ms. The spectrogram is computed, log-scaled, normalised to [–1, 1], and chunked into 30-second blocks. The convolutional stem of Whisper's encoder processes these chunks in parallel before Transformer layers attend across frames.

Raw Waveform (Wav2Vec 2.0)

Wav2Vec 2.0 consumes normalised 16kHz waveform values directly. Its first stage, a stack of temporal 1D convolutions, acts as a learned feature extractor that replaces hand-crafted MFCCs. This end-to-end approach captures fine-grained acoustic detail that fixed feature extractors may discard.

The custom model is trained in src/train.py and a complete Colab-ready walkthrough is provided in notebooks/02_cnn_lstm_training.ipynb. The notebook covers data loading, feature extraction, model construction, training loop, and evaluation in sequential cells.

CTC Loss: no forced alignment needed

CTC (Connectionist Temporal Classification) allows training on (audio, transcript) pairs without frame-level phoneme annotations. It marginalises over every possible alignment between the output sequence and the target label by summing probabilities across all valid paths (including repetitions and blank tokens). This makes large-scale ASR training practical.

import torch.nn as nn

criterion = nn.CTCLoss(blank=0, zero_infinity=True)

# log_probs: (T, N, C)  — time × batch × vocab_size
# targets, input_lengths, target_lengths
loss = criterion(log_probs, targets, input_lengths, target_lengths)

Configuration

Word Error Rate (WER)

WER is the primary ASR metric. It computes the minimum edit distance between the model's hypothesis and the ground-truth reference at the word level, then normalises by the total number of reference words. Lower is better; perfect transcription = 0%.

WER = (S + D + I) / N

  S = Substitutions  (wrong word predicted)
  D = Deletions      (reference word missing from hypothesis)
  I = Insertions     (extra word in hypothesis)
  N = Total reference words

Example —
  Reference:   أنا   أُحِبّ   العِلْم   كَثيرًا
  Hypothesis:  أنا   أحب     العلوم              ← 3 errors (S, S, D)
  WER = 3/4 = 75%

Character Error Rate (CER)

CER applies the same formula character-by-character. For Arabic, CER is often more informative than WER because a single morphological suffix difference counts as one word substitution in WER but only a few character errors in CER, giving credit for partially-correct words.

Benchmark comparison

Arabic is a root-and-pattern language. Grammatical information (tense, person, number, gender, definiteness) is encoded as inflectional patterns applied to three- or four-letter roots, producing a vast number of unique surface forms. A single word like وسيكتبونها(wa-sa-yaktubu-na-hā, “and they will write it”) encodes five pieces of information in one token. This dramatically inflates the effective vocabulary, making any statistical model harder to train compared to analytic languages like English.

Diacritics typically omitted

Written Arabic omits short vowel marks (ḥarakāt) in most everyday text. The word كتب is ambiguous: kataba (he wrote), kutub (books), kutiba (it was written). Models must infer the correct reading entirely from context, a challenge that compounds with the morphological ambiguity above.

Diglossia and dialectal variation

The Arabic-speaking world has 22 countries with spoken dialects that differ substantially in phonology, lexicon, and grammar. Egyptian, Levantine, Gulf, and Maghrebi Arabic are all linguistically Arabic but differ the way Portuguese differs from Spanish in some dimensions. Most labelled datasets focus on Modern Standard Arabic (MSA), which is the formal written register but rarely spoken in casual conversation, creating a systematic domain mismatch for real-world deployment.

Data scarcity

As of 2025 the best publicly available labelled Arabic speech corpus (Mozilla Common Voice Arabic) contains roughly 25 hours of validated recordings. English ASR models are trained on datasets that are 4 to 5 orders of magnitude larger. This scarcity is the primary reason transfer learning from massively multilingual models (Whisper, XLSR) dramatically outperforms training-from-scratch approaches on limited data, and is the key academic motivation for including both the custom CNN+LSTM and the pre-trained models in this project.

The notebook at notebooks/02_cnn_lstm_training.ipynb is a self-contained walkthrough of the custom CNN+LSTM model, from raw dataset to trained checkpoint with evaluation metrics. It is designed to run in Google Colab (free T4 GPU) or any local Jupyter environment with CUDA available.

Environment setup

The first cells install everything needed and mount Google Drive if running in Colab. No manual package management is required beyond running the cells in order.

# If running locally, activate your virtual environment first:
source venv/bin/activate        # macOS/Linux
venv\Scripts\activate           # Windows

# Then launch Jupyter:
jupyter lab notebooks/02_cnn_lstm_training.ipynb

# Or open in VS Code → right-click → Open With → Jupyter Notebook

Cell-by-cell structure

Expected runtime

On a Colab T4 GPU, one full epoch over the Common Voice Arabic validated set (≈25h audio) takes roughly 35–45 minutes. The notebook defaults to 5 epochs for a quick demo run. Set MAX_EPOCHS = 30 and enable early stopping for production-quality training. A pre-trained checkpoint is included in checkpoints/ so the notebook can also be run in evaluation-only mode by skipping cells 4–5.

Connecting the checkpoint to the backend

Once training is complete, copy the saved checkpoint path into configs/config.yaml under the cnn_lstm.checkpoint key. The FastAPI backend reads this config at startup and loads the model automatically; no code changes required.

# configs/config.yaml
models:
  cnn_lstm:
    checkpoint: "checkpoints/best_model.pt"
    vocab_size: 50
    n_mfcc: 128
    hidden_size: 256
    num_layers: 2

Every deliverable is present in the repository. The table below maps each item to its exact location. The demo interface (deliverable 6) can be started in under two minutes; see the Local Setup tab for the full command sequence.

Verifying the demo runs correctly

With the backend running on port 8000 and the frontend on port 3000, the indicator dot in the side-nav will turn green. The same status is reflected as ONLINE in the home screen header. If the dot is red, the most common causes are: backend not started, a missing Python dependency, or a firewall blocking the port.

Running the full pipeline end-to-end

# Terminal 1 — Backend
cd arabic-asr-project
source venv/bin/activate
python backend/main.py
# → Uvicorn running on http://0.0.0.0:8000

# Terminal 2 — Frontend
cd arabic-asr-project/frontend
npm run dev
# → Next.js ready on http://localhost:3000

# Verify API is healthy:
curl http://localhost:8000/api/health
# → {"status":"ok","models_loaded":["whisper","wav2vec2","cnn_lstm"]}

Task coverage at a glance