A High-Level Look#

Let’s begin by looking at the model as a single black box. In a machine translation application, it would take a sentence in one language, and output its translation in another.

Popping open that Optimus Prime goodness, we see an encoding component, a decoding component, and connections between them.

The encoding component is a stack of encoders (In its original paper Attention is All You Need, 6 encoders and decoders each (12 in total) were stacked on top of each other - there’s nothing magical about the number 6, one can definitely experiment with other arrangements). The decoding component is a stack of decoders of the same number.

The encoders are all identical in structure (yet they do not share weights). Each one is broken down into two sub-layers:

The encoder’s inputs first flow through a self-attention layer - a layer that helps the encoder look at other words in the input sentence as it encodes a specific word. We’ll look closer at self-attention later in this post.

The outputs of the self-attention layer are fed to a feed-forward neural network. The exact same feed-forward network is independently applied to each position.

The decoder has both those layers, but between them is an attention layer that helps the decoder focus on relevant parts of the input sentence (similar to what attention does in seq2seq models).

Bringing The Tensors Into The Picture#

Now that we’ve seen the major components of the model, let’s start to look at the various vectors/tensors and how they flow between these components to turn the input of a trained model into an output.

As is the case in NLP applications in general, we begin by turning each input word into a vector using embedding algorithm.

Each word is embedded into a vector of size 512. We’ll represent those vectors with these simple boxes.

The embedding only happens in the bottom-most encoder. The abstraction that is common to all the encoders is that they receive a list of vectors each of the size 512 – In the bottom encoder that would be the word embeddings, but in other encoders, it would be the output of the encoder that’s directly below. The size of this list is hyperparameter we can set – basically it would be the length of the longest sentence in our training dataset.

After embedding the words in our input sequence, each of them flows through each of the two layers of the encoder:

Here we begin to see one key property of the Transformer, which is that the word in each position flows through its own path in the encoder. There are dependencies between these paths in the self-attention layer. The feed-forward layer, however, does not have those dependencies and thus the various paths can be executed in parallel while flowing through the feed-forward layer.

Next, we’ll switch up the example to a shorter sentence and look at what happens in each sub-layer of the encoder.

Encoding#

As we’ve mentioned already, an encoder receives a list of vectors as input. It processes this list by passing these vectors into a ‘self-attention’ layer, then into a feed-forward neural network, then sends out the output upwards to the next encoder.

The word at each position passes through a self-attention process. Then, they each pass through a feed-forward neural network - the exact same network with each vector flowing through it separately.

Self-Attention at a High Level#

Let’s say the following sentence is an input sentence we want to translate:

1
The animal didn't cross the street because it was too tired

What does “it” in this sentence refer to? Is it referring to the street or to the animal? It’s a simple question to a human, but not as simple to an algorithm. When the model is processing the word “it”, self-attention allows it to associate “it” with “animal”, because as the model processes each word (each position in the input sequence), self-attention allows it to look at other positions in the input sequence for clues that can help lead to a better encoding for this word.

Recall that with RNNs, maintaining a hidden state allows an RNN to incorporate its representation of previous words/vectors it has processed with the current one it’s processing. Self-attention is the method the Transformer uses to bake the “understanding” of other relevant words into the one we’re currently processing.

As we are encoding the word “it” in encoder #5 (the top encoder in the stack), part of the attention mechanism was focusing on “The Animal”, and baked a part of its representation into the encoding of “it”.

Self-Attention in Detail#

Let’s first look at how to calculate self-attention using vectors in steps, then proceed to look at how it’s actually implemented – using matrices.

1. Create 3 vectors from each of the encoder’s input vectors (in this case, the embedding of each word):

   1. a Query vector
   2. a Key vector, and
   3. a Value vector

   These vectors are created by multiplying the embedding by three matrices from the training process and are smaller in dimension than the embedding vector. Their dimensionality is 64, while the embedding and encoder input/output vectors have dimensionality of 512. They don’t have to be smaller though; this is an architecture choice to make the computation of multiheaded attention (mostly) constant.

   

   Multiplying $X_1$ by the $W^Q$ weight matrix produces $q_1$, the “query” vector associated with that word. We end up creating a “query”, a “key”, and a “value” projection of each word in the input sentence.

   What are the “query”, “key”, and “value” vectors?

   They’re abstractions that are useful for calculating and thinking about attention. We will make them clear in the following discussions

2. Compute scores of every word in a sentence with respect to one particular word. For instance, when we calculate the self-attention for the first word in this example, “Thinking”, we need to score each word of the input sentence against this word. The score determines how much focus to place on other parts of the input sentence as we encode a word, i.e. “Thinking”, at a certain position.

   The score is calculated by taking the dot product of the query vector with the key vectors of each word we’re scoring. For example, in our 2-word sentence, if we’re processing the self-attention for the word “Thinking”, there would be 2 scores:

   1. the dot product of $q_1$ and $k_1$: $q_1 \cdot k_1$
   2. the dot product of $q_1$ and $k_2$: $q_1 \cdot k_2$

   

3. Divide the scores by 8 (the square root of the dimension of the key vectors used in the paper – 64. This leads to having more stable gradients. There could be other possible values, but this is the default)

4. Pass the result through a softmax operation, which normalizes the scores so they’re all positive and add up to 1

   

   This softmax score determines how much each word will be expressed at this position. It would be obvious that a word at its own position has the highest softmax score (such as 0.88 for “Thinking” itself), but sometimes it’s useful to attend to another word that is relevant to the current word.

5. Multiply each value vector by the softmax score (in preparation to sum them up). The intuition here is to keep intact the values of the word(s) we want to focus on, and drown-out irrelevant words (by multiplying them by tiny numbers like 0.001, for example).

6. Sum up the weighted value vectors. This produces the output of the self-attention layer at this position (for the first word, “Thinking”).

   

That concludes the self-attention calculation. The resulting vector is one we can send along to the feed-forward neural network. In the actual implementation, however, this calculation is done in matrix form for faster processing in the following way:

To calculate the Query, Key, and Value matrices, we pack our embeddings into a matrix $X$, and multiplying it by the weight matrices we’ve trained ($W^Q$, $W^K$, $W^V$):

Every row in the $X$ matrix corresponds to a word in the input sentence. We again see the difference in size of the embedding vector (512, or 4 boxes in the figure), and the $q$/$k$/$v$ vectors (64, or 3 boxes in the figure above)

Then we, since we are dealing with matrices, condense the rest of the vector steps into one formula to calculate the outputs of the self-attention layer:

The self-attention calculation in matrix form

1
import numpy as np
2
import matplotlib.pyplot as plt
3

4
# https://www.tensorflow.org/tutorials/text/transformer
5
def get_angles(pos, i, d_model):
6
    angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))
7
    return pos * angle_rates
8

9
def positional_encoding(position, d_model):
10
    angle_rads = get_angles(
11
            np.arange(position)[:, np.newaxis],
12
            np.arange(d_model)[np.newaxis, :],
13
            d_model
14
    )
15

16
  # apply sin to even indices in the array; 2i
17
  angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])
18

19
  # apply cos to odd indices in the array; 2i+1
20
  angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])
21

22
  pos_encoding = angle_rads[np.newaxis, ...]
23

24
  return pos_encoding
25

26
tokens = 10
27
dimensions = 64
28

29
pos_encoding = positional_encoding(tokens, dimensions)
30
print (pos_encoding.shape)
31

32
plt.figure(figsize=(12,8))
33
plt.pcolormesh(pos_encoding[0], cmap='viridis')
34
plt.xlabel('Embedding Dimensions')
35
plt.xlim((0, dimensions))
36
plt.ylim((tokens,0))
37
plt.ylabel('Token Position')
38
plt.colorbar()
39
plt.show()

Preparing Hugging Face Datasets#

To fine-tune a pre-trained model with Hugging Face Transformer, we must train with data format that works with Hugging Face ecosystem. This is when Hugging Face Datasets comes into play. The essential code for using Hugging Face Datasets to generate such data is the following:

1
from datasets import Dataset, Audio
2

3
DATASET_PATH = Path("my/dataset/output/path")
4

5
valid_records = []
6

7
for autio_path, transcript in zip(autio_paths, transcripts):
8
    valid_records.append({
9
        "audio": str(audio_path),
10
        "sentence": transcript
11
    })
12

13
hf_dataset = Dataset.from_list(valid_records)
14
hf_dataset = hf_dataset.cast_column("audio", Audio(sampling_rate=16000))
15
hf_dataset.save_to_disk(DATASET_PATH)

We can imagine

• autio_paths is a list of file system paths pointing to a collection of raw audio files for training
• transcripts is another list of same size containing ordered transcripts of each raw audio files

TIP

hf_dataset.cast_column() casts the ‘audio’ column to Audio feature which handles loading and resampling on the fly

Load a Pre-trained Model#

To begin training, we need to load both the model and its corresponding processor. The processor handles 2 tasks:

1. converting raw audio into the log-Mel spectrograms the model expects, and
2. decoding the model’s output tokens back into human-readable German text.

and is loaded with

1
from transformers import WhisperProcessor
2

3
# We use the 'small' variant for a balance between performance and speed
4
model_id = "openai/whisper-small"
5

6
processor = WhisperProcessor.from_pretrained(model_id, language="german", task="transcribe")

Note that the processor is configured specifically for German transcription

To load pre-trained whisper model:

1
from transformers import WhisperForConditionalGeneration
2

3
model = WhisperForConditionalGeneration.from_pretrained(model_id)

TIP

Whisper is designed to be a “zero-shot” model, meaning it usually tries to guess what language is being spoken and whether it should translate it to English or just transcribe it. Since Whisper is a multilingual model capable of many tasks (like translating or identifying languages), there needs some settings to lock it into a specific behavior sometimes after the model is loaded. Two most frequently used settings are

1
# Force the model to generate German text during training and inference
2
model.config.forced_decoder_ids = None
3
model.config.suppress_tokens = []

If we are training a model for a specific task like transcription on one target language only, we would usually use

1
model.config.forced_decoder_ids = processor.get_decoder_prompt_ids(language="german", task="transcribe")

When we set forced_decoder_ids this way, we are forcing the model to output German text. Without this, the model might occasionally hallucinate and switch to English or another language if the audio is fuzzy. By using the processor to get these IDs, we are telling the model: “Your target language is German, and your task is transcription”. But for a general purpose fine-tuning, we would simply give it None

By default, the pre-trained Whisper model has a list of “suppressed tokens” - specific symbols or characters it is instructed never to say. This often includes special formatting tokens or certain technical markers that were restricted during its initial training to keep its output clean.

Setting suppress_tokens = [] clears that list. In the context of fine-tuning, we want the model to have full access to its entire vocabulary. We are essentially telling the model: “Do not ignore any parts of your vocabulary; you have permission to use all tokens necessary to accurately represent the German training data”.

Train#

Training involves loading a German dataset (such as Mozilla Data Collective), preprocessing the audio to 16kHz, and using the Seq2SeqTrainer. This component manages the training loop, evaluation, and checkpoint saving.

1. Load dataset we just prepared

   1
   from datasets import load_from_disk
   2
   
   3
   dataset = load_from_disk(DATASET_PATH)
   

2. Define training parameters

   1
   from transformers import Seq2SeqTrainingArguments
   2
   
   3
   training_args = Seq2SeqTrainingArguments(
   4
       output_dir="./whisper-german-asr",
   5
       per_device_train_batch_size=16,
   6
       gradient_accumulation_steps=1,
   7
       learning_rate=1e-5,
   8
       warmup_steps=500,
   9
       max_steps=4000,
   10
       fp16=True,
   11
       predict_with_generate=True,
   12
       generation_max_length=225,
   13
       save_steps=1000,
   14
       eval_steps=1000,
   15
       report_to=["tensorboard"],
   16
   )
   

3. Define preprocessing function to prepare audio and text labels. Initialize the trainer

   1
   from transformers import Seq2SeqTrainer, DataCollatorForSeq2Seq
   2
   
   3
   def prepare_dataset(batch):
   4
       audio = batch["audio"]
   5
       batch["input_features"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]
   6
       batch["labels"] = processor.tokenizer(batch["sentence"]).input_ids
   7
       return batch
   8
   
   9
   trainer = Seq2SeqTrainer(
   10
       args=training_args,
   11
       model=model,
   12
       train_dataset=dataset.map(prepare_dataset),
   13
       data_collator=DataCollatorForSeq2Seq(processor.tokenizer, model=model, padding=True),
   14
       tokenizer=processor.feature_extractor,
   15
   )
   

4. Start the training process

   1
   trainer.train()
   

Inference#

To deploy the model to a production-ready webservice that adapts to both CPU and GPU hardware ready for inferencing, we load the model we just trained.

We first define the model path and make things work on both CPU and GPU machine:

1
MODEL_PATH = "./whisper-german-production"
2
DEVICE = "cuda:0" if torch.cuda.is_available() else "cpu"
3
BATCH_SIZE = 16 if DEVICE != "cpu" else 1

Then we load the Hugging Face Pipeline for ASR:

1
pipe = pipeline(
2
    "automatic-speech-recognition",
3
    model=MODEL_PATH,
4
    tokenizer=MODEL_PATH,
5
    device=DEVICE,
6
    chunk_length_s=30,
7
    batch_size=BATCH_SIZE
8
)

TIP

chunk_length_s=30 enables processing of long audio files by splitting them into 30s chunks (Whisper’s native context window)

To transcribe:

1
result = pipe(
2
    audio_file_path,
3
    return_timestamps=True,
4
    generate_kwargs={"language": "german", "task": "transcribe"}
5
)
6

7
transcript_text = result.get("text", "").strip()

• The pipeline handles 16kHz resampling and normalization automatically.
• return_timestamps=True allows detailed output if needed later.
• We force the language to German because this is a specialized model
• The result is usually a dict {'text': "..."} or a list of chunks. The pipeline normalizes this to a simple dictionary access for ‘text’.