Calibrating GigaAM-Multilingual for Kyrgyz

This note documents how ai-sage/GigaAM-Multilingual was benchmarked and then
adjusted before it was used to transcribe the Jashoo Tatymy corpus, and it is
written so the numbers can be reproduced end to end. It covers one finding the
model's authors may care about: on Kyrgyz audio the shared multilingual
character vocabulary leaks Kazakh-only letters, and the obvious fix for that
makes recognition worse rather than better.

Everything below ran on a single RTX 2060 SUPER (8 GB).

1. Environment

torch 2.11.0+cu128, torchaudio 2.11.0+cu128, transformers 5.17.0
hydra-core 1.3.7, omegaconf 2.3.1, pyannote.audio 4.0.7
datasets 5.0.1, librosa 1.0.0, soundfile 0.14.0

pyannote.audio is only needed because modeling_gigaam.py imports it at module
scope for transcribe_longform; short-form transcription never touches it, but
transformers.dynamic_module_utils.check_imports refuses to load the module
without it.

One packaging note: installing transformers into a fresh environment pulled
torch down from 2.11.0 to 2.9.1 while leaving torchvision 0.26.0 in place,
which then failed with operator torchvision::nms does not exist. Pinning torch,
torchvision and torchaudio to the same build resolves it.

2. Memory and throughput

Measured with dynamic batching (below) on 8 GB. Both fit comfortably.

A fixed batch size does not survive the whole corpus. Utterances are sorted
short-to-long so that padding is cheap, but Conformer self-attention is quadratic
in sequence length, so a batch of 32 that is comfortable at 1 s of audio runs out
of memory at 20 s. Batching by total padded duration fixes it:

BATCH_SECONDS = 150.0
MAX_BATCH = 48
order = sorted(range(len(rows)), key=lambda i: rows[i]["duration"])
batches, cur = [], []
for k in order:
    d = rows[k]["duration"]
    if cur and (max(d, cur_max) * (len(cur) + 1) > BATCH_SECONDS or len(cur) >= MAX_BATCH):
        batches.append(cur); cur = []
    cur_max = d if not cur else max(cur_max, d)
    cur.append(k)
if cur:
    batches.append(cur)

Note also LONGFORM_THRESHOLD = 25 * SAMPLE_RATE in modeling_gigaam.py:
transcribe() raises above 25 s, so utterances have to be split before this
point.

3. Batched inference

transcribe() takes one file path at a time. For a corpus it is worth calling
the encoder and the CTC head directly — model.model.forward() accepts a padded
waveform batch plus lengths and runs the preprocessor and encoder under fp16
autocast:

import torch
from transformers import AutoModel
model = AutoModel.from_pretrained(
    "ai-sage/GigaAM-Multilingual", revision="large_ctc", trust_remote_code=True
).to("cuda").eval()
inner = model.model
blank = len(model.config.cfg["model"]["cfg"]["decoding"]["vocabulary"])
x = torch.zeros(len(wavs), max(len(w) for w in wavs), device="cuda")   # 16 kHz mono
lengths = torch.tensor([len(w) for w in wavs], device="cuda")
for j, w in enumerate(wavs):
    x[j, :len(w)] = torch.from_numpy(w).cuda()
with torch.inference_mode():
    enc, enc_len = inner.forward(x, lengths)
    log_probs = inner.head(encoder_output=enc)
    labels = log_probs.argmax(-1)
    keep = labels != blank                                  # CTC collapse
    keep[:, 1:] &= labels[:, 1:] != labels[:, :-1]
    keep &= torch.arange(labels.shape[1], device=labels.device)[None, :] < enc_len[:, None]
texts = [inner.decoding.tokenizer.decode(labels[b][keep[b]].tolist())
         for b in range(len(wavs))]

4. FLEURS Kyrgyz benchmark

google/fleurs, config ky_kg, split test: 977 utterances, of which 968
remain after dropping those longer than 25 s (3.18 h). References are
transcription (already lowercased and stripped of punctuation upstream).

Normalisation before scoring — lowercase, ё → е, strip anything that is not a
word character, whitespace or apostrophe, collapse whitespace:

def norm(t):
    t = t.lower().replace("ё", "е")
    t = re.sub(r"[^\w\s']", " ", t, flags=re.U)
    return " ".join(t.split())

WER and CER are corpus-level: total edit distance over total reference length,
not a mean of per-utterance rates. Greedy decoding, no beam search, no LM.

Results

5. The Kazakh orthography leak

GigaAM-Multilingual decodes ru/kk/ky/uz through a single 71-class character
vocabulary with no language conditioning. On Kyrgyz audio it therefore sometimes
selects Kazakh-only letters — і ғ қ ұ һ ә, none of which exist in the Kyrgyz
alphabet. On 709 utterances of in-domain Kyrgyz drama audio:

абдан жакшы                                    →  абдан жақшы
мен бул жердемин                               →  мен бұл жердемін
ал болгону бала да ... ал кайдан билет         →  ... ал қайдан біледі
менин кызыгуум дегениң кандай                  →  менің қызығуым дегенің қандай

What does not work: banning the classes

The obvious fix is to mask the six Kazakh-only columns to -inf before the
argmax. It does remove every Kazakh letter, but greedy CTC then frequently falls
through to blank rather than to the Kyrgyz counterpart, and the character is
dropped entirely:

тоқта              →  тота              (к lost)
биледі             →  биледт
сен өзің баласың   →  сен өзң баласың   (и lost)
менденемді         →  менденемд

What works: folding the probability mass

Add each Kazakh class's probability to its Kyrgyz counterpart in log space
before the argmax, so the model's own distribution decides:

KZ_TO_KY = {"қ": "к", "ғ": "г", "ұ": "у", "і": "и", "һ": "х", "ә": "а"}
vocab = list(model.config.cfg["model"]["cfg"]["decoding"]["vocabulary"])
v2i = {c: i for i, c in enumerate(vocab)}
pairs = [(v2i[k], v2i[v]) for k, v in KZ_TO_KY.items()]
log_probs = inner.head(encoder_output=enc).clone()
for kz, ky in pairs:
    log_probs[..., ky] = torch.logaddexp(log_probs[..., ky], log_probs[..., kz])
    log_probs[..., kz] = -1e9
labels = log_probs.argmax(-1)

тоқта → токта, жақшы → жакшы, бұл жердемін → бул жердемин.

Measured effect

Against the ElevenLabs Scribe transcripts that existed for part of this corpus,
over the same 709 utterances:

Folding and post-hoc transliteration land in the same place on aggregate,
because the merged mass usually only reorders the two classes involved. Folding
is preferred anyway: it can never produce a character the acoustics did not
support, whereas string replacement rewrites і to и even where the model was
right to hesitate. Banning is measurably worse for the reason shown above.

Across the full 22.08-hour corpus the released transcripts contain zero
Kazakh-only letters.

A residual limitation

Folding maps і → и unconditionally, but Kyrgyz vowel harmony sometimes calls
for ү instead — түшінішің should be түшүнүшүң. A language-conditioned
decoder, or a Kyrgyz-restricted vocabulary trained as such, would handle this;
a per-class merge cannot.

6. A methodological caution

Before running FLEURS, the same models were compared against pre-existing
ElevenLabs Scribe transcripts of this corpus, on the assumption that an
in-domain reference beats an out-of-domain benchmark. That comparison ranked
kyrgyz-whisper-medium ahead of GigaAM (median CER 0.071 vs 0.079).

The FLEURS run with human references reverses it decisively (0.166 vs 0.103
WER). The in-domain "reference" was itself machine output with its own error
profile, and an autoregressive model with a language-model prior produces text
that resembles another such system more closely than a pure CTC model does —
regardless of which is nearer the truth. Agreement with a machine reference is
not accuracy, and on this corpus it pointed the wrong way by a wide margin.

7. Reproducing

The benchmark harness is scripts/fleurs_eval.py in the pipeline repository
that produced this dataset; the decoding path is scripts/transcribe.py. Both
are self-contained and need only the package versions listed in §1. The FLEURS
run takes about two minutes per GigaAM revision on a 2060 SUPER, most of which
is dataset decoding rather than inference.