How We Optimized whisper.cpp for Malayalam: Native Script Output Without Cloud Translation

Transcribing local Indic languages with high accuracy on-device has always been a hard engineering challenge. Most open-source speech models default to translating non-English speech into English, or fail to output complex scripts like Malayalam correctly.

When compiling the whisper.cpp runtime for Parayu, we set out to build a native Malayalam dictation pipeline that runs 100% locally with zero translation latency. Here is a technical breakdown of how we achieved it.

Bypassing Default Translation Logic

Whisper models naturally output translation tokens when prompted. In standard setups, developers pass the --translate flag or default language settings. To ensure native Malayalam output, we modified the model parameters passed to the transcription engine:

1. We force-disable translation mode by setting translate: false in the Whisper model parameters.

2. We explicitly inject the Malayalam language token (ml) and prompt the model with common local terms to initialize correct spelling contexts.

3. We bypass screenwriting translate conditions, allowing direct phonetic mappings.

Optimizing Column & Language Priorities

In our UI, we implemented a custom function getOrderedScreenwritingLangs() that dynamically positions Malayalam as the primary column on the left side of the dashboard, optimizing layouts for our core user base. Here is a snippet of our language sorting algorithm:

export function getOrderedScreenwritingLangs() {
  const list = [...SUPPORTED_LANGUAGES];
  // Malayalam (ml) is positioned first
  const mlIndex = list.findIndex(l => l.code === 'ml');
  if (mlIndex > -1) {
    const [ml] = list.splice(mlIndex, 1);
    return [ml, ...list];
  }
  return list;
}

Handling System RAM Constraints

Running native Malayalam models locally requires careful memory alignment. By using 16-bit floating point weights optimized for Apple Silicon Metal framework and Windows DirectML, we enabled real-time transcription speeds (under 1.5x audio length) while keeping RAM consumption under 1.2GB. This ensures the app can remain running in the background of your system without slowing down other tools.

We are continuing to refine our custom Indic dictionaries. If you notice any spelling edge cases, you can add them to your local personal dictionary inside the Parayu desktop app.