RagLeap Docs
Docs for ragleap-core v0.13.6 · latest
Use

Language detection

RagLeap Core auto-detects language during document ingestion (per chunk) and during chat (per query), using FastText (through the fast-langdetect package) with a character n-gram fallback for hard short queries, plus script-based heuristics for CJK, Hangul, and Kana text. Since every channel (WhatsApp, Telegram, Discord, Voice, and the API directly) routes through the same core chat pipeline, detection applies consistently everywhere without per-channel wiring.

Setup: works out of the box with no configuration. Optionally set DEFAULT_LANGUAGE (fallback when detection fails or text is too short), LANGUAGE_DETECTION_CONFIDENCE_THRESHOLD (default 0.7), and LANGUAGE_DETECTION_SUPPORTED_LANGUAGES (comma-separated allowlist, blank = unrestricted).

Honest status: verified working end-to-end — document-level detection tested at high confidence (0.9999) on a real mixed-language document, and query-level detection confirmed working via both the API and CLI.

Known limitations:

  • Short queries in closely-related languages can still be misdetected, because there is little text to work with. Pull request #452 (included in this release) added the FastText plus character n-gram approach, which fixed the reported French/Italian case; it scored 32/32 on a 32-query benchmark whose queries were also used to choose its threshold, so treat that as evidence on a small sample, not a general accuracy figure
  • Detection is one-way only: RagLeap Core detects the query's language and surfaces it, but does not yet steer the AI's response language to match — that's a reasonable next step for a contributor

Generated from the ragleap-core v0.13.6 source. The repository is the source of truth and may be newer.