RagLeap Docs
Docs for ragleap-core v0.13.6 · latest
Use

Channels & integrations

Voice Channel (Twilio)

RagLeap Core includes a real-time voice channel: Twilio Media Streams connects via WebSocket, your speech is transcribed with OpenAI Whisper, answered by the core RAG pipeline, and spoken back with OpenAI TTS. The voice-activity detection and echo-suppression logic is carried over from a production system tuned against real call traffic.

Runs as a separate service on port 8765 (see docker-compose.yml), since Twilio's real-time audio protocol needs a raw WebSocket server, not an HTTP route.

Setup:

  1. Set OPENAI_API_KEY in .env (used for both Whisper STT and TTS by default; VOICE_STT_PROVIDER=deepgram uses DEEPGRAM_API_KEY instead, and VOICE_TTS_PROVIDER=elevenlabs uses ELEVENLABS_API_KEY with the voice id from ELEVENLABS_VOICE_ID)
  2. Optionally set VOICE_BOT_NAME, VOICE_GREETING, VOICE_TTS_VOICE
  3. Set VOICE_WS_TOKEN in .env to a long random value (the installer does this for new installs; openssl rand -hex 24 makes one). Without it the voice server refuses every call.
  4. Point a Twilio phone number's <Connect><Stream> TwiML at wss://your-domain.com:8765/<your VOICE_WS_TOKEN>. Use wss (not ws): the token is part of the address, and a connection without it never becomes a call. If the token ever leaks, change it in .env and in the TwiML.

Security: anyone who can phone your number can talk to the bot, which is the point of a public line, so decide what your documents and role instructions should say to a stranger. Spoken feedback ("good answer" / "bad answer") is ignored on the voice channel, because callers are anonymous and that feedback changes the memory used on every channel. VOICE_MAX_CALLS (default 10) and VOICE_MAX_CALL_MINUTES (default 30) limit simultaneous calls and call length, which also limits what a caller can spend of your speech and model credits.

Honest status: the WebSocket server, Twilio event protocol handling, and error handling are verified working. The full Whisper/TTS round-trip has not yet been live-tested end-to-end (requires OpenAI API credits). If you try it and hit issues, please open one — this is exactly the kind of real-world testing this project needs.

Known limitations, carried over from production and not yet fixed here:

  • OpenAI Whisper (STT) and OpenAI TTS are the defaults — Deepgram STT and ElevenLabs TTS (multi-language support) are now also supported via VOICE_STT_PROVIDER / VOICE_TTS_PROVIDER, but have not been live-tested
  • Non-English TTS quality varies since OpenAI's TTS voices are English-tuned
  • Typical round-trip latency in production was 6-8 seconds

Integrations

flowchart TD
    subgraph Connectors["10 connectors, one shared interface"]
        C1[MySQL] --- C2[PostgreSQL] --- C3[MongoDB]
        C4[REST API] --- C5[Salesforce] --- C6[HubSpot]
        C7[Shopify] --- C8[Google Sheets] --- C9[Stripe]
        C10[CSV Upload]
    end
    Connectors --> Svc["RealTimeExternalDataService<br/>real-time query, 5-min cache, no separate sync step"]
    Owner["Owner configures an action:<br/>trigger phrase or auto-match on SQL/field names<br/>+ a query template (SELECT / UPDATE / INSERT / DELETE)"] --> Svc
    Svc --> Match["match_and_execute_action(workspace_id, user_message, user_identifier)<br/>extracts order_id / email / phone from the message,<br/>substitutes into the template, runs the query"]
    Match --> Chat["Chat channels (WhatsApp/Telegram/Discord/Web)<br/>api/personal_bot_views.py"]
    Match --> Voice["Voice channel<br/>memory/voice_views.py::twilio_voice_speech<br/>wired in 2026-08"]
    Chat --> RAGCtx["Result injected as context<br/>into the RAG prompt"]
    Voice --> RAGCtx
    RAGCtx --> Answer["AI answers with real account/order/appointment<br/>data, not just document knowledge"]
Loading

Verified live, read from api/addon_realtime.py's RealTimeExternalDataService. Two things worth being direct about: match_and_execute_action genuinely supports write queries (UPDATE/INSERT/DELETE), not just read-only lookups — owner-configured, so the safety boundary is whatever SQL the owner writes into the template, not something the framework restricts on its own. And until 2026-08, this action-matching step only ran on chat channels; voice calls had no equivalent, which is the gap closed in the Voice Channel Routing diagram above.

RagLeap Core connects to external databases and business tools, syncing per-user context to personalize RAG responses. Nine connectors are included: MySQL, PostgreSQL, MongoDB, generic REST APIs, Salesforce, HubSpot, Shopify, Google Sheets, and Stripe. Newer connector code for CSV upload, Airtable, BigQuery, Snowflake, WooCommerce, Notion, Slack, Gmail and Razorpay is also in the repo; treat those as experimental until you have verified them against your own account (tracked in #27).

Every CRM/SaaS connector uses credentials you provide directly — a username/password, a private-app token, an admin API token, a service-account JSON file, or a secret key, depending on the service. None require registering an OAuth app; nothing here depends on RagLeap owning any third-party developer account.

Credentials are encrypted at rest (Fernet/AES-128) before being stored.

Setup:

  1. Generate an encryption key: python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
  2. Set ADDON_ENCRYPTION_KEY in .env to that value
  3. Install the SDK for the connector(s) you want (each is optional — see requirements.txt)
  4. Create a data source: POST /integrations with name, source_type, and the relevant credential fields
  5. Test it: POST /integrations/{id}/test
  6. Sync it: POST /integrations/{id}/sync

Honest status: verified end-to-end against a real public API — connection testing, syncing, correct identifier-field matching, and credential encryption (checked as actual ciphertext in the database, not just assumed) all confirmed working.

Known limitations:

  • CSV Upload, Snowflake, BigQuery, WooCommerce, Airtable, Notion, Razorpay, Slack, and Gmail are good-first-issue candidates for anyone wanting to add one
  • Sync is on-demand only (POST /integrations/{id}/sync) — no scheduled background sync yet, though the schema tracks sync_interval_minutes for a future Celery-beat-equivalent
  • Synced context isn't automatically injected into chat responses yet — each channel adapter would need to know its own user's identifier first, which is a reasonable next contribution

Generated from the ragleap-core v0.13.6 source. The repository is the source of truth and may be newer.