@nolag/voice-engine

The real-time half of a voice agent: turn taking, interruption, voice activity detection, filler speech, call-screener handling and recording.

Overview

@nolag/voice-engine is the one NoLag package with no NoLag dependency. It has no opinion about which telephony provider carries the call, which models do the thinking, or whether anything is coordinating from outside. Those are interfaces.

What it owns is the part that is genuinely hard, and that no prompt can fix: deciding when someone has stopped talking, who holds the channel, and what the caller actually hears.

It is one half of a voice agent. This half has to answer in about a second, so it stays fast and knows nothing. Pair it with @nolag/voice and the call gains the other half: a room through which it reaches a larger orchestrator for knowledge and tool calls, and a human who can watch or approve.

Key Features

• Voice activity detection that adapts to the room rather than using a fixed threshold
• Barge-in: the caller can interrupt the agent mid-sentence, and the agent's own voice cannot interrupt itself
• Filler speech ("let me have a look") to cover the wait while a turn is generated
• Call screeners and voicemail recognised and answered from a script, never from the model
• Recording: time-aligned caller and agent audio, plus a transcript and per-turn latency
• A browser simulator, so you can build a voice agent without a phone number

Installation

npm install @nolag/voice-engine

Quick Start

Everything below is one complete program. Run it, open the page, and talk to the agent. No telephony account, no phone number, no public tunnel.

import {
  VoiceSession,
  OpenRouterSpeechToText,
  OpenRouterLanguageModel,
  OpenRouterTextToSpeech
} from '@nolag/voice-engine'
import { startSimulator } from '@nolag/voice-engine/simulator'

const apiKey = process.env.OPENROUTER_API_KEY

// One key covers all three legs here, but each is an interface.
const providers = {
  stt: new OpenRouterSpeechToText({ apiKey, model: 'openai/gpt-4o-mini-transcribe' }),
  llm: new OpenRouterLanguageModel({ apiKey, model: 'mistralai/ministral-8b-2512' }),
  tts: new OpenRouterTextToSpeech({
    apiKey,
    model: 'deepgram/aura-2',
    voice: 'aura-2-thalia-en',
    sampleRate: 24000   // the rate this model emits, not the phone line
  })
}

await startSimulator({
  port: 3000,
  onCall: (transport) => {
    new VoiceSession({
      transport,
      providers,
      systemPrompt:
        'You are a friendly phone assistant. Keep replies to one or two short ' +
        'spoken sentences. Never use markdown or lists.',
      lines: { greeting: 'Hi, how can I help?' }
    })
  }
})

The simulator page speaks the same Media Streams protocol a carrier does, at the same endpoint, so the session cannot tell it apart from a real call. That matters more than convenience: barge-in and turn taking cannot be checked from a unit test, and iterating on them through real phone calls is slow and costs money every attempt.

Taking real calls

Twilio Media Streams is included as a transport. The webhook answers with TwiML pointing at your WebSocket, and each socket becomes one session.

import { WebSocketServer } from 'ws'
import { TwilioMediaStreamTransport, VoiceSession } from '@nolag/voice-engine'

// Answer the voice webhook with TwiML, where PUBLIC_HOST is reachable from the
// internet. The <Parameter> tags become CallInfo on the session:
//   <Response><Connect>
//     <Stream url="wss://PUBLIC_HOST/media">
//       <Parameter name="peer" value="+61400000000" />
//       <Parameter name="outbound" value="0" />
//     </Stream>
//   </Connect></Response>

const wss = new WebSocketServer({ server, path: '/media' })

wss.on('connection', (socket) => {
  const transport = new TwilioMediaStreamTransport({ socket })
  const session = new VoiceSession({ transport, providers, systemPrompt, lines })
  socket.on('close', () => session.close('socket closed'))
})

Configuration

VoiceSession

Methods: say(text) speaks something now and remembers it, instruct(text) adds silent guidance the model sees from its next turn, close(reason) ends the session.

Scripted lines

Said verbatim, with no model call, because the moments they cover cannot afford a surprise or a delay.

Listening behaviour

The detector adapts to the room, so most calls need none of this. Reach for it when the agent replies to nothing (raise noiseMultiplier), never hears a quiet speaker (lower it), or cuts people off mid-thought (raise silenceHangMs, at the cost of adding that delay to every reply).

Recording

A separate import, because it needs the filesystem, and opt-in, because nothing is captured unless you pass one.

import { createRecorder } from '@nolag/voice-engine/recorder'

const recorder = createRecorder({ dir: './recordings', callId: 'CA123' })
new VoiceSession({ ...options, observer: recorder })

Per call it writes <callId>-caller.wav, <callId>-agent.wav, a .jsonl event log with per-turn latency, and a readable .txt transcript. The two tracks are time-aligned, so they play side by side, and audio is streamed to disk rather than buffered, so a long call costs no more memory than a short one.

Bringing your own vendors

Implement three interfaces and the engine neither knows nor cares who is behind them:

interface SpeechToText { transcribe(req): Promise<string> }
interface LanguageModel { chat(req): Promise<string> }   // streams sentences
interface TextToSpeech  { readonly sampleRate: number; speak(req): Promise<void> }

speak streams deliberately: the caller hears the first byte, so time to first byte is what governs how responsive the agent feels, and the gap between providers is measured in seconds rather than milliseconds.

AudioTransport is the same idea for telephony. Implement it and any carrier, or a browser, or a test double, can drive a session.

What it does that is not obvious

Voice detection adapts to the room. A fixed energy threshold cannot serve both a quiet phone line and a noisy room. In a room noisier than the fixed floor, a static threshold does not merely misfire, it latches open permanently and streams noise to your transcription bill.

Interruption is judged independently of utterance state. The obvious implementation triggers barge-in when an utterance begins, which fails silently: if noise or the tail of the caller's own sentence already opened one, the opening transition never comes again and the caller can never interrupt for the rest of the call.

Playback is tracked by mark, not by clock. Audio goes out far faster than it is heard, so only the far end knows when the agent has finished speaking. A turn can contain several clips, so marks are counted rather than treated as a flag.

The model is the last resort, not the first. Noise, screeners and goodbyes are recognised and answered from a script before the model is consulted. Instructing a model not to do something works most of the time, which is another way of saying it fails in front of customers. Constraining it at the boundary works every time.

Speaking starts on the first sentence. The reply is streamed and synthesis of the first finished sentence begins while the model is still writing. Synthesis latency is dominated by fixed per-request overhead rather than text length, so splitting a reply into many small clips makes it slower, not faster.

Where to run it

A phone call is a long-lived WebSocket, which request-oriented platforms cannot hold. AWS Lambda and Cloud Functions will not work directly, so the natural targets are Cloud Run, Fargate, or Lambda behind API Gateway WebSockets. Match the platform to the socket first, then enjoy the small bundle for cold starts.

Coordination

Pair it with @nolag/voice and each call becomes a NoLag room, so dashboards, supervisors and other agents can watch a call and steer it while it is happening. VoiceSession takes an observer, and that package's publisher is shaped to be one, so the two clip together without either knowing much about the other.

Together they are a complete voice agent: fast reflexes on the phone, real capability behind it.