thinge docs Open the app →

Audio pipeline

Always-on voice is the product, so the audio pipeline is where most of the engineering went. This page explains how it works on the desktop client and why it is built differently from a typical WebRTC application. Read Architecture first for the surrounding system.

Why a native engine at all

Most web-era voice apps hand audio to a black box — the browser's WebRTC stack, or libwebrtc — and accept whatever it does. thinge's desktop client instead owns the pipeline end to end, in native Rust, outside the webview.

The reason is quality. OS webviews (WKWebView, WebKitGTK) have inconsistent, often poor WebRTC audio, and give you no control over capture, timing, or echo handling. For an app you sit in all day, that is not good enough. So on the desktop, voice media never touches the webview: capture, encode, the WebRTC engine, decode, mixing, and playback are all native. (In a browser, where there is no native process, thinge falls back to the browser's own WebRTC engine — that is the web product.)

The engine is built on str0m, a sans-IO WebRTC library: it does the protocol, but thinge owns the event loop and the clock. That ownership is what makes the timing work below possible — and testable.

The two directions

Outbound (your microphone)

cpal capture → resample to 48 kHz → VAD gate → Opus encode (DTX) → RTP → UDP
  • Capture comes from the OS audio device via cpal, resampled to Opus's 48 kHz.
  • A voice-activity detector (VAD) gates transmission: when you are not speaking, nothing is sent. It also produces the "is speaking" and level signals that drive the speaker rings in the UI.
  • Opus encodes the voiced audio with DTX (discontinuous transmission), so silence costs almost nothing on the wire — the thing that makes an open-mic channel cheap to leave running.

Inbound (everyone else)

UDP → per-stream RTP → Opus decode (per speaker) → per-stream jitter buffer → mix → cpal playback

The SFU forwards each speaker as a separate stream. The client decodes each one independently, runs each through its own jitter buffer, then mixes the decoded audio down to what your speakers play. Crucially, mixing happens on the client, not the server — the SFU only forwards. That keeps the server cheap and gives the client full control over timing and playout.

The hard part: the receive/timing edge

Getting audio sent is easy; making N remote speakers sound good on a real network is the hard part, and it is where most of the work goes:

  • Per-stream jitter buffering. Each speaker's packets arrive reordered, late, or lost. A per-stream jitter buffer absorbs that, conceals loss, and adapts its depth: it grows under congestion until stalls stop, then shrinks back toward a low-latency floor at talk-spurt boundaries.
  • Time-stretching (WSOLA). A buffer that grew against a continuous sender (a browser that never pauses to yield a talk-spurt boundary) can't shrink at boundaries because there are none. So the playout plays such a stream slightly faster to drain it — and slightly slower to ride out a near-empty buffer — without shifting pitch, using waveform-similarity overlap-add (the technique libwebrtc's NetEq uses). Clean audio at target passes through untouched.
  • Clock-skew-immune playout. Capture and playback clocks drift relative to each other; over a long session that drift would accumulate into stalls or overruns. Playout is paced by playback-buffer occupancy rather than the wall clock, so skew cannot build up — trading a small, bounded amount of latency for long-session stability.
  • Bitrate adaptation. The encoder follows a bandwidth estimate (transport congestion control): it backs the Opus bitrate off under congestion and recovers when the link clears, within a voice-appropriate band.
  • Lock-free audio callbacks. The real-time audio thread never blocks; the heavy work is kept off it.
  • Silence done right. Because DTX means the wire goes quiet between words, the pipeline reconstructs proper RTP timing across silence (timestamp jumps with the talk-spurt marker) so gaps are not mistaken for lost time.

Echo cancellation

An open-mic channel played through speakers will feed your own output back into your microphone unless it is cancelled. Acoustic echo cancellation (AEC) needs the far-end reference — what the speakers are actually playing — so the swappable unit in thinge is the capture-and-playback pair, behind a common seam. There is a backend ladder:

  • macOS — Apple's VoiceProcessingIO audio unit (hardware-grade AEC and noise suppression).
  • Linux — an in-process WebRTC AEC3 insert running on a dedicated worker thread, with the far-end reference tapped at the output.
  • Fallback (everywhere) — the plain device pair plus a half-duplex echo guard, armed only when no hardware AEC is active.

The UI is told which mode is live, and there are environment switches to A/B the backends.

How this differs from a typical WebRTC app

  • The client owns the pipeline, end to end, instead of delegating to a browser/libwebrtc black box — so capture, timing, and echo are all tunable.
  • The SFU does no mixing. Each speaker is a separate forwarded stream; the client decodes and mixes them. The server forwards packets and nothing more.
  • Audio-only, so no video machinery. No keyframes, no PLI/FIR — an entire class of complexity is simply absent.
  • A sans-IO engine with an owned clock, which makes the jitter-buffer and playout timing deterministic and unit-testable against seeded network- impairment traces, rather than something you can only observe in production.

It works well on a LAN or decent Wi-Fi. The receive/timing-edge machinery — adaptive jitter buffering, WSOLA time-stretching, and bitrate adaptation — is in place; tuning it across a wide range of real-world links is ongoing work. That edge is the difference between "fine most of the time" and "reliable on a bad network."