Skip to Content
InternalsAudio pipeline

Audio pipeline

Audio is where Viora spends the most engineering per line. There are three capture modes, a native denoiser, a user gain stage, and a two-layer processing model (hardware + software) whose layers rebuild at different times.

The two layers

The hardware layer is baked into the JavaAudioDeviceModule - the capture source and platform AEC/NS flags. It cannot hot-swap. The software layer is the libwebrtc APM, configured with MediaConstraints on the AudioSource; it can be recreated mid-call. The external chain is one WebRTC post-processor slot, so CompositeAudioProcessing runs four stages back-to-back on the same buffer.

The three modes

MicMode (in data/AppSettings.kt):

ModeUI labelCapture sourcePlatform AEC / NSSoftware APMRNNoise
DEFAULTStandardVOICE_COMMUNICATIONon / onNS + AGC + HPF + AECoff
DENOISEVoice IsolationVOICE_COMMUNICATIONon / onAEC + AGC + HPF, APM NS offon
RAWWide SpectrumUNPROCESSED (→ MIC)off / offeverything off, including AECoff
  • Standard - maximum cleanup. The OEM voice path plus the full software APM. Good for most calls.
  • Voice Isolation - the OEM voice path and RNNoise stacked. APM’s own NS is turned off so it doesn’t fight RNNoise; if RNNoise can’t load, APM NS falls back on.
  • Wide Spectrum - nothing between the mic and the encoder. UNPROCESSED source where the device supports it, no AEC at all. This is the musician / “original sound” mode: faithful room and instrument audio with no suppressor chopping sustained non-voice. It has no echo cancellation, so needsHeadphones = true and the call flow silently forces Standard when no headphones are connected.

Mode groups and mid-call switching

MicMode.hardwareGroup: RAW is group 1, the other two are group 0. Modes in the same group share a hardware capture profile, so switching between Standard and Voice Isolation mid-call is a live swap:

fun setMicMode(mode: MicMode) {
    currentMicMode = mode
    rnnoiseProcessor.applyEnabled(mode == MicMode.DENOISE && rnnoiseProcessor.isAvailable)
    if (mode.hardwareGroup != factoryMicMode.hardwareGroup || <source differs>) {
        audioStackDirty = true            // defer HW rebuild to next call
    }
    // live swap: new AudioSource with the mode's constraints, hot-swap the sender track
    val newSource = pcf.createAudioSource(audioConstraintsFor(mode))
    val newTrack = pcf.createAudioTrack("ARDAMSa0", newSource).apply { setEnabled(audioEnabled) }
    audioSender?.setTrack(newTrack, /* takeOwnership = */ false)   // no renegotiation
    oldTrack.dispose()
}

Crossing into RAW (or back) needs a different capture source, which the ADM can’t change while a call is running. So the picker locks the other group out once a call starts, audioStackDirty is set, and rebuildAudioStackIfDirty() rebuilds the PeerConnectionFactory + ADM (and re-adds every track to the PC) lazily just before the next call’s createPeerConnection().

RNNoise (RnnoiseProcessor.kt + cpp/)

RNNoise  v0.1.1 (BSD-3-Clause), vendored under app/src/main/cpp/rnnoise/ and built by CMake to libomnicall_audio.so for arm64-v8a, armeabi-v7a, x86_64 (C only, ANDROID_STL=none, -O2 -ffast-math -DFLOAT_APPROX -std=c99).

The JNI surface is four functions: nativeFrameSize, nativeCreate, nativeDestroy, nativeProcess(handle, FloatArray).

The resampling problem. RNNoise is hard-wired to 48 kHz mono, 480-sample (10 ms) frames. WebRTC’s APM capture path runs at 16 kHz on some devices (older Samsung voice paths) and 48 kHz on others. So process():

  1. Reads the APM frame (int16 or float32, capacity / numFrames bytes/sample).
  2. If mono and 16/24/48 kHz: linearly upsamples to 480 samples, runs nativeProcess, then box-averages back down (cheap anti-alias for the decimation).
  3. Any other format (stereo, odd rate) → pass-through, logged once.

applyEnabled(on) recreates native state against the last known capture format so a mid-call toggle takes effect immediately. release() and process() are both @Synchronized so the native handle can’t be freed while a frame is being processed on the APM thread.

User gain (MicGainProcessor.kt)

A final output trim / boost applied after AEC/NS/AGC, so it behaves predictably in every mode.

  • gain is a @Volatile Float, 1.0 = unity. Settable from any thread mid-call.
  • The buffer arrives as float32 in libwebrtc’s S16 range (±32768), capacity == numFrames * numChannels * 4. Capture is forced mono, so the processor ignores the band/channel split and walks capacity / 4 floats.
  • Within ±1 % of unity, process() is a no-op - dragging the slider back to 100 % fully restores the raw signal.
  • A tanh soft knee above ±26000 keeps a hot or over-boosted signal from wrapping into harsh digital clipping.

The settings slider snaps to 10 % increments; range is 0–200 %, or 0–500 % with Overboost on.

The zero-gain footgun

Silence is a selectable value

gain = 0.0 multiplies every sample by zero - total silence, indistinguishable from a broken mic. This has bitten the reference S9+ (a stored mic_gain=0.0). The 10 %-step slider and Overboost gating reduce the chance of landing there by accident, but 0 is still a selectable value.

Level metering (AudioLevelMeter.kt)

Two taps feed the speaking waveforms:

  • MicLevelProbe - a read-only stage in the external chain. Computes RMS of the outgoing mic buffer → a fast-attack / slow-release envelope → 0..1. Drives the local self-view waveform.
  • remoteAudioSink - an AudioTrackSink on the remote audio track. Same RMS → envelope. Drives the “other person is talking” waveform.

Both expose a plain Float getter (localAudioLevel, remoteAudioLevel) that is safe to read every frame from the Compose UI thread.

Opus configuration

Set two ways (see media negotiation):

  • SDP (tuneOpus): stereo=1, sprop-stereo=1, maxaveragebitrate=262144, useinbandfec=1, usedtx=0.
  • Sender parameters (applyAudioBitrate): encoding.maxBitrateBps = 262144 (128 kbps × 2 channels).

Plain speech VBR self-regulates far below the ceiling; the headroom is for shared media audio and, on a lossy link, FEC redundancy.

Media negotiationVideo capture