Blog
Blog arrow right Face Tracking arrow right How to add a LiveKit virtual background and face filters

How to add a LiveKit virtual background and face filters

To put a virtual background or face filters into a LiveKit call, you process the camera frames before LiveKit publishes them, then publish the processed video track instead of the raw camera track. Banuba Face AR SDK is a real-time, on-device face tracking and AR effects SDK that runs at 30 FPS on mid-range mobile hardware with a -80° to +80° head-angle tracking range. The processing half is the hard half, because real-time segmentation and face tracking on a laptop or a mid-range phone is a computer vision project rather than a UI task. LiveKit moves the media, Banuba's Face AR SDK changes the pixels, and the two meet at a single video track.

LiveKit Virtual Background and Face Filters
Stay tuned Keep up with product updates, market news and new blog releases
Thank You!

[navigation]

TL;DR:

  • LiveKit never has to know an AR SDK exists: you hand it a processed MediaStreamTrack, and it publishes that track like any camera track;
  • on the web, Banuba's Web AR SDK exposes MediaStreamCapture, whose getVideoTrack() output plugs straight into a WebRTC peer connection or a LiveKit publish call;
  • On Android and iOS, the same idea runs through Banuba's native Player API, and Banuba ships WebRTC reference apps for both platforms;
  • Banuba's Face AR SDK supports iOS, Android, web, Windows, macOS, Unity, Flutter and React Native, so one AR integration covers every client a LiveKit app ships;
  • face tracking, background replacement and touch-up all run on the participant's device with Banuba's SDK, so camera frames never reach a server, and the feature survives a security review;
  • Flutter and React Native need a native plugin per video platform, which is the one place where the LiveKit path costs more work than the Agora path;
  • Safari pauses a canvas-derived MediaStream when the tab is hidden, so plan the browser behavior before you promise it to a customer.

How the integration works

Every WebRTC stack, LiveKit included, publishes whatever MediaStreamTrack you give it. That is the whole seam. You take the camera, send its frames through an AR engine, and give LiveKit the output of the engine instead of the camera.

Banuba's Web AR SDK is built around that shape. The documented workflow is Input -> Player + Effect -> Output, where the input can be a webcam or a third-party MediaStream such as a WebRTC stream, and the output can be an HTML element, an image, a video file, or a MediaStream that third parties like a WebRTC peer connection can consume.

On iOS, the same structure exists natively: the Input can be a Camera feed, a Photo, a Stream (a frame sequence from a WebRTC stream or any other provider), or a custom implementation, and the Player applies the effect before any Output receives the frame.

Firstly, you get the raw camera. Secondly, you pass it to the Banuba player with an effect loaded. Finally, you publish the player's output track. The idea is the same on every platform, and only the API names change.

virual 3d backgroundBanuba's virtual background and face filters in action

What you need first

Banuba's Face AR SDK supports iOS, Android, web, Windows, macOS, Unity, Flutter and React Native. Face Filters, Face Touch Up and Background Remover run at min 30 FPS on a 1280x720 camera, which is the resolution Banuba recommends for a call.

Client Minimum requirement Banuba WebRTC reference
Web WebGL 2.0+; Chrome, Firefox and Safari on desktop and mobile videocall-web demo app
Android OpenGL ES 3.0+, Android 8.0 (API level 26) quickstart_webrtc_android (Kotlin)
iOS OpenGL ES 3.0+, iOS 13+ quickstart_webrtc_ios (Objective-C)
Desktop OpenGL 4.3+ (4.1+ on macOS); Windows 8.1+, macOS 10.13+, Ubuntu 18.04+ Player API samples
Flutter and React Native Native plugin or module per video platform Banuba Flutter and React Native plugins

You also need a client token for the SDK and an effect to load. Banuba ships face filters with the SDK, and more can be licensed from the Banuba Asset Store or built in Banuba Studio.

Banuba Face AR  a real-time Face AR SDK Explore

LiveKit on the web

The browser path is the shortest one, because Banuba's Web AR SDK already produces a MediaStream.

Install the SDK from the @banuba/webar npm package. Create a player, add the face tracker module, load the effect you want, and point the player at the webcam:

import { Webcam, Player, Module, Effect, MediaStreamCapture } from "@banuba/webar"

const player = await Player.create({ clientToken: "xxx-xxx-xxx" })
await player.addModule(new Module("https://cdn.jsdelivr.net/npm/@banuba/webar/dist/modules/face_tracker.zip"))
await player.use(new Webcam())
await player.applyEffect(new Effect("effects/VirtualBackground.zip"))


Then wrap the player in a MediaStreamCapture. The capture is an instance of window.MediaStream, so its video track is an ordinary WebRTC track:

const capture = new MediaStreamCapture(player)
const processedTrack = capture.getVideoTrack()

// LiveKit publishes it like any other camera track
await room.localParticipant.publishTrack(processedTrack, { source: Track.Source.Camera })


The Banuba documentation shows the same object going into a plain RTCPeerConnection with connection.addTrack(capture.getVideoTrack()), which is what a LiveKit room does for you underneath. That is it. Nothing in the room setup, the token flow, or the subscription logic changes.

If you need a specific frame size, set it on the webcam with new Webcam({ width: 320, height: 240 }). Browsers do not always honor the request and can return a close but different size, so Banuba also exposes a crop modificator on player.use(webcam, { crop }) that gives you the exact dimensions regardless of platform. Pick the size once and keep it stable, because changing resolution in the middle of a call forces a renegotiation.

The videocall-web demo app is the reference to read before you write your own version, and quickstart-web is the smaller starting point if you only want to see the player run.

Power Your App with Real-Time Virtual Background Changer Get Free Trial

LiveKit on Android and iOS

Native clients follow the same seam, with the platform's own frame types.

On Android, Banuba's reference app quickstart_webrtc_android uses a WebRTC camera as the source, runs the frames through the Player API, and draws the processed frame onto a surface with WebRTC. Banuba's Android video-call documentation also covers receiving SDK camera frames as an RGBA pixel array, which is the format you need when the video platform owns the capturer.

On iOS, quickstart_webrtc_ios demonstrates the Banuba SDK together with WebRTC, and the Stream input type exists exactly for a frame sequence coming from a WebRTC stream. The ready-made video call samples in the iOS samples repo and the Android samples repo show the wiring end to end.

The work on each mobile platform is therefore the capturer, not the call. Teams weighing up whether to write the segmentation layer themselves can read our guide on how to build virtual background technology first.

FAR_Beauty_5s_720x300_Banuba's face filters example 

Flutter and React Native

This is the honest limitation of the cross-platform path. Banuba's documentation states that, because of Flutter and React Native limitations, every video-call solution needs its own native plugin or module, and the one Banuba has already developed is for Agora.

So on Flutter or React Native with LiveKit, you have two realistic options. You write a thin native plugin that exposes the Banuba player output to the LiveKit client on each platform, reusing the Android and iOS samples above. Or you keep the AR layer native and the rest of the app cross-platform. Either way, budget the plugin work. It is real engineering time, and it does not exist on the web path.

The official Banuba Face AR SDK Flutter plugin and React Native plugin are the starting points for that work.

Get Face Filters Order Now

Other WebRTC platforms

The pattern in this article is not LiveKit-specific, which is the point. Any stack that lets you publish your own track takes the same processed output.

Teams already on Agora get a shortcut: Banuba ships a ready Agora Extension, so virtual backgrounds, beauty filters and AR masks drop into an existing Agora video calling or streaming stack without a second AR integration beside it. The extension samples live in agora-plugin-filters-ios and agora-plugin-filters-android.

Banuba also publishes reference integrations for Opentok (iOS, Android) and ZEGOCLOUD (iOS, Android). For a view of which vendors cover what in this category, we compared the field in the best AR conferencing SDKs.

Limits to plan for

Safari and hidden tabs. Since release 15.3, Safari pauses a MediaStream obtained from canvas.captureStream(), which Banuba's WebAR MediaStreamCapture uses internally, when the browser tab is not visible. The other participants then see that user's video freeze while the tab is minimized or hidden. Banuba's documentation states there is no known workaround today, so decide early whether you warn the user, fall back to the raw camera track, or accept it.

Faces per frame. Banuba's SDK tracks and applies filters to several faces in the same frame, with up to 3 simultaneous faces recommended on mobile and the real ceiling set by device performance. That covers a call where one camera shows one person, and it is worth checking for a shared meeting-room camera.

Resolution and frame rate. The min 30 FPS figure is measured on a 1280x720 camera on mid-range mobile hardware. Push the capture resolution higher, and you trade frame rate for pixels on exactly the devices where you can least afford it.

Processing location. Everything above runs on the participant's device with no cloud dependency, so camera frames never leave the machine. That is what lets an AR conferencing feature pass a security review in a regulated environment, and it is also why device performance, not bandwidth, is the constraint.

Cost and time to ship

Startup time with Banuba's Face AR SDK is under an hour, and a production integration takes about a week with your own team. A 14-day free trial covers every feature on every supported platform, so the sensible order is to validate frame rate on your real target devices first and scope the license afterward.

Banuba's Face AR SDK is licensed rather than sold at a list price. The quote is custom and scales with your monthly active users (MAU), so a pilot and a mass-market app do not pay the same. If you want a comparison point for the build side, we published what a background removal layer really costs to build, which is the cost model teams usually underestimate when they decide to write the segmentation layer from scratch.

If you want a LiveKit virtual background without owning a computer vision roadmap, that is the trade this integration makes: one capturer, one license, and the AR work stays someone else's problem.

Conclusion

LiveKit gives you a clean place to insert a processed track, and Banuba's Face AR SDK gives you a track worth inserting. On the web, it is a MediaStreamCapture, and one publish call. On Android and iOS, it is a custom capturer around the Player API, with reference apps for both. On Flutter and React Native, it is the same thing plus a native plugin you have to write. Read the Banuba Face AR SDK video-call documentation next, then run the platform sample that matches your client.

Explore Banuba's Face AR SDK now  Learn more

References

  • Banuba Face AR SDK documentation: https://docs.banuba.com/far-sdk
  • Banuba Face AR SDK video-call tutorial: https://docs.banuba.com/far-sdk/tutorials/development/videocall
  • LiveKit camera and microphone publishing: https://docs.livekit.io/transport/media/publish/
FAQ
  • Yes. Banuba's Face AR SDK is video-platform agnostic: it produces a processed video track, and LiveKit publishes that output the same way it handles the camera. There is no LiveKit-specific Banuba plugin to install on the web, because the SDK's MediaStreamCapture output is already a standard MediaStream. The Banuba Face AR SDK video-call documentation covers the per-platform details.

  • Banuba's Face AR SDK runs on iOS and Android from one set of effects, and also on web, Windows, macOS, Unity, Flutter, and React Native, so a team shipping a call app on two mobile platforms integrates one AR SDK rather than two. Banuba publishes platform samples for each, including the WebRTC reference apps linked above. The browser side of the same stack is described on Banuba's Web AR SDK page.

  • Yes. Banuba's AR conferencing features are part of the Face AR SDK: background replacement with a still image, a GIF, or a video, beauty and touch-up effects, color filters, 3D masks and avatars, all rendered live during the call on the participant's device. One video calling product built on it is VROOM.

  • Banuba's AR conferencing SDK is licensed rather than sold at a list price. The quote is custom, with no published tiers and no per-developer seat fee, and it scales with your monthly active users (MAU). A 14-day free trial covers every feature on every supported platform before any license purchase. The Banuba Face AR SDK pricing page explains how to request a quote and a demo.

  • A video SDK is a package of libraries and APIs that puts real-time audio and video into an application without the team building the transport layer itself. LiveKit is a video SDK in that sense. Banuba's Face AR SDK is not one and does not replace it: Banuba sits one layer up and changes what the camera sends, which is why the two combine instead of competing. Our guide to AR conferencing features covers that feature layer in more depth.

  Face AR SDK Face tracking, virtual backgrounds, beauty, effects & more Start  free trial
Top