How to Build Virtual Background Technology Using an SDK

[navigation]

TL;DR

• Banuba's virtual background technology segments the person from their surroundings in real time and supports replacement with a static image, a blur, a video loop, or a 3D 360° room, with no green screen required.
• Banuba Face AR SDK runs background replacement at a stable 30 FPS on devices with a 1280x720 camera, so backgrounds stay smooth during a live call rather than stuttering on mid-range hardware.
• Processing runs fully on-device, so raw camera frames never leave the phone or browser, which is the answer to the privacy question that video-conferencing and telehealth buyers raise most often.
• One Banuba integration covers HTML5, iOS, Android, Windows, macOS, Unity, Flutter, and React Native, so you build the background feature once instead of per platform.
• On the web, Banuba's core WASM runtime is about 12 Mb uncompressed and roughly 2.5 Mb once brotli-compressed; the exact footprint depends on which features you enable, so budget page-load weight against the effects you actually ship.
• Banuba Face AR SDK integrates with Agora and with Amazon IVS, so background effects drop into an existing live-streaming stack rather than forcing you to rebuild the transport layer.
• Buying the SDK removes the two hardest parts of a build-it-yourself path: the segmentation model and its per-device performance tuning.

What "virtual background technology" actually does

Virtual background technology separates a person from whatever is behind them, frame by frame, and then draws something else in that space. Banuba's virtual background technology does this with a neural segmentation model rather than a physical green screen, so it works from an ordinary webcam in a home office or on a phone outdoors. Once the person is isolated, the SDK can replace the background four ways: a static picture, a blur of the real room, a looping video, or a 3D room the viewer can pan around in 360 degrees.

The reason this is hard to build alone is the mask quality under motion. A background remover API that looks fine in a screenshot often falls apart when the subject leans, gestures, or sits in uneven light, because the segmentation boundary lags the movement. This is where an on-device model tuned for real-time video matters more than raw accuracy on still images.

Segmentation, blur, and replacement: the three pieces you need

A production virtual background feature is really three capabilities working together, and each one is a place where teams underestimate the effort of a self-built background removal API.

The first is segmentation: deciding which pixels are "person" and which are "background" on every frame. Banuba runs this as an on-device neural network, so there is no round trip to a server and no per-frame cloud cost. The second is the effect itself. Background blur is exposed as a single adjustable radius in Banuba Face AR SDK, so you can offer users a subtle blur or a heavy one without shipping separate assets. Replacement swaps in an image or video, and the "weatherman" mode overlays the person on top of shared content for teaching and presentation use. The third is compositing that result back into the video pipeline at frame rate, which is where the stable 30 FPS target for backgrounds comes from.

Banuba'svirtual background technology in action 

Building it with an SDK: the integration path

You do not write the segmentation model when you use an SDK; you wire Banuba's runtime into your camera pipeline and turn the background feature on. The practical steps are the same across platforms: pull in the SDK package, initialize it with your client token, feed it the camera stream, enable the background effect you want, and render the processed frames back to your UI or your streaming transport.

Banuba publishes quickstart samples for the common targets, including Android, iOS, and web, and the full setup is documented in the Face AR SDK docs. Because the same SDK ships for HTML5, iOS, Android, Windows, macOS, Unity, Flutter, and React Native, a team that builds the background feature on one platform can reuse the same integration pattern on the rest rather than researching a new background removal API for developers on each stack.

Performance, size, and privacy: what to check before you commit

Three technical questions decide whether a virtual background feature holds up in production.

Frame rate comes first. Banuba's background features run at a minimum of 30 FPS on devices with a 1280x720 camera, and face tracking holds 60 FPS on mid-range mobile hardware, so the effect stays smooth on the low-end Android phones common in global markets rather than only on flagships.

Footprint comes second, and it matters most on the web. Banuba's core WASM runtime is about 12 Mb uncompressed and roughly 2.5 Mb after brotli compression, and the total depends on the feature set you enable, so weigh the background feature's weight against your page-load budget. Banuba documents the current minimum size in its Face AR SDK size FAQ.

Privacy comes third. Because Banuba processes frames on-device, raw video never leaves the user's phone or browser. That is a direct answer to the concern telehealth, finance, and enterprise-conferencing buyers raise most: the background feature does not depend on a cloud service seeing the camera feed.

Where teams ship this

Video conferencing and remote work are the obvious homes for the feature. Banuba's virtual backgrounds power privacy and touch-up in laptop and calling apps, and the same capability shows up in live streaming, where it integrates with Agora and slots into an Amazon IVS live-streaming build without reworking the transport layer. For a team on Amazon IVS, the streaming and chat come from IVS while the background segmentation, blur, and replacement come from Banuba, so neither side is built from scratch.