How to Build AR Video Conferencing Features Using an API

[navigation]

TL;DR:

• A video conferencing API connects your app to a calling service; the AR layer (filters, backgrounds, beautification) is added separately with a face AR SDK, so you rarely rebuild the call stack, only the visual layer.
• Building face tracking and AR rendering in-house typically takes months and a computer-vision specialist; Banuba Face AR SDK gets the same features running in about a week.
• Banuba's virtual webcam features, including Background Remover, run at a minimum of 30 FPS on devices with a 1280×720 camera, so virtual backgrounds stay smooth during a live call.
• On web, the Banuba SDK adds a 12 Mb minimum footprint (the WebAssembly module), which affects page-load speed and bandwidth, so budget for it when weighing an SDK against an in-house build.
• Banuba Face AR SDK processes video on-device, with no cloud round-trip, which matters for privacy-sensitive telehealth and enterprise calls.
• It runs across iOS, Android, Web, Desktop, Flutter, and React Native, so one integration covers most platforms.
• Native call-provider integrations reduce friction: Banuba Face AR SDK ships an Agora extension, so AR effects pair with an existing transport layer without conflicts.
• The main trade-off is long-term cost: an SDK shifts maintenance to the vendor, but license fees over 2 to 3 years should be weighed against a one-time in-house build.

What is a video chat with filters?

"AR filters" is an umbrella term for a host of visual effects. These may include:

• LUTs (Instagram filters). A basic color scheme change for an entire picture/frame. For example, sepia or black-and-white.
• Background replacement. Detecting a person in a picture and changing the rest. Powered by background removal technology, the new background can be a blur, a static image, a GIF, a video, or even a 3D environment.
• Face masks and stickers. The effects that TikTok and Snapchat helped make popular. Masks are complex 3D objects, while stickers are simpler and two-dimensional. Read the article on this topic for more information.
• Beautification. Subtle changes that improve the person's appearance: smoothing the skin, whitening teeth, removing acne and eye bags, etc.
• Virtual makeup. Applying mascara, lipstick, eyeliner, and other cosmetic products as AR effects. Today they can be realistic enough to be used for virtual try-on.
• Avatars. 3D characters that replace the user on screen but mimic the person's movements and facial expressions.
• Overlays. Images, text, and GIFs placed over the video feed.
• Triggers. Effects that are launched by making a gesture or a certain facial expression.

Most of these effects are based on neural networks that track certain facial features. There are more things to take into account if you are a developer and intend to make your own video chat app with filters:

• Real-time work. Some AR technologies can only transform your face on a static picture, as they aren't optimized for real-time video feed.
• Anti-jitter. In many situations, the camera in the person's hands will be shaking slightly. This will affect the image quality, so an anti-jitter algorithm is needed to stabilize the feed.
• Multi-face tracking. What if there are several people on one screen (e.g. in a group chat) and you need to apply filters and effects to all of them? Face detection algorithms are scalable, but there are still nuances to their implementation.
• Occlusion and lighting. Not all cameras are created equal, and people will not always have perfect conditions for seamless AR experiences. This means you need to account for situations when the lighting is bad or a person is obscured.
• Video settings. Adjusting the parameters of a video feed automatically, or letting users do this. Some filters and effects can also be tuned, e.g., the degree of skin smoothing or the size of a virtual hat.

To add these effects to a call without building the underlying computer-vision pipeline, teams integrate a dedicated video conferencing api or AR SDK. Banuba's virtual webcam features (Face Filters, Face Touch Up, and Background Remover) run at a minimum of 30 FPS on devices with a 1280×720 camera, so effects stay smooth in real time.

Beautification at work

What can you build with AR video chat? Success stories

Rather than being just a standalone type of app, video chats are often part of a larger whole. If you are still comparing providers, this rundown of face-tracking SDKs suited to real-time video conferencing walks through the main options.

Social media 

TikTok has been a top dog for a while now, and its competitors try to win the user base back. Filters provide opportunities to create engaging and viral content, which motivates people.

However, there are many other examples. Chingari is an Indian video-based social network. When New Delhi banned several Chinese apps, including TikTok, the company sprang into action. They launched an app with a distinctive Indian style, partially achieved by using custom face filters designed specifically for their country. As a result, they have over 50 million installs, and the number keeps growing.

Virtual meetings

Professional video calls are popular among business users, politicians, lawyers, and many other people. So many online meeting providers use augmented reality to improve the group call experience.

Some, like Google Meet, use inbuilt functionality to provide background replacement, the core feature that helps protect users from embarrassment. Others go a different route. For example, Agora provides an SDK for conference calls. They have also launched an extensions marketplace where companies can buy complementary modules: sound suppression, automated moderation, and, of course, AR filters and effects.

Casual video calls

Sometimes people just want to chat with one another while using fun effects, so the messengers adjusted accordingly.

Snapchat video calls have been popular for a while now. The app now has 319 million daily active users, and the number is growing. The company actively pushes its filters, for example by letting users make their own "lenses" or selling branded effects.

Live streaming

This can include anything from gaming marathons to live shopping. In all cases, the filters have their use.

For example, Restream lets users show live feeds on several social networks at once, delivering 8+ million broadcasts per month. And they have AR filter capabilities, although they use integration with Snapchat to deliver them.

A live-streaming app built with Amazon IVS and Banuba Face AR SDK can also layer real-time beauty filters and AR masks directly onto the broadcast feed, without the broadcaster running a separate effects pipeline.

How do you plan a video chat with filters?

Now let's get to the actual app. The first step is defining the type of app, its target audience, and its positioning.

You probably already know what kind of software you want to create (e.g., a live streaming service). But it is important to see other options in case the business environment changes or your expectations turn out to be different from reality. YouTube used to be a dating service before it became the video hosting powerhouse it is today.

Specifying a target audience is a way to narrow down the niche you want to operate in. This will help you make informed decisions about design, marketing, feature set, etc. For example, Discord and Slack are both messaging platforms with similar functionality. But they cater to different audiences (gamers and professionals) and so have different approaches.

Positioning is a way to distinguish yourself and show users your unique offering. For example, TikTok could have been another social network, but it showed something no one else did before: a fully automated video feed. This is something that can make or break your app.

How do you monetize a video chat app?

In the vast majority of cases, the app you are working on will be a for-profit venture. As such, you need to think about how it will earn you money. These are the most common options:

• Advertising. If you want to keep the product free for users, you can sell ad space. Note that filters and effects can be a form of advertising too. For example, Snapchat sells branded filters.
• Freemium. Keeping advanced features behind a paywall. Certain high-level filters can be one of the paid options.
• License. Selling temporary or lifetime licenses is very common in both B2B and B2C software. This is a way to ensure a steady income stream.
• In-app purchases. Filters can be turned into a commodity and sold to users.
• Cut of donations. One of the monetization options used by TikTok is taking a cut of the donations to content creators that use their platform.
• Free. If your app is not intended to directly earn money, you can keep it free entirely.

How do you integrate face filters with an SDK?

In the vast majority of cases, building such AR technology from scratch is unviable. Firstly, it is very expensive: you need to keep a team of skilled developers on hand for at least six months to even build an MVP. If you are a startup, this is a huge drain on your budget, and if your main focus isn't technology, this is plain unviable. Secondly, you will need time to actually get everything working, which slows down your entry into the market.

The more reasonable choice for such situations is using a ready-made face filter SDK, a module that already performs everything you need and can be integrated with your app within a couple of days. It is a complementary SDK, as it doesn't provide the video call transport itself but enhances it. For apps focused on appearance-enhancing effects, Banuba also offers a dedicated beauty AR SDK that covers skin smoothing, virtual makeup, and other real-time beautification tools out of the box.

This is how the video conferencing API integration is handled. The first step is the same for all platforms: to get the SDK and a token for a 14-day free trial, send us a message through the contact form.

Android

This is what you will get from us:

• Banuba Effect Player (compiled Android library project with .aar extension),
• Android project folder (src) which contains demo apps, located in src/app and src/offscreen respectively.

You will also get a trial token that should be placed here: src/app/src/main/java/com/banuba/sdk/demo/BanubaClientToken.java.

Requirements:

• Latest stable Android Studio
• Latest Gradle plugin
• Latest NDK

Then use the guide to build and launch the demo app, or the integration instruction to connect the SDK to a project.

iOS

This is what you'll get in the archive:

• Banuba Effect Player XCFramework (BanubaEffectPlayer.xcframework),
• BanubaSdk Xcode project,
• Effect examples in the effects folder.

The token should be placed here: src/BanubaSdk/BanubaSdkApp/BanubaSdkApp/BanubaClientToken.swift.

Requirements:

• iPhone devices 6+
• iOS 11.0+

You can find what you need in the demo app instruction and integration guide.

Web

Requirements:

• The latest release of Banuba SDK Web AR
• Banuba client token
• git client
• Node.js installed
• A browser with support for WebGL 2.0

The instructions can be found in this document, and you can do it in just a few lines of code. Note that on the web, the core Banuba SDK WebAssembly module has a 12 Mb minimum size, which affects page-load speed and bandwidth, so factor it into your budget when comparing an SDK against an in-house build.

Desktop

Requirements:

• Latest stable Xcode on macOS
• Latest stable Microsoft Visual Studio on Windows

The SDK archive will contain:

• bnb_sdk folder with dynamic libs and include files,
• third folder with third-party libraries,
• resources folder

Then follow the steps in this guide to integrate the SDK.

Conclusion

Building a video chat with filters isn't easy, both from a technical and a business point of view. But it becomes much simpler when you buy the AR layer instead of building it: a face AR SDK adds background replacement, beautification, and AR masks to an existing video conferencing API in about a week, runs on-device across mobile, web, and desktop, and keeps effects smooth at a minimum of 30 FPS. With this guide, you can get started with an AR chat on any platform. And if you need assistance, feel free to send us a message.