Real-time Background Subtraction in Unity

Bring privacy and fun to your Unity3D face tracking apps and games with our real-time background subtraction technology for Unity.

unity background subtraction banuba

Our Face AR SDK for Unity now supports background segmentation along with face detection and tracking, allowing users to replace backgrounds in real-time. 

The best part? The background changer is compatible with face filters and supports multi-face tracking. You can build cross-platform games with face tracking, video chats and augmented reality apps in Unity or even use this feature for mixed reality projects.

Why Background Subtraction in Unity

Unity is the world's most popular cross-platform game engine. However, developers still face a range of limitations when it comes to building face tracking games and apps in Unity. One of them is the ability to separate a person from the background in real-time.

Our background remover for Unity is designed specifically for selfie-focused experience. It works accurately with all skin tones and hides whatever is behind the person in real-time. You can change user background into a green screen, replace it with background images, animated wallpapers or videos per your choice.

unity background subtraction banubaBackground Subtraction in Unity_Banuba Demo

The existing Unity chroma key tools select the color to be removed, e.g. green used in physical green screens. Developers need to avoid colors that match the skin tone so it won’t blend into the background. Our Unity background segmentation is based on the neural network trained to recognize whether a pixel belongs to the class 'person' or 'background', enabling automatic real-time background separation.

Technical Features

We tested how background subtraction works with other Face AR features in Unity both on early and latest devices (iOS, Android) and the results are pretty good.

Unity for iOS, FPS

  iPhone 5s iPhone Xr
Background 21 30
Background + Face Filter 18-20 30
Background 2 faces 16-19 29-30

Unity for Android, FPS

  Galaxy A 50 Galaxy S 10+
Background 24-30 29-31
Background + Face Filter 17-19 29-31
Background 2 faces 25-29 29-30

Background Subtraction Use Cases 

The demand for virtual backgrounds has grown fast within recent months, becoming a number one feature for any face tracking and video chatting app. Here are some ideas which Unity apps benefit of real-time background replacement feature.

Games with face tracking

You can build face-to-face games in Unity where players can see each other or themselves while playing. Players may have fun changing backgrounds, applying face filters or even full face avatars if they don’t want to show their faces. 

Video chat apps 

Video chat apps are gaining more popularity. By adding Face AR features, you can make the user experience more immersive and fun, allowing users to change boring backgrounds into scenery per their choice.

Live streaming

Virtual backgrounds for streaming apps bring digital privacy and comfort, letting users not bother about their real surrounding. Users can easily create high-quality streams teleporting themselves into the center of the video featuring their favorite game, movie or just abstract animation.

Unity Chroma Key (Green screen) 

Our background subtraction technology can be perfectly used as an alternative to Unity Chroma Key tools to create mixed reality and augmented reality videos and experiences. Developers can record the user without the background and then overlay video animation to get high-quality video clips.

Building cross-platform AR apps in Unity? Discover Face AR SDK  possibilities.

Start Free Trial