Blog
Unity

Real-time Background Subtraction in Unity: Full Guide

Unity is the world's most popular cross-platform game engine.

However, developers still face a range of limitations when it comes to building face tracking games and apps in Unity.

One of them is the ability to separate a person from the background in real-time.

In this post, you'll learn how to create a background Unity subtraction featuring:

- Why is Unity background segmentation beneficial

- 4 use cases for background subtraction

- How to subtract background in Unity with Banuba.

unity background subtraction banuba

[navigation]

TL;DR:

  • Background subtraction is useful for games, video calling apps, and live streaming software;
  • It can be integrated using Banuba's Face AR plugin for Unity;
  • See the step-by-step installation instructions below.

Why Background Subtraction in Unity

Being the most popular cross-platform game engine, Unity still provides limitations.

One of them is real-time virtual background which is crucial for building face tracking games and apps in Unity. 

Our background remover is designed specifically for a selfie-focused experience. It works accurately with all skin tones and hides whatever is behind the person in real-time.

Our Face AR SDK for Unity now supports this feature along with face detection and tracking, allowing users to replace backgrounds in real-time.

Power Your App with AR Face Filters SDK  Start Free Trial

What's more, the backdrops are compatible with face filters and support multi-face tracking.

You can build cross-platform games with face tracking, video chats, and augmented reality software in Unity or even use this feature for mixed reality projects.

You can place the user in front of a green screen, and replace it with images, animated wallpapers, or videos.

unity background subtraction banubaBackground segmentation in Unity by Banuba

The existing Unity chroma key tools select the color to be removed, e.g. green used in physical green screens. Developers need to avoid colors that match the skin tone so it won’t blend into the background.

Our Unity background segmentation is based on the neural network trained to recognize whether a pixel belongs to the class 'person' or 'background', enabling automatic real-time background separation.

Unity Background Segmentation: Technical Features

We tested how background removal works with other Face AR features in Unity both on early and latest devices (iOS, Android) and the results are pretty good.

Unity for iOS, FPS

  iPhone 5s iPhone Xr
Background 21 30
Background + Face Filter 18-20 30
Background 2 faces 16-19 29-30

Unity for Android, FPS

  Galaxy A 50 Galaxy S 10+
Background 24-30 29-31
Background + Face Filter 17-19 29-31
Background 2 faces 25-29 29-30

4 Background Subtraction Use Cases 

The demand for virtual backgrounds has grown fast in recent months, becoming a number one feature for any face tracking and video chatting app.

Here are 4 common use cases when Unity background subtraction becomes a must-have feature for your app.


Games with face tracking

You can build face-to-face games in Unity where players can see each other or themselves while playing. Players may have fun changing backgrounds, applying face filters, or even full-face avatars if they don’t want to show their faces. 

Hyperspeed AR video chat examplesHyperspeed | Face Tracking Case Study


Video chat apps 

Video chat apps are gaining more popularity. By adding Face AR features, you can make the user experience more immersive and fun, allowing users to change boring backgrounds into scenery per their choice.

Power your app with Background Changer  Start Free Trial


Live streaming

Background replacement for streaming apps brings digital privacy and comfort, letting users not bother about their real surroundings. Users can easily create high-quality streams teleporting themselves into the center of the video featuring their favorite game, movie, or just abstract animation.


Unity Chroma Key (Green screen) 

Our technology can be perfectly used as an alternative to Unity Chroma Key tools to create mixed reality and augmented reality videos and experiences.

Developers can record the user without the background and then overlay video animation to get high-quality video clips.

Integration

This is how you can integrate a background replacement SDK in your Unity project. It will also allow you to use other AR features, if you wish.

1. Obtain a trial token

This is simple. Just send us a message through a contact form.

Power Your App with AR Face Filters SDK  Start Free Trial

Run

  1. Clone the repository
  2. Download the latest libs package BanubaSDK-quickstart.unitypackage and import it using Unity.
  3. Put your Client Token to the Assets/Resources/BanubaClientToken.txt
  4. Open the Assets/Scenes/SampleScene and click Run.

Import To Your Project

  1. Download the latest assets package BanubaSDK-import.unitypackage and import it using Unity.
  2. Put your Client Token to the Assets/Resources/BanubaClientToken.txt
  3. Open the Assets/BanubaFaceAR/Demo and click Run.

Note:

if you use BanubaSDK-import.unitypackage you need to add BanubaFaceAR/BaseAssets/Scenes/LoaderScene.unity for Windows and Android Build and set Scene for loading like on the image below:

loader_scene.jpgBanuba Face AR SDK Unity integration

FAQ
  • The purpose of background subtraction is to detect and separate moving objects in video or image processing. The most common use cases are AR-powered face tracking apps, video chat apps, and live streaming. 
  • The background subtraction algorithm in computer vision means detecting moving objects in video content. It separates objects from the background utilizing so-called classes which identify the background and input images.
Top