The use cases for AR filters mostly describe a single person on screen. This is not always the case: what if the user needs to replace the background for a group picture or apply multiple augmented reality masks for a fun social media collab? This is where multi-face tracking technology comes in. In this article, we will describe its nuances and show how to implement it in your mobile app.
Peculiarities of multi-face tracking
Face tracking is the ability to segment out a person’s face in a picture or a video frame. This is not to be confused with facial recognition: tracking implies software seeing just a face, while recognition means that the application can attribute this face to a specific person.
As the name suggests, multi-face tracking implies that the software is able to see several people on one screen. Then you can work with them just as with a single face. However, there are certain nuances you need to be aware of.
The first one is performance. The algorithm will be running separately for each face on the screen. This could quickly overwhelm a less powerful system. For desktop devices or powerful smartphones, this is less of a problem, as they have resources to spare. For low-end mobile phones or web AR tracking many faces could be challenging - you can notice up to a 50% decrease in performance.
The second is physical limitations. It is hard enough for a human to see someone in the dark, especially if that someone is hiding behind something. The software has the same issue. For example, Banuba Face AR SDK can work with up to 70% occlusion - one of the best results on the market. Yet even it can’t see through objects.
Fortunately, face tracking is very scalable. As long as you have enough processing power, you can sic the algorithms at as many faces as you need. Though for mobile devices we recommend tracking up to 3 - it’s enough in the vast majority of cases.
Multi-face tracking app type
This technology has a wide range of applications.
- Videoconferencing. Users will be able to add beautification or face masks to everyone in the picture or replace the background for multiple users without loss in quality.
- Social media. For making viral videos involving groups of people. Or adding a bit of flavor to live streams.
- eCommerce. So that people can virtually try things on with friends.
- Photo/video editing. To spice up the pictures in AR photo booths, change various aspects of a family portrait, or act as a green screen for a fan-made movie.
- eLearning. For the cases when there are multiple people presenting.
These are just the basics. Given enough time and creativity, you can definitely use it in other ways. However, when aiming to build a multi-face tracking app or trying to create AR app, this will help narrow down the type of app you want.
Example of beautification in Banuba Face AR SDK
Advantages of multi-face tracking
Augmented reality brings its own set of benefits. However, if your app is able to work with multiple people, you will gain a few extra ones:
- Gain competitive advantage. When you have something that your competitors don’t, you can always mention it in your advertising. Moreover, the users will spread this bit of knowledge themselves, which means you can attract more people to your app.
- Promote content creation. Giving people more opportunities to express their creativity is a major factor in increasing the amount of user-generated content. The more videos and pictures you have, the longer will your users stay in your app, thus raising your metrics.
- Increase sales. When you let couples try outfits together, they will be more inclined to buy them.
Conceptualization
Once you know the type of app you are going for, it’s time to narrow down the niche. There are two factors to consider: audience and message.
Firstly, you need to figure out the people you intend to serve. For example, TikTok and Likee are similar in their offering (video-based social media) but the latter focuses more on younger audiences. Depending on the target audience, you can choose the fitting design approach and advertising directions.
Secondly, you need to choose the message your app sends. The main point of it is to become more specific in your offering. For example, Discord and Slack are both messengers with similar feature sets. However, the former positions itself as a gamer-first app, while the latter targets businesses.
This might change later depending on the business climate. YouTube used to be a dating service, after all.
Background replacement is a popular AR feature
Monetization
This is another major part of any application. Monetization models are rather common and can be used in many kinds of apps. Moreover, they can sometimes be combined.
- Ads. Pretty straightforward: people watch the ads, you get paid. The more users you have, the better. Advertising can take many forms, from banners to branded AR filters. And they can let you keep the app free.
- Freemium. Most features are free to use, while some of them (e.g. multi-face tracking or other advanced functionality) require payment to be used.
- Paid. You can simply charge money for your app. Either a one-time payment or a subscription could work.
- In-app purchases. There are different ways you can approach this. For example, you can sell various in-app items (e.g. face filters) to users.
- Free. Perhaps you need the app to promote a product, attract people to a charity, or fulfill other goals that aren’t directly bringing profit. In this case, you don’t need a monetization model at all.
Multi-face tracking SDK
Face tracking isn’t easy to develop. The technology is complex and requires a lot of specialized knowledge to build from scratch. That’s assuming you can even find the team that can pull it off and in the case you have a few hundred thousand dollars to fund the R&D efforts.
For small and medium enterprises, as well as large companies that don’t have software as their main business, this is either unviable or unreasonable. So they turn to augmented reality SDKs - pre-made modules that can perform specific functions. They don’t require as much upfront investment (a license payment is a few orders of magnitude lower than the development budget), already have an expansive and proven feature set, and can be integrated quickly. With an SDK, you will be able to go to market several months earlier.
For the purpose of this article, we will assume that you are going to choose this option.
Integration
We will use Banuba Face AR SDK as an example. It has solid multi-face tracking, which is useful in many domains.
The first step is the same for all platforms. To get the SDK and the token to try it out, send us a message. The free trial lasts for 14 days - enough time to check out the features you are interested in.
Android
You will receive the following:
- Banuba Effect Player (compiled Android library project with .aar extension),
- Android project folder (src) which contains demo apps, located in src/app and src/offscreen respectively.
You will also get a trial token that goes here: src/app/src/main/java/com/banuba/sdk/demo/BanubaClientToken.java.
Requirements:
- Latest stable Android Studio
- Latest Gradle plugin
- Latest NDK
Then follow the guide to build and launch the demo app or the integration instruction to connect the SDK to a project.
iOS
The archive you will receive contains the following:
- Banuba Effect Player XCFramework (BanubaEffectPlayer.xcframework),
- BanubaSdk Xcode project,
- Effect examples in the effects folder.
Token should be placed here: src/BanubaSdk/BanubaSdkApp/BanubaSdkApp/BanubaClientToken.swift.
Requirements:
- iPhone devices 6+
- iOS 11.0+
You can find what you need in the demo app instruction and integration guide.
Web
Requirements
- The latest release of Banuba SDK Web AR
- Banuba client token
- git client
- Nodejs installed
- Browser with support of WebGL 2.0
The instructions can be found in this document. You can do it in just a few lines of code.
Desktop
Requirements:
- Latest stable xCode on MacOS
- Latest stable Microsoft Visual Studio on Windows
The SDK archive will contain:
- bnb_sdk folder with dynamic libs and include files,
- third folder with third-party libraries,
- resources folder
Then follow the steps in this guide to integrate the SDK.
Conclusion
Multi-face tracking can be a cool and flexible addition to many audiovisual apps. With the right SDK, it can be quickly integrated into your project and begin giving you all its benefits. So if this is something you need for your next app, drop us a line and start testing!
