The headlining feature in the new update for Banuba Face AR SDK is hand gesture recognition on mobile devices. However, it is mostly focused on improving the performance across all platforms. Thanks to extensive optimization, the users of Windows PCs, Android, and iOS will feel a major uptick in quality. Read the article to see the details of how we did it and the specific benefits!
Desktop, Android, and iOS: gesture recognition
Now Banuba Face AR SDK can recognize several gestures: Palm, Like, Victory, OK, Rock. The software tracks the hand, creates a skeletal model and uses it to detect specific gestures.
While unassuming at first glance, this feature has a lot of potential applications. The hand tracking can let people interact with AR items (e.g. picking them up, throwing them, etc.). Virtual objects can be attached to the hand and react according to the movements, similar to the AR avatars. Or the gestures can serve as a contactless interface, allowing the user to give commands by waving their palm and making various signs - just like in sci-fi.
This feature is currently available on mobile devices and desktop computers but will soon be expanded to other platforms as well.
Android: better device detection
Banuba Face AR SDK is compatible with the vast majority of Android-based smartphones. While this is clearly advantageous, it also creates an issue of ensuring the best performance on both cheap and high-end devices. To address it, we separated the devices into three tiers (low, middle, and high) and had the SDK adapt to each tier.
Before we used to do it by detecting the CPU of the smartphone. Now we have begun to focus on the GPU instead, as the testing showed this allows for more reliable separation and better performance.
iOS: lighter neural networks
We have conducted additional tests and found out that the more complex neural networks that we use for high-end iOS devices don’t really outperform the more simple ones. So we have cut them from the build and decided to only use the lighter neural networks.
E.g. for background segmentation, instead of 4 of them, the SDK only includes two, as they are used in pairs: one for landscapes and one for portraits. This decreases the size of the SDK and the apps it is integrated with.
Windows: MSMF camera
We have rewritten the code responsible for managing the camera and its output. The new one is based on Chromium rather than OpenCV that was used before. As a result, the camera now loads the CPU by 50% less. This should let users run more applications simultaneously without loss in performance.
Bug fixes:
- tflite_runner assorted fixes
- iOS: White eyelashes when making photo
- iOS: Time range issues in the video player
- macOS: Crash on M1
- CubemapEverest test effect autorotation
- WebAR: broken SIMD support
- WebAR: Fixed creation of MediaStreamCapture
- Android: Multitouch crash in the demo app
- Unity: min version of Android SDK
- Unity: Android build on Windows
- Win32: Background NN work
- Incorrect lips shine work
Just a friendly reminder: trying out Banuba Face AR SDK costs nothing for two weeks, so seize the opportunity!
-
The official technical documentation for Banuba Face AR SDK is available online and includes requirements, setup instructions, and configuration details:
- Banuba Face AR SDK iOS documentation
- Banuba Face AR SDK Android documentation
- Banuba Face AR SDK Flutter documentation
- Banuba Face AR SDK React Native documentation
- Banuba Face AR SDK Web documentation
- Banuba Face AR SDK Unity documentation
- Banuba Face AR SDK Desktop documentation
This Banuba Face AR SDK documentation should be your primary reference during integration.
-
You can access official integration code samples for the Banuba Face AR SDK on GitHub. Banuba provides platform-specific examples to help you integrate faster:
- Banuba Face AR SDK iOS integration code sample
- Banuba Face AR SDK Android integration code sample
- Banuba Face AR SDK Flutter integration code sample
- Banuba Face AR SDK React Native integration code sample
- Banuba Face AR SDK Desktop integration code sample
- Banuba Face AR SDK Web integration code sample
- Banuba Face AR SDK Unity integration code sample
These Banuba Face AR SDK repositories include setup instructions and minimal working examples for quick deployment.
-
Yes. The Banuba Face AR SDK includes an LLM-friendly documentation file designed for AI-assisted development and internal knowledge systems:
This structured Banuba Face AR SDK text file can be used with AI tools to streamline onboarding and developer workflows.
-
The Banuba Face AR SDK provides official cross-platform plugins to simplify integration into hybrid applications:
These Banuba Face AR SDK plugins are maintained by Banuba and support faster implementation for cross-platform apps.
