Get Free Demo
Wanna give our technologies a whirl?
  • It takes 7 min to install our SDK demo kit. Support documentation and guides provided!
  • Have other questions? Need more info or consultancy? Email us! We'd love to hear from you.

Tutorial: How To Optimize and Test 3D Face Filters

Share
optimize 3d face filter

A high-quality 3D face filter needs to look visually appealing and run smoothly on a wide range of devices. Therefore, AR mask optimization and testing are the 2 most important stages in its creation process. Learn how to optimize and test your 3D face filters for better performance in your app. This post provides general rules for designing graphical files of your AR effect, technical guidelines for its optimization and QA checklist.

NOTE: The guidelines below refer to working with Banuba Effect Constructor which is our proprietary AR effect construction engine. It allows designers to make custom AR face filters and test them in real-time before uploading to  mobile apps or websites built with our Face Filter SDK

Explore Face Filters SDK

What impacts 3D face filter performance

Before we look at how to optimize your 3D face filter, let’s see which components impact its performance.

Textures

Textures are assigned to a 3D object to add information about its color, brightness, and roughness. Basically, there most commonly used types of PBR textures in 3D face filter include:

    • Base color to define the overall color of the material.

    • Metallic to define how “metal-like” the surface is.

    • Roughness to define the sharpness of the reflection.

    • Ambient occlusion to define the shadowing properties.

    • Normal to add bumps and dents and create detailed shadows and highlights of the surface.

    • Emissive to define areas of the textured material that will illuminate.

When designing your 3D face filter, you don’t need to use all of the textures. To display PBR materials, you need to use a set of textures which transfer the surface of the model in the rendering engine. The most optimal set includes 4 textures:

textures 3d face filter

These textures are generally used to create most types of AR face filters ensuring a realistic representation and a nice user experience with AR effect in your app. However, this set is just a recommendation. The final number of textures in your effect may differ depending on its type and task. For example, for non-PBR materials, you can use just one BaseColor texture, while in other AR filters you may need an additional Emissive texture.

Polygons

A polygon mesh is a collection of vertices and polygons that defines the shape or surface of 3D objects in your AR filter. The fewer polygons you have, the "lighter" your effect is, however, polygons have less impact on the overall face filter performance compared to texture resolution. For example, you can increase the number of polygons and reduce the size of textures to achieve greater effect productivity.

Making 3D face filter: General Rules & Tech Guidelines

When you make your 3D face filter, you need to consider its performance on different types of devices — both powerful and low-end ones. While "heavy" effects run perfectly fine on iPhone X, their performance on low-end Androids may be laggy, which results in poor user experience. In some cases, you can even make two versions of one AR effect with different sizes of textures and properties for devices with different capacities.

Generally, 30 FPS performance on iPhone 5s level devices provides a quality user experience. To achieve it, we recommend adhering to the following technical guidelines:

  • Keep the total resolution for textures in one scene not more than 1024x1024px for each type of texture. For example, you can have four sets of textures for PBR where each texture is 512x512px in size.

  • Use the video resolution not more than 512x512px. If other filter parameters are well-optimized and the performance allows it, you can go with more. 

  • For beauty filter or beautification layer, we recommend using textures of the following sizes: softlight_layer (256x256px), eye_highlite (768x768px), normal_layer (512x512px)

  • The total number of triangles in one scene should not be over 50,000 and your 3d object should be divided into several files. You may have 100,000 triangles and AR filter will work fine if textures are small. The number of vertices in one FBX file cannot exceed 65536, however, you can bypass this restriction by loading a scene with several FBX files.

How to optimize 3D face filters

1. Reduce and optimize filter textures


To reduce the memory load and optimize AR face filter performance in your app, you need to use the texture resolution which provides the minimum acceptable quality level. In other words, it needs to be as small as possible yet still, allow for a good-looking effect. Additionally, textures within one 3d model can have different resolutions. For example, Roughness or Metallic textures have less impact on the final rendering, so you can make their resolution smaller.

To get your effect load faster, particularly with large textures, it is important to have textures in KTX format. This is the native format for the GPU, and therefore it loads into memory quicker without conversion.

To get textures in KTX format, place them in the SRC folder when assembling your AR filter with the Effect Constructor. It automatically converts texture files into KTX format so your final effect will load faster.

Your source files must be in PNG format and with a specific suffix.

effect constructor ui

2. Optimize beautification 


The Beautification layer may come as a standalone filter or as part of your AR try-on or entertainment effect. It significantly improves the overall face filter experience, retouches user face imperfection and ensures a professional and harmonic AR face filter. 

Read more: How To Create Face Beauty Filter 

To reduce the memory load, you can also optimize your beauty filter. If your softlight_layer or normal_layer beauty textures are symmetrical, you can cut off one part.

beautification texture optimize

After that, in the effect folder find and edit the retouch.frag file

1. Find and uncomment the following line in your effect, in our example, it's *145 (in your effect, еру line number may differ)

   vec2 uvh = vec2(abs(2.0 * (var_uv_bg_uv.x - 0.5)),var_uv_bg_uv.y);

2. If you use the right part of softlight_layer texture

-find the SOFT_LIGHT_LAYER block, line *148 (in your effect, the line number may differ) 

-replace the var_uv_bg_uv.xy variable with the uvh variable uncommented above

-the new line will look like this:

    res.xyz = blendSoftLight( res.xyz, texture( tex_softLight, uvh ).xyz );

If you use the right part of normal_layer texture

- find the NORMAL_LAYER block, line *152 (in your version of the effect, the line number may differ)

- replace the var_uv_bg_uv.xy variable with the uvh variable uncommented above

- the new line will look like this:

    vec4 makeup2 = texture( tex_normalMakeup, var_uv_bg_uv.xy );

3. Optimize and organize polygons


Generally, the fewer polygons you use, the more productive your effect is. However, polygons have less impact on the effect performance than texture resolution. Therefore, you can achieve greater productivity and improve your 3D model details by increasing the number of polygons and reducing the size of textures.

The number of vertices in one FBX file cannot exceed 65536, however, you can bypass this restriction by loading a scene with several FBX files.

head model with 3d face filter

Also, you can optimize the geometry so that the front-facing surfaces located closer to the camera had a denser topology, and you can completely omit the invisible polygons. This way, you save on the number of polygons and optimize resources without compromising on the filter experience since the user always sees the front part.head model with 3d face filter optimized

Polygons with a transparent shader especially affect the AR filter performance, especially if they are arranged in layers following one by another, for example, as in hairstyles or virtual wigs.

You can test this impact by bringing the camera closer to your face so that polygons with a transparent shader occupied the maximum screen area. You'll see the productivity decreases as the size of the polygon area increases. That's why it's important to understand the principle of polygon rendering and draw order when working with transparency.

The real-time engine draws polygons based on the position of the shader in the draw order starting from top to bottom.

effect constructor draw orderInside one shader, the drawing starts with a polygon which has the lowest sequence number and goes to a polygon with the higher number.

polygons in 3d face filter

A polygon with a higher serial number or lower position in DrawOrder overlaps one with a lower serial number or a higher position in DrawOrder, even if it's placed farther from the camera. The transparency of this higher polygon will be drawn on top and erase the preceding polygons located closer to the camera.

You can organize polygons as follows:

  • In any 3D editor, copy an object to its polygons with the maximum polygon numbers assigned. You can copy objects in any order and then delete the old objects to get the required polygon order. 

  • In any 3D editor, when you combine polygons into one object, the numbering will correspond to the order of the attached objects. 

  • In Houdini modelling software, you can use Sort SOP node for sorting primitives. 

  • In Blender software, you can sort primitives by Z-axis using Miscellaneous Editing Tools

4. Optimize skin files

The real-time engine supports a maximum of 4 influences per one point.

The number of Bones in one FBX file cannot be more than 256. You can bypass this restriction by loading the scene with several FBX files.

5. Optimize video files


video textureFor animated textures, you can use an MP4 video file instead of a sequence of images. It allows getting a significantly higher frame rate with a relatively small size of the file. Our filters achieve 30 FPS for 3d animation and video textures. You can read more about Animated texture and billboard and converting to this format.

The rendering engine limits the number of video files allowed for simultaneous use in one effect. You can use only one video file for textures, another one for the foreground and one more for the background. If your AR effect needs more video textures above that, you can bypass this restriction by composing all video textures in one file.

How to test your 3D face filter

When testing a 3D face filter, you need to keep an eye on its performance, user experience and device compatibility. Here’s a short checklist for AR face filter testing, however, each filter requires a unique testing approach with more or fewer actions needed to estimate the quality of its performance.

Performance iOS, Android

  • Real-time AR face filter — at least 30 FPS on iPhone 5s, 6, 6plus | Samsung s8/Huawei P10/Mi6 and higher

  • AR mask and video recording — at least 30 FPS on iPhone 5s, 6, 6plus | Samsung s8/Huawei P10/Mi6 and higher

User experience

  • You can take a photo or/and video with the effect

  • The effect works stably on the front and rear camera

  • Effects with a distinctive feature which involves user action e.g. trigger, tap, swipe or voice changer have a hint about it (first couple of seconds).

  • The effect works as required with face in the camera, no face, and disappearing face. 

  • You can take a photo or/and video with the effect

  • The effect works stably on the front and rear camera

  • Effects with a distinctive feature which involves user action e.g. trigger, tap, swipe or voice changer have a hint about it (first couple of seconds).

  • The effect works as required with face in the camera, no face, and disappearing face.  

  • Effects with analytics perform correctly (the name of the effect matches, the functional properties match)

  • Effects are stable and perform correctly at different angles (90 degrees rotation, 180 degrees rotation (upside down))

  • Check how the masks stretch on the face (wide open mouth, eyes, grimacing) to make sure that no holes in the mask appear and no element overlap one another

Device compatibility

  • Check effects on Xsmax for crashes (some effects fall on Xsmax)

  • Check effects on Xsmax for incomplete filling of the screen (sometimes the effect does not fill the entire screen with high resolution)

ALSO READ: Tutorial: How To Create AR Mask in Effect Constructor

Tags
Share
Next up for you
Augmented Reality Video Editing: 12 Features For Developing Your Next TikTok App
February 24, 2020

The term ‘influencer’ has been a buzzword just a few years ago, but now many young people consider it as a career. As the demand i...  

#AR Video Editing
How To Create Face Masks in Unity Face AR Plugin
February 18, 2020

Banuba Face AR plugin for Unity is an easy-to-use tool allowing developers to create realistic Snapchat-like lens, AR face filters...  

Unity Face AR
Introducing Unity Face Tracking Plugin to Create Snapchat-like Filters in Unity
February 18, 2020

Our Face AR extension for Unity brings Banuba’s 3D face tracking technology cross-platform and allows developers to create realist...  

Unity Face AR
Join the conversation