AI Skin Analysis: Shade Matching, Skin Tone Detection and Try-On

[navigation]

TL;DR:

• AI skin analysis reads skin tone, face shape, and visible imperfections from a photo or live camera feed, then maps that profile onto a product catalog.
• Shade drift, meaning a color that renders correctly on one skin tone and wrongly on another, is the single most common complaint beauty product teams raise about virtual try-on.
• Banuba TINT works with all skin tones, holds up in dim lighting, and includes photo upscaling for low-quality images, which are the three conditions where naive color matching breaks.
• Photo-based analysis suits diagnostic and recommendation flows; live video suits color try-on, because the shopper can move and judge the shade under their own lighting.
• Processing location is a compliance decision, not just a performance one: Banuba Face AR SDK runs entirely on-device with no cloud round trip, while Banuba TINT's skincare recommendation layer is cloud-based.
• Banuba's TINT catalog carries 22,000+ digitized products, and new collections can be added in under 48 hours.
• Océane, a Brazilian cosmetics manufacturer and retailer, moved its add-to-cart rate from the 3% industry average to 20.15% in the first month with TINT, and later peaked at 32%.
• A typical Banuba TINT widget rollout launches in under two weeks, which matters more to most product managers than any single accuracy metric.
• Real-time acne detection in Banuba Face AR SDK is explicitly cosmetic, not medical diagnosis. Any vendor claiming diagnosis is making a regulatory claim, not a product claim.

What is driving AI adoption in the skincare market?

The skincare industry, valued at $181.2 billion globally in 2023, according to Statista, is being reshaped by the integration of artificial intelligence. The Covid-19 pandemic caused tangible damage to the beauty market, with up to 20-30% revenue loss. While masks led to decreased demand for colored cosmetics, the lockdown motivated consumers to take more conscious and advanced care of their skin. An even skin tone won the fight over the filtered, foundation-covered look, and skincare took budget share from color cosmetics as a result.

As consumers became more deliberate about what they apply to their skin, the industry had to adapt. Besides active components and sustainability awareness, individuals are now mindful of specific skin concerns and seek products that cater to their own needs and conditions. They no longer settle for generic skincare products but demand tailored solutions and a hyper-personalized approach, driven by heightened awareness of acne, dark spots, signs of aging, and overall skin health.

Historically, individuals relied on generic skincare regimens. They were told that a radium-based cream would give their skin radiance, which it did, but there was a nuance. When the market was not overfilled with offers, and people had little access to information other than marketing pitches, the generic approach seemed practical.

With the advance of social media, information sources, and self-awareness, consumers realized there is no magical pill to fit them all. AI disrupted that model by offering recommendations based on individual skin types, needs, and conditions. This transition from one-size-fits-all to customized skincare regimens is a genuine shift in how beauty products are sold. And while an average visit to a dermatologist costs around $221, leveraging a software-based match is a far cheaper first step for the shopper.

How does the skincare industry use artificial intelligence?

Three jobs account for most production deployments: analyzing the face, recommending products against that analysis, and rendering the predicted result so the shopper can see it before buying.

Why does AI-powered skin analysis improve personalization?

Because it removes the guesswork the shopper cannot resolve alone. McKinsey found that 71% of consumers expect companies to deliver personalized interactions, and 76% get frustrated when they don't. In skincare, personalization used to be possible only in an offline store or through a dermatologist consultation.

Banuba's AI skin analysis sits inside TINT, the company's virtual try-on platform for skincare, accessories, and makeup. Its algorithms analyze the user's face to determine imperfections and recommend products that could help address them, then demonstrate the predicted result on the person's face. The flow is deliberately short: TINT takes a photo from the gallery or the face of the person on camera, the user answers a few questions about their skin condition, and the recommendation AI matches that data to the products in the database.

The engine looks at more than color. Banuba's AI makeup recommendations layer detects skin imperfections, determines whether the person's hair has been dyed, analyses face shape, and runs seasonal color analysis automatically while the try-on tool is opening. Personalization at that level is worth up to 200% better conversions and up to 30% higher average order value, and it cuts return rates by up to 40%.

In addition to analysis, augmented reality in TINT gives a preview of both short-term and long-term effects of using a product. Shoppers often feel blindfolded when buying skincare online. As with makeup virtual try-on, TINT makes it possible to try a solution, get guidance, and effectively replace a shop consultant.

How do virtual dermatology consultations use AI?

They pair a software pre-screen with a human specialist, so the clinician spends time on judgment rather than intake. Through AI, individuals can have virtual consultations with board-certified dermatologists from home, which removes geographical constraints and shortens the wait for advice.

AI's role in skin diagnostics for dermatology research is expanding rapidly. It is being utilized for skin cancer detection, analyzing skin biopsies, and predicting disease progression, and it helps researchers develop treatment options for conditions from acne to psoriasis by analyzing large volumes of clinical data.

Piction Health is a virtual dermatology clinic built around this pattern. A patient answers a few questions and sends pictures of their skin concern; the algorithm performs an analysis against a database of over 500k case images, and a dermatologist reviews the result, typically within two days. Once the issue is determined, the patient receives a treatment plan and product recommendations.

This is a useful boundary to hold in mind for commerce. A retail try-on tool is not a diagnostic device, and the two should not be marketed as if they were the same thing.

How are skincare brands putting AI into devices?

The skincare devices market is growing at a compound annual rate of 12% until 2030, and AI is what separates a connected gadget from a cleansing brush with a motor.

What do smart skincare devices actually analyze?

Mostly hydration, texture, and a composite "skin age" score, refreshed on a schedule. Luna Fofo by Foreo scans the skin, rates its condition on a scale of 100, estimates skin age and reports hydration levels, then builds a cleansing routine from that data. Users repeat the scan every two weeks to track progress.

AI-powered LED masks work on a different axis, using blue and red light therapy to stimulate collagen production and target acne-causing bacteria, with the treatment adjusted to the user's skin type. The Tri-Light +SABI AI by Skin Inc. pairs a device with a companion app that also factors in water intake, sleep, stress, and physical activity, on the reasonable premise that skin health is not decided by topical treatments alone.

How does AI speed up cosmetic product development?

By compressing ingredient screening. AI analyses large datasets of ingredients and their effects across skin types and conditions, which shortens the path to a formulation that addresses a specific concern while limiting questionable ingredients.

Industry players including L'Oréal and Novi use AI to power ingredient databases and manage chemical data from suppliers at scale. Platforms such as the Good Face Project apply the same approach to regulatory compliance during formulation. The same models also help predict emerging trends from consumer preferences, research output, and market dynamics, which is how brands decide what to develop next rather than what to develop faster.

Banuba's makeup virtual try-on example 

What does a personalized treatment plan look like?

A profile, a routine, and a feedback loop that updates both.

What do AI skin diagnostic apps measure?

Everyday diagnostic apps report on visible attributes such as puffiness under the eyes, dark circles, hair thinning, and similar markers, and they store the history so a user can see change over time rather than judging from memory.

How do prescriptive skincare regimens adapt over time?

The goal is a prescriptive routine that changes as the skin does. AI-enabled feedback loops let users track progress and adjust, rather than following one static regimen indefinitely. Skincare brands including Olay and La Roche-Posay already run smart skin analysis on their own websites and apps, feeding regimens and product recommendations back to the customer.

What should you evaluate before choosing a skin analysis vendor?

Five questions separate vendors that work in production from vendors that demo well.

How do you prevent shade drift across skin tones?

You prevent it by testing color rendering on the extremes of the tone range and under bad lighting, not on the mid-range faces in the vendor's demo reel. Shade drift is what beauty product teams describe as "the product color is not displayed correctly", and it is the top complaint against existing virtual try-on tools.

Three product properties decide the outcome. First, color rendering has to be built to interact with any skin tone rather than being tinted onto a generic base, which is what "real-looking products on all skin colors" means in practice. Second, the system needs to hold up in dim lighting, because shoppers use their phones indoors in the evening. Third, it needs photo upscaling for low-quality images, since a compressed selfie is the normal input, not the exception. Banuba's skincare AI and try-on platform documents all three.

Accuracy claims in this category are also easy to inflate, because "accuracy" is used to mean several different measurements. It is worth reading what the individual skin-analysis accuracy numbers actually measure before comparing two vendors' figures side by side.

Photo vs live-video try-on: what are the trade-offs?

Photo analysis is better for diagnosis and recommendation. Live video is better for color judgment.

A still photo gives the model a stable, high-detail frame, which suits skin-condition analysis and a recommendation flow where the user is willing to wait a second or two. It also works from the gallery, so the shopper does not need to grant camera access to get a result.

Live video lets the shopper move, tilt, and check the shade under their own lighting, which is the only way a color decision actually gets made. The cost is that every frame has to be processed in real time, so the performance and platform requirements are stricter. Banuba TINT covers both, working on photos, videos, and live streams.

The practical rule: if the output is a recommendation, a photo is enough. If the output is a purchase decision about a color, the shopper needs live video.

Which computer-vision approach fits skin-tone detection?

Dense mesh tracking, not sparse landmarks, if color placement matters. TINT uses patented face tracking with 3,308 vertices, which is more precise than landmark-based tracking. The difference shows up exactly where the complaints are: at the boundary of the lips, along the lash line, and on the cheek where blush has to follow the curve of the face rather than sit on a flat plane.

Where the analysis needs to be continuous rather than a single pass, the work moves into the SDK layer. Banuba Face AR SDK handles real-time acne detection, highlight, and removal as a native feature, holding 30 FPS on mid-range Android and equivalent iOS hardware even with other effects such as background replacement and makeup running in the same stack. Integration is a single API entry point, and the developer documentation is at docs.banuba.com, with a browser reference implementation in the beauty-web repository.

One boundary is worth stating plainly, because it is a compliance question rather than a technical one. Banuba's acne feature is designed to stay clearly cosmetic and visual, and not to drift into unsupported medical diagnosis. A vendor that markets skin analysis as diagnosis is making a regulatory claim, and it should be treated as one during procurement.

How do you stay privacy-compliant with face data in beauty apps?

You decide where the frames are processed before you decide anything else, because that choice determines what you have to disclose and what you have to store.

Two architectures are in play. On-device processing keeps camera frames on the phone, with no cloud round trip, which is why Banuba Face AR SDK keeps working offline and suits privacy-sensitive deployments such as kiosks and verification flows. Cloud processing sends the image to a server, which is how TINT's skincare recommendation functionality works, and it is what makes the same experience reachable from smartphones, tablets, desktop computers, and in-store smart mirrors without shipping an app.

Neither is automatically the right answer. On-device is the stronger privacy posture and the easier consent conversation. Cloud is what allows a heavy recommendation model and a 22,000-product catalog to stay in one place and update without a client release. What matters is that the choice is deliberate, documented in your privacy policy, and matched to the jurisdictions you sell into.

How do lipstick shade-recommendation engines work?

They build a color profile of the person, then rank the catalog against it rather than filtering it.

The sequence is consistent across serious implementations. The system captures the face, derives attributes including skin tone, face shape, feature proportions, and whether the hair has been coloured, assigns a seasonal color type, and then sorts products so the closest matches surface first. Banuba's engine does the sorting while the try-on tool is still opening, so the shopper never sees an unranked grid. The shopper then confirms with a try-on, which is the step that converts, because the ranking is a hypothesis and the render is the evidence.

Catalog depth is what makes ranking meaningful. A shade-matching engine over 40 SKUs is a filter. Over 22,000 digitized products, with new collections added in under 48 hours, it is a recommendation system.

How do the main approaches compare?

What's in it for a business?

AI changes the skincare category for both consumers and businesses. Brand owners and retailers should remember that it is still a data-driven approach: artificial intelligence, whether in a device or in software, needs data to learn, and it also hands back analytical information that improves the business. The three main benefits, setting aside competitive advantage, are:

• Personalization lifts sales. According to McKinsey, companies that personalize customer experiences see revenue increases of 5-15%. In skincare, that effect is amplified, because the purchase depends on a match to an individual concern.
• Engagement improves. The AI-powered skincare products reviewed above all provide round-the-clock assistance in place of a shop consultant or chat operator, which is what closes the gap McKinsey describes when 71% of consumers expect a personalized interaction and most brands cannot staff one.
• Insight compounds. AI processes large volumes of behavioral data quickly, which surfaces preferences and emerging trends. According to Adobe, companies using data-driven insights are 1.5 times more likely to report revenue growth of 15% or more.

The proof point most product managers ask for is a live deployment rather than a benchmark. Océane, a Brazilian cosmetics manufacturer and retailer, reached a record 32% add-to-cart rate with TINT. The pilot covered only concealer and foundation, and within the first month the add-to-cart rate for those items moved from the 3% industry average to 20.15%, an increase of over 600%. Demand outran the stock they had prepared. The rate later peaked at 32%, meaning a third of the shoppers who tried a product online put it in their cart.