Spreading the Joy of Makeup to Everyone: Development Story of KOSE’s MR Makeup Simulator 'Color Machine'
Konnichiwa! My name is Yuki. I am a tech project manager at BASSDRUM and I have a huge passion for makeup and cosmetics.
In the beauty industry, the latest technologies are being transformed into various services and products. Driven by my enthusiasm for this rapidly evolving field, I felt compelled to write and share my insights on the intersection between beauty and technology.
Today, I'll be featuring KOSÉ's MR Makeup Simulator ‘Color Machine,’ which is gaining attention for its use of projection mapping technology, allowing users to easily try makeup on their own faces. After trying it, I had the opportunity to speak with Ms. Kaoru Ohishi, a researcher at the company who was involved in conceptualizing the project and in charge of its technical development, to learn more about the development process.
《Color Machine》
The makeup simulator installed at Maison KOSÉ Ginza in Tokyo. Developed jointly by KOSÉ Corporation Research Laboratories and the Watanabe Laboratory at Tokyo Institute of Technology, it utilizes 3D high-speed tracking projection mapping technology, allowing anyone to try various makeup styles easily without physical contact. Unlike typical makeup simulators that display a virtually made-up appearance on a flat screen, this technology projects makeup such as eyeshadow, blush, and lipstick directly and three-dimensionally onto the customer's face. This enables a realistic color makeup experience that can't be achieved on 2D virtual makeup. It follows facial movements and expressions in real time, offering a new sensation of 3D virtual makeup as if it has been applied directly onto one's skin. A distinctive feature is the ability to check the made-up face three-dimensionally from various angles.
《Kaoru Ohishi, Ph.D.》
Kaoru joined KOSÉ Corporation in 2015 and is part of the Makeup Product Research Laboratory. She has been engaged in research, as well as the creation and development of makeup items, being extensively involved in everything from basic research to product development. Subsequently, she has worked on the development of new materials, formulations, and content, and is currently responsible for the development of the MR Makeup Simulator Color Machine.
《KOSÉ Corporation》
KOSÉ Corporation, a Japanese company established in 1946, has been engaged in the business of creating beauty, primarily focusing on cosmetics. It operates with the philosophy of creating unique values and culture through the advanced integration of "intelligence" and "sensitivity." Leveraging its distinctive combination of tradition and innovation, KOSÉ is continuously building its future.
The development story of 'Color Machine'
Yuki: In recent years, there has been a rapid development and spread of content and services utilizing digital technology in the beauty industry. However, when it comes to applying makeup to an actual human face, like with Color Machine, there seems to be a scarcity of such content examples. What was the inspiration behind the creation of Color Machine, which combines makeup with projection mapping?
Kaoru: We are a cosmetics company, so we didn't start with the goal of creating projection mapping content. The trigger was our desire to address the challenge of "How can we bring out the individuality of each person?" in this era where diversity is emphasized.
One of the areas that KOSÉ's research labs particularly focus on is "light." For instance, we study how changing the color pattern of light reflected on the skin with foundation can alter personal impressions. While researching technologies related to light, we stumbled upon the high-speed projection mapping research being conducted in the laboratory of Professor Yoshihiro Watanabe at the Tokyo Institute of Technology. It was then we came up with the idea of applying it to makeup in some way.
Yuki: After discovering the technology in Professor Watanabe's lab, how did the collaboration for joint research come about?
Kaoru: Around November 2019, We attended a symposium where Professor Watanabe was speaking, specifically to meet him. The actual decision to engage in joint research was made in April 2020, and Color Machine was launched in 2022.
This was a quick development within just two years. We were also unsure to what extent content combining such new technology would be accepted, so we thought it best to have customers experience it as soon as possible and update it based on their feedback.
Yuki: I'm interested in the project that brought together members of the Watanabe lab, who research digital technology, and members from KOSÉ, who focus on cosmetics and makeup. It was a team formed by members from completely different industries. What was the scale and composition of the team like?
Kaoru: Surprisingly, it was a very small team. Only four researchers from KOSÉ were directly involved in this project. In addition, there was Professor Watanabe and one student from his lab, making a total of six people. This team of six handled everything from development to proposing a service that involved beauty consultants in its operation.
Yuki: Your team not only worked on the development but also handled the aspects of user experience beyond that, it seems.
Kaoru: We considered using AR (Augmented Reality) or VR (Virtual Reality), but in the context of beauty, being able to experience makeup on one's actual face is extremely important. Also, KOSÉ not only has strong research and development capabilities but also prides itself on the skills of its beauty consultants. We focused on creating an experience that emphasizes the fusion of real and digital by choosing MR (Mixed Reality), which allows trying makeup on one's actual face, and also by involving beauty consultants in the process.
Yuki: Before this interview, I had the opportunity to try it myself. Initially, I was excited by the colorful and fun space of the Color Machine booth. However, as I proceeded to choose the type of makeup with a beauty consultant, I gradually became captivated by her ability to make suggestions. While the technology embedded is certainly impressive, I felt that it is not just an independent content but a tool that allows the beauty consultants to showcase their skills.
Kaoru: In the early stages of development, we considered full automation, but we were concerned that it might not be satisfying for customers. We also wanted to better utilize the excellent suggestion skills of our beauty consultants. For example, customers may want to try various makeup looks, but doing so with actual makeup can be time-consuming. However, with this technology, we can propose multiple makeup patterns in a short time, enhancing customer satisfaction. Additionally, during the development, the pandemic made it impossible for beauty consultants to touch customers' faces directly, and we strongly felt the need to solve such challenges.
Yuki: You mentioned that the development took two years. Could you please describe the general stages and process that the project went through to reach practical application?
Kaoru: We didn't have a clear plan or phases as such. Our sole directive was to complete the project as quickly as possible. In this context, we first developed the technology to express makeup using light, such as creating eyeshadows and blushes and their blending techniques. Additionally, the tracking mapping technology, which didn’t pose much concern when projecting onto T-shirts in the university lab, became problematic when projecting onto faces. The tracking speed wasn’t sufficient, making the projected makeup look unnatural, and making it necessary to develop faster tracking technology.
Once these component technologies took shape, we proposed internally to incorporate Color Machine as a beauty attraction at Maison KOSÉ, where various KOSÉ products can be tried out. Subsequently, various individuals joined in to provide support, including Maison KOSÉ's facility managers and UI/UX specialists and designers, who helped to shape the overall experience.
Yuki: Who else supported you in making updates to the project?
Kaoru: The makeup simulation didn't look natural with just one adjustment, so we repeatedly discussed and explored the best combinations with our makeup artists and beauty consultants. There are makeup types available to try like 'Spring, Summer, Autumn, Winter,' and these designs were also created by our makeup artists. They helped select which of our products would be best for replicating these looks in actual makeup. Human intervention was essential for this part of the project. We worked on this adjustment phase without compromise until about three days before the launch.
Yuki: It has been over a year since the launch in August 2022. How has the customer response been?
Kaoru: At the time of its launch, Covid was still widely spreading, so we were unsure about the outcome. However, thankfully, from the launch until now, we have always been fully booked. Customers who have actually tried it have given positive feedback, saying things like, "I discovered colors that surprisingly suit me, which I had never dared to try before" and "I discovered a new version of myself." Despite the quick launch, the system has been operating smoothly, even while undergoing maintenance.
Yuki: This project is a successful example where researchers typically involved in developing cosmetics have embraced technology to solve problems, brilliantly achieving everything from planning and development to practical application.
Kaoru: As researchers who have continually created cosmetics, we have become aware that simply making products isn't enough to deliver our carefully crafted items to customers, especially with the abundance of products available. Color Machine was born out of our consideration of how to integrate technology with reality to deliver cosmetics and communicate the allure of beauty products. When asked, "How did you accomplish this?" our response is always, "We achieved it through passion."
Yuki: The achievement of not only developing but also perfecting the experience in just two years is truly remarkable. When I tried it myself, I was amazed by the meticulous attention to detail. The presence of a full-length mirror allowed me to check the makeup with my outfit, and the distance from the small mirror for viewing my face was just right.
Kaoru: Great attention was indeed paid to the placement of every single mirror. We set up the distance and angle so that customers could see themselves clearly, along with lighting that enhances their expressions.
In addition to the spatial design, particular care was also given to the technological development that affects the experience. For instance, the way light appears differs when projected onto a white screen versus human skin. To address this challenge, we developed a unique color correction formula. Initially, Color Machine was set up with a color correction suited for Asian skin tones, specifically common Japanese skin tones. However, by the time of our presentation at the IFSCC (33rd Congress of the International Federation of Societies of Cosmetic Chemists), we had completed a correction formula that could project appropriate colors based on the reflective properties of all skin tones.
Yuki: Rather than fine-tuning for each skin tone, do you have a correction formula that adapts to all skin tones?
Kaoru: We have been conducting research on light for some time and have published results categorizing skin tones into six distinct types based on their characteristics. This research focuses on the spectral reflectance of the skin. We developed a color correction formula that calculates how colored light would appear when applied to each of these six types of skin. As a result, we were able to provide a highly satisfying MR makeup simulation for people with various skin tones.
Yuki: I heard that you exhibited at CES this year, the world's largest consumer electronics trade show held annually in the United States. Looking at the photos from the event, it seems that the makeup provided there was different from the makeup designs offered at Maison KOSÉ.
Kaoru: When it was decided that we would exhibit at CES, we asked our makeup artists to come up with new makeup designs incorporating international trends. Color Machine allows for updating makeup designs. We didn’t only prepare for the general public, there was also a focus on show business applications, providing unique designs that go beyond everyday makeup.
Yuki: Makeup trends change very quickly, so having a versatile system that can update designs is truly fantastic.
Kaoru: The great thing about projection mapping is not only that anyone can easily try it with just one touch, but also that the designs can be easily changed.
Making Makeup Enjoyable for Everyone
Yuki: I think a major appeal of using projection mapping technology is that it eliminates the need for touch-ups by beauty consultants, allowing one to try various makeup looks multiple times in a short period without putting any strain on the skin.
Kaoru: We have received feedback from men who usually do not wear makeup that 'Color Machine' makes it easy for them to try makeup without feeling uncomfortable, and there have also been cases where couples enjoyed it together.
Yuki: May I ask about the future prospects for Color Machine?
Kaoru: Currently, the majority of our users are Japanese and other Asians in Japan, but eventually, we want to make it enjoyable for people of all ages and skin tones around the world. Our company upholds the keywords of 3G: Global, Gender, and Generation, in our approach to 'customer creation'. We are committed to delivering the allure of makeup to everyone. We aim to encourage those who are hesitant to try makeup or want to discover a new side of themselves, proposing something that they can enjoy.
Yuki: Creating products accessible to everyone is indeed the essence of KOSÉ's philosophy.
Technology Boosting the Beauty Experience
Yuki: With the maturation of digital technology, it is anticipated that beauty experiences and products utilizing technology will increase in the future. What impact do you think the spread of beauty tech will have on the industry?
Kaoru: Digital technology offers many benefits, such as reaching people who previously had no access to it, and providing services through just a smartphone to those who were unable to try or enjoy makeup. I believe digital technology is a tool that allows customers to first learn about makeup, enjoy it, and feel it as something close and personal.
From a corporate perspective, another advantage is gaining a deeper understanding of customers through data analysis and similar activities.
Yuki: Thank you very much for your cooperation in the interview today.
[Editor's Note]
Color Machine Facilitates New Self-Discovery Through Makeup
Writing this article, I myself have now come to enjoy makeup, but initially, I had no idea where to start. I repeated many mistakes, quickly concluding that certain styles didn't suit me and hesitating to try new makeup for a long time.
There are as many makeup combinations and methods of expression as there are faces and as many desired self-images as there are cosmetics. Nowadays, with the abundance of makeup tutorials and cosmetic reviews available on social media and YouTube, anyone can learn about makeup. However, I believe there's no makeup experience quite like trying it on your own face and feeling the excitement and self-love that comes with it. If I had found Color Machine when I first started exploring makeup, I might have discovered the joy of it much sooner.
Color Machine is a beauty attraction offering a novel experience where you can discover a new version of yourself, not just through makeup that suits you but also through styles you've never dared to try before. Furthermore, the realization of the project team's vision for Color Machine – making makeup enjoyable for everyone – will likely encourage many who have wanted to experiment with makeup and self-expression but have not been able to do so.
Makeup is not just about enhancing physical appearance or beauty; it has the power to make one fall in love with themselves, inspire courage to try new things, and even change the way one sees the world. I'm really excited about the future content and services that will emerge as KOSÉ engages with its 3G approach (Global, Gender, Generation).
Author / Yuki Narumi
Tech Project Manager
Born in Aomori prefecture. Co-founder and producer of TDSW. In 2018, Yuki co-founded Tokyo Developers Study Weekend (TDSW), a node-based visual programming community, while she was in university. They invite artists and developers from Japan and abroad to provide a subscription service for live content and tutorials that expose and archive their knowledge of technology and expression. The aim is to create a community where people who are interested in manufacturing can learn about exciting technical expressions and connect with people who are interested in the same area. In July 2021, Yuki joined BASSDRUM.
Translated by Chiemi Isozaki
Special thanks to Astrid (Kejora Karimunjawa)