Nestle Puts the AI in Baking

Artificial intelligence and digital animation result in an autonomous virtual person that offers branded outreach and customer interaction. This season’s holiday bakers can get virtual assistance when and where they need it, and (hopefully) perfectly delicious cookies.


Article Key

Image credit: Soul Machines

You know Dasher and Dancer and Prancer and Vixen, but do you know Ruth? She’s the digital “cookie coach” that uses artificial intelligence to help holiday bakers. Developed by Nestle USA with Soul Machines, the digital avatar is named after Ruth Wakefield, who created the iconic Toll House Cookie recipe for which Nestle is famous. Ruth uses conversational AI and reads facial expressions to help bakers on the spot.

The multimodal cookie bot is Web-based, so users can access Ruth from any location, no special apps needed. Home bakers can speak to Ruth online using voice or chat bot interactions. Users can also see images, watch videos, find recipes, and get detailed baking instructions. Ruth can answer questions about how to customize cookies (do you prefer crispy or chewy?) and make ingredient substitutions to meet gluten-free, dairy-free, and other dietary restrictions.

If things aren’t going well in the kitchen, the cookie bot also offers “Cookie First Aid” to help bakers remedy problems on the fly or after the fact. She offers pro tips and suggestions on how to bake the perfect cookie for your palate.

One Smart Cookie

Nestle collaborated with Soul Machines to create Ruth. The Soul Machines team creates life-like “Digital People” that can talk to and interact with humans in a natural manner. To develop the avatars, Soul Machines uses streamed video and audio, which is integrated with the company’s Human OS. The Human OS is a cloud-based artificial general intelligence (AGI) platform with a patented “Digital Brain” and autonomous animation technology. Clients then use Soul Machine’s Digital DNA Studio to customize the visual image and voice characteristics of their Digital Person.

The Soul Machines technology enables its avatars to see and hear users through a computer’s cameras and microphones. The technology supports natural language processing (NLP) and can assess a user’s facial expressions and tone of voice to offer empathy and more natural interactions. The software supports 12 languages and multiple dialects.

Soul Machines supplied Nestle with the NLP engine that enables the conversational AI and supports speech-to-text from the user or text-to-speech from Ruth. The team is working to expand Ruth. In addition to developing her back story, future versions will support increased personalization and communication in multiple languages.

Female chatbot and chocolate chip cookies

Image credit: Nestle

Something to Chew On

Similar in concept to the Butterball Turkey hotline, Nestle decided to create a customer-facing avatar to engage consumers and answer frequently asked questions. Ruth launched in February 2021 after two years of research and collecting data to fuel the AI engine.

About 70 percent of the calls fielded by Nestle’s customer service line relate to making the iconic Toll House cookie, but responses varied based on the expertise and experience of the operator. Nestle wanted to improve consistency and free its call center to handle other concerns. Average interactions range from nine to 13 minutes, and users are asked to provide feedback after their interactions to assess and improve Ruth’s capabilities.

Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries.