Expression-detecting Robots Assist Humans with Everyday Tasks

Researchers at Franklin & Marshall College have developed a tabletop robot that can understand some forms of human expression. Using artificial intelligence and computer vision, the robot can detect through eye movement, motion, and speech patterns when a human needs guidance or assistance with a task. 


Article Key

Image credit: Franklin & Marshall College

A few decades ago, robot assistants were limited to the imagination, science fiction books, and movies. Cartoons such as the Jetsons featured a robotic maid and housekeeper named Rosey, who rolled around the home on wheels and could show different emotions. Fast forward to 2022, when service robots including the Roomba® from iRobot can clean floors, robotic lawn mowers trim grass, and room service robots at hotels deliver food, toothbrushes, and towels to guests.

Today, advances in artificial intelligence (AI), coupled with machine vision technology and sophisticated sensors, give developers the means to create task-oriented robots that can add real value for their end users. Robots can not only help people with domestic tasks, but they can also sit side-by-side with humans and detect and predict when help is actually needed. These robots are not exactly like The Jetsons’ Rosey, but are becoming more responsive like a human assistant might be.

Next-Gen Robotic Assistants

Robotics developers are now making remarkable progress on the technology needed for more intelligent robotic assistants. A team at Franklin & Marshall College is developing a robot that can read human expressions and behavior, giving it the ability to determine when–and how–to help people. 

The tabletop-sized robot can read cues from humans, including eye gaze, motion, and speech. It uses those cues to know when to assist a person with a task and also when to remain silent, if their assistance is not really needed.

The student and faculty researchers at Franklin & Marshall have designed their socially assistive robot, “Misty,” to help people with a variety of everyday tasks, including cooking dinner and doing homework. The robot doesn’t provide actual physical assistance. Instead, the idea is to provide conversational and verbal direction and assist the user by offering guidance only.

Man in mask directing robot by smartphone

Image credit: Franklin & Marshall College

Using Vision and NLP

Misty’s functions are built on the Misty Robotics development platform and hardware. With a high-resolution camera, Misty has built-in, on-the-edge face detection and face recognition capabilities. Developers can get size and positional information that they can use to code Misty to follow the face she sees. That gives the Franklin and Marshall College team the ability to create a variety of greeter skills, conversational skills, and security skills. 

Misty has a built-in array of three far-field microphones that she can use to record audio and get positional information about the sounds in her environment. With beam-forming, Misty can filter out sounds (such as music) that the robot detects at an unspecified angle. 

Misty uses a native end-of-speech detection engine to automatically record until the user stops speaking. Developers can code Misty to send the audio file to a natural language processing (NLP) service—such as Google’s Dialogflow—and use Misty’s text-to-speech engine or a third-party TTS service to create deep voice interactions between Misty and the users.

A Helping Robotic Hand

Instead of replacing people doing the tasks, the robot is programmed to supplement human effort. Misty is designed to strike a balance between helping too much or offering too little assistance. The robot detects patterns of a human’s non-verbal communication, which can then prompt the device to speak up.

In particular, the robot can recognize eye gaze patterns, such as a glance or stare. Franklin and Marshall College researchers are also currently working on giving Misty the capability to analyze facial expressions to detect feelings of boredom, engagement, or frustration, which could help in education and learning for students. 

  • Watch a video to learn about how the Franklin & Marshall College robot can provide assistance in response to user needs.
  • Learn more about the Misty Robotics technology.
  • Listen to the podcast on the future of robotics with Savioke Founder and CEO Steve Cousins.


Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries.