Image credit: StoryFile
The combination of artificial intelligence, natural language processing, and volumetric cameras are enabling the creation of holographic images of people who can share stories and respond to questions through natural conversations, even when they are in another state or have passed away.
A desire to preserve history led Heather Maio Smith to develop an AI-based technology that would capture the stories of Holocaust survivors for future generations. She teamed up with the University of Southern California’s Shoah Foundation, which has recorded interviews with more than 55,000 survivors and was initially founded by Steven Spielberg after filming Schindler’s List.
Smith’s idea was to make the interviews interactive, so people could converse with the survivors as if they were in the same room. Technical advances would make it possible, but it would require a vast database and artificial intelligence.
Archivable Interactive Interviews
Smith pulled together resources from the Shoah Foundation and USC’s Institute for Creative Technologies (ICT). ICT had the technology to capture 3D images with language-processing and voice recognition capabilities that would support interactive conversations. The first interview was with Holocaust survivor Pinchus Gutter.
The week-long interview sessions took place under a dome rigged with 6,000 LED lights and more than 20 cameras. Although holographic technology wasn’t readily available at the start of this project, the forward-thinking team took video from all angles to support 3D imagery down the line. Gutter answered hundreds of questions, each of which was isolated, categorized, and manually entered into a database searchable by machine learning algorithms.
After testing, tweaking, and tagging—think of the myriad ways a single question can be posed—the AI technology has proven itself. Today, holograms of Gutter and at least 20 other survivors can be seen at the Illinois Holocaust Museum and other institutions. They share their stories and answer questions posed by museum goers.
From Schindler’s List to Star Trek
The success of the initial holographic interviews led Smith to found StoryFile. The company’s proprietary technology, Conversa, combines 3D volumetrics video with conversational AI to capture the stories of public figures and everyday people, including an astronaut, a 911 operator, a civil rights leader, and Star Trek actor William Shatner.
StoryFile interviews can be viewed in 2D or 3D, on home, mobile, or augmented reality screens. The company claims to have recorded nearly 1,700 profiles and even created Ask Santa, an interactive experience for kids stuck at home during the pandemic. Future uses will likely include dating apps, recruitment, and educational or medical training.
Image credit: StoryFile
Chatting From Beyond
Speaking to the departed led Microsoft into this space as well, though it uses a chatbot rather than a hologram. The company’s recently approved patent uses 3D motion images, voice reconstruction technology, and personality traits that it culls from social media platforms to develop a “personality index” of an individual. That index is used to train the chatbot, which converses with the living.
The software can analyze photos, videos, letters, electronic messages, and other personal data to capture the character of that specific person, even the deceased. The chatbot taps AI and machine-learning to assess “crowd-based perceptions” and “psychographic data” to fill the gaps in the personality index. Microsoft’s technology also can generate a “voice font,” based on voice recordings and speech synthesis. Photo analysis will allow for 3D images.
Microsoft filed the patent in 2017, and it was awarded in January 2021. Future iterations could even support chatbots of one person in different stages of life. The company has no current plans to develop it into a product, but it underscores the capabilities and potential uses of artificial intelligence.