AI-Based Vision Applications in the Classroom

Engaging students in class—especially during remote learning—can be a challenge for teachers. Artificial intelligence can help teachers be more aware of when their students lose focus and new devices can encourage student interaction.

Image credit: The City College of New York

Best practices in education and approaches to teaching are constantly evolving. Now more than ever we have the opportunity to reevaluate the learning environment. Shuttered schools forced educators to shift to remote learning. A full year later, and with increased investments in digital learning platforms, online learning likely will remain an integral part of our academic environment, even as schools open their doors to students for in-person classes.

The pandemic is providing artificial intelligence researchers with a unique opportunity. When classrooms shifted to Zoom rooms and students began learning online, researchers realized they had a very clear ability to correlate student engagement with student success. Unlike in-class learners, online students are separate, they face the same direction, and their eyes are trained on the screen and camera. That makes them good candidates for visual assessment.

A-Eye Tracking Indicates Engagement

Researchers at the City College of New York tapped into this phenomenon to study student engagement during online learning by monitoring eye movement. Using standard web cameras, the researchers were able to predict test scores of online students by analyzing the students’ eye movement. The cameras captured the eye activity of more than 1,000 students, without requiring any personal or uploaded data from the student computers. The researchers found that students with similar eye movements during an online presentation were more attentive to the lesson and performed better on tests related to that material.

Zoom classes pose challenges for teachers at all grade levels. In addition to potential technical issues, teachers often struggle to gauge how well their remote students grasp a given topic. In a physical classroom, a teacher typically can sense when students are restless or don’t understand a lesson, and they can adjust on the fly. That kind of interaction is very difficult in an online setting.

The CCNY research indicates that teachers, armed with eye movement data, can adapt their teaching style as needed. Teachers can monitor student focus and take a break if their attention wanes. This approach also could be applied to the entertainment or advertising industries.

Cautious Adoption

One consideration with AI-based assessments is to ensure diversity in the pool where machine learning takes place. Just as facial recognition software performs better in certain demographic groups, educational AI applications are likely to skew in favor of wealthier students and schools, where camera-enabled laptops are readily available. That could inadvertently impact the algorithms that determine when people are focused on the message or lesson.

While AI can play a role in the classroom, it isn’t a replacement for human instruction. Most researchers have relinquished the idea of fully automated teachers, choosing instead to find technologies that aid teachers. One critical reason is that AI works best when teaching repetitive lessons or memorization rather than creative, non-structured lessons. That works well for memorizing multiplication tables but not for multi-step calculus problems or collaborative learning, though some scientists are tackling that.

Tools for Teaching

A group of researchers from the US National Science Foundation (NSF) AI Institute for Student-AI Teaming is exploring the use of AI-based teaching “partners” that will encourage collaborative learning for classroom use. These partners could take the form of a notebook computer with a camera and microphone or be an interactive avatar that assesses student language, gestures, and facial expressions.

The research team will employ multimodal machine learning to develop social signal processing algorithms that can identify who speaks and when and can assess the emotion of each person. The AI Partners can nudge quiet students into participating in a class discussion or keep the discussion on topic.

Research shows that AI can be a helpful tool for teaching and student learning. As new technologies are designed for student instruction, researchers should be mindful of how they will be incorporated in the classroom.