Handshapes—distinctive hand configurations used to form signs—are the building blocks of any sign language. But despite their linguistic importance, many AI models for processing sign language rarely model them explicitly.
Among the challenges in AI hand shape recognition are that the same handshape can appear in different configurations in one video, and a single sign can comprise multiple handshapes.
Alessa Carbo, a third-year computer science student working with Eric Nalisnick, an assistant professor of computer science, has taken an important step toward addressing this challenge. She and Nalisnick developed a neural network to improve the computational recognition and translation of signed languages; Carbo is first author of their findings, which were published in the Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
Their approach could give researchers new insights into how people use sign languages and help pave the way for enhanced linguistic analysis and better communication technologies for the Deaf and hard-of-hearing communities.
Johns Hopkins computer scientists previously demonstrated that modeling handshapes can improve American Sign Language translation accuracy by 15%. Carbo and Nalisnick wanted to see if they could improve on that by separating a hand’s static orientation from a signer’s motions over time.
To do this, they used a graph neural network divided into two submodels: One looks at how the hand’s shape changes over time during a single sign, while the other searches for video frames that show a handshape in its most recognizable orientation. Testing this approach on a new dataset of annotated ASL videos, they found that separating time dynamics and hand configuration improves handshape recognition significantly.
“And because handshape is a fundamental parameter of all sign languages, our model can be a useful tool for sign language linguists,” Carbo says. “Say a linguist has a large collection of videos of people signing and wants to find all occurrences of a particular handshape—our model could allow her to quickly and automatically find those video segments.
“We’re excited that our work could help with both AI applications and researchers trying to find new insights about how people use sign languages around the globe,” Carbo says.
— JAIMIE PATTERSON