For millions of visually impaired people, board games can be difficult to navigate, often requiring help from others to track pieces, interpret visual cues, and follow gameplay. A pastime meant to encourage social connection and strategic thinking can instead become frustrating and inaccessible.
To meet this challenge, Johns Hopkins electrical and computer engineering students Joon Cho and Hyewon Jung developed Envisioning Play, presented at the Whiting School of Engineering’s Design Day. Using the classic board game Sorry! as a prototype, the system uses a camera to monitor gameplay and converts visual information into touch and audio cues that help visually impaired players participate more independently.
Cho and Jung were motivated by a broader lack of inclusive recreational experiences for visually impaired players and wanted to apply engineering to a social setting often overlooked in accessibility design. “Most existing solutions require constant assistance,” says Jung. “We wanted to create something more interactive.”
At the system’s core is a fine-tuned YOLOv26n machine learning model that tracks the board, player hand movements, and piece locations. To train it, the team filmed multiple games and manually annotated video footage frame by frame by labeling game pieces, hand positions, and board changes. This custom dataset taught the system how to recognize gameplay patterns and interpret board states as the game unfolds.
Once that information is processed, Envisioning Play translates it into coordinated non-visual feedback. A dynamic tactile display creates a refreshable, braille-like map of the board, a vibrating wristband helps guide players to the correct piece or destination, and audio cues provide gameplay instructions and updates.
Integrating machine learning, game logic, and multisensory outputs into one seamless platform presented major technical hurdles. “The biggest challenge was training the computer vision model and integrating the game logic for a successful guidance algorithm,” says Cho. By combining perception, rules tracking, and responsive feedback, the team created a closed-loop system that transforms visual gameplay into accessible interaction.
Because Envisioning Play was intentionally designed as a flexible platform, its potential extends beyond Sorry! “Its generalizable design allows it to be adapted across multiple games and settings, including homes, board game venues, and classrooms.”