How Squirl reads sign language: movement, not video
A plain-language look at how Squirl Signs recognizes American Sign Language from hand, face and body landmarks, and why we don't learn from raw video.
People often picture sign language recognition as a camera that “watches” someone and guesses the word. The reality is a little different, and the difference matters for accuracy, speed and privacy. Here’s how Squirl works, without the jargon.
Step one: find the points that matter
When you sign in front of Squirl, the first model doesn’t try to understand anything. Its only job is to find points on your body in each frame of video: 21 points on each hand, points around your eyebrows, eyes and mouth, and points on your shoulders and arms.
What comes out is a moving constellation of dots. The background, your clothes, the lighting and what you look like are all left behind. The next model only ever sees how those points move.
Step two: read the movement over time
A single frame rarely tells you which sign someone made. ASL signs are defined by handshape, location, movement, palm orientation and facial expression together, and many signs share most of those. The recognition model reads a short window of movement, a second or two, and learns which patterns belong to which signs.
We train it on thousands of short clips from many different signers, and we always test on signers the model has never seen. That second part is important. A model that only works for the people it trained on isn’t useful to anyone else.
The face is part of the grammar
In ASL, the face carries grammar as well as emotion. Raised eyebrows can turn a statement into a yes or no question. Furrowed brows mark questions like who, what and where. A head shake negates a sentence. Squirl tracks these signals alongside the hands, so “YOU HUNGRY” with raised eyebrows can become “Are you hungry?” instead of “You’re hungry.”
From signs to a sentence
ASL has its own word order and grammar, so a word-for-word list of signs doesn’t read like natural English. “STORE ME GO YESTERDAY” means “I went to the store yesterday.” Once Squirl has recognized a sequence of signs, a language step turns them into a natural sentence in English, French or Spanish, and the app can read it aloud.
Why this approach protects privacy
Because recognition runs on points rather than footage, we can improve the model without keeping anyone’s video. Signers who want to help can choose to share recordings of their signing. Nothing is kept without a clear yes, and every thumbs up or thumbs down in the app helps us see where the model is right and where it still has work to do.
What’s next
Our vocabulary is growing with every round of training, and we test each release with Deaf signers before it reaches the app. If you’d like to be part of that, get in touch or join the waitlist.