Building an ASL translator for video and media

I am a Computer Engineering student at the Rochester Institute of Technology (NY). Apart from being a great technological school, it also has the biggest college institute for deaf students in the United States of America. Naturally, I get to interact a lot with deaf and hard-of-hearing students.

Last November, I had the opportunity to take part in an inter-college hackathon hosted by my college, with students from the tri-state area. Being in the environment I was in, I understood a lot of deaf students wanted to access media in a way that felt natural to them. Yes, a large chunk of media had subtitles, but from what I understood, American Sign Language felt more natural to deaf and hard-of-hearing students. That became the foundation of our team's platform. We started brainstorming on how we could come up with a way to achieve our goal. After a few hours, we decided to develop an ASL translator for YouTube videos.

To bring this concept to life, we had to architect a multi-stage data pipeline within the hackathon timeframe, handling everything from data scraping to video processing.

Here are the technical details.

Data warehousing: we built a custom web scraper to parse and download hundreds of English-to-ASL translation videos. This raw data was cleaned to build a proprietary relational data warehouse, mapping specific English tokens and phrases to their corresponding ASL video clips.

Real-time NLP pipeline: we engineered a script to hook into YouTube's backend to extract video transcripts. The text was parsed and normalized in real time, setting up the framework to match spoken English with our ASL video database.

System integration via Python: using Python as the backbone of our backend, we integrated the transcript extraction tool with our data warehouse. The logic analyzed the transcript timestamps and queried the database to retrieve the correct ASL clip, dynamically stitching the translation together alongside the original media.

Cloud infrastructure: to handle the high I/O demands of video processing and dataset querying, we deployed our backend on Google Cloud. We utilized GCP to host our ASL video repository and manage the real-time transcript data, ensuring low-latency retrieval.

Multilingual expansion: after optimizing the core Python pipeline, we had a few hours in hand, so we used our remaining time to integrate a multilingual translation API. This scaled the architecture from a single-language tool to a multi-format media translator.

When it came time to present, we successfully pitched our working prototype to the judges. More importantly, we were able to demonstrate the tangible, positive impact our platform could have on users. My team ended up winning first place at the hackathon.

Anyways, thanks for reading, and have a great day.