Developed by Jeffrey Zhang, Rohan Narayan, Abhishek Mangla.
Many people whose communication abilities are impaired must use Braille to comprehend the world around them, and talk to the world around them. But with Writing2Learn, the options to communicate with the classical languages of our world open up to these people. Through hand motions on a Synaptic TouchPad, Writing2Learn voices out the letter being drawn and also records the letter virtually.
1) The applications of Writing2Learn are limitless. Besides helping the blind and mute, it can be used by teachers to improve their student's retention of letters or symbols for languages beyond English. For example, Mandarin consists of over 8,000 characters and it takes years for native speakers to learn the language, and even longer for secondary learners. But Writing2Learn quickly allows learners to pick up a new language like a flashcard app except it is much more interactive with the user.
2) Another benefit of Writing2Learn is improvement of calligraphy- if the style of writing is unrecognizable by our algorithms, it is highly likely that it is unrecognizable by a human! Hone the art of handwriting without wasting paper or ink. Welcome to the new age of writing.
3) Finally, Writing2Learn can potentially be used as a ground-breaking command-line prompt program for developers because instead of typing out common commands repeatedly like cd, ls, git status, git pull, git push, etc., user of Writing2Learn can integrate his/her own handwritten signature which corresponds to these commands.
We encountered and solved three major challenges.
Playing Audio on a HTML page is simple, but playing audio on a local host's index page is a different story. We had to learn server-side http requests and just being able to play the .wav sound of a particular letter after it is written was difficult.
Perhaps the crux of this product was being able to recognize which letter is being written at a given time. We are taking the image and flattening it into a vector and then pushing it into an automated averaging algorithm which takes many, many image samples of say the letter "O" and determines the characteristics of a handwritten "O".
We are taking the vectors that we created as described in the section to the left and then computing the inner product of the x and y coordinates of every pixel that is affected by the touch on the Synaptics TouchPad. We take the highest inner product and correlate it to the best represented letter or symbol in our dataset JSON file.