Experiments: Alternate Controls: Noddinggam - an Interactive Audio Platform (WIP)
Updated: Mar 17
This is a work in progress. Constructive feedback is appreciated.
After a long trip driving, and being very board, I wanted to create an alternate control that could be used to engage my mind while driving. This eliminates the use of eyes and hands, and would probably require minimal interaction. Thinking about it, I realized that there are not many options for "video" games for the blind, or those that can't use controllers for whatever reason. It was this thought that gave me the idea for an interactive audio platform that could be controlled without eyes or hands using Arduinos and sensors mounted inside of a headset.
Misc. Stuff I already have:
Ideation of User Interface:
Noddinggam is to be an interactive audio platform for the blind, busy, or just a little lazy. I am creating a headset that makes it possible to play interactive books and audio games using simple nods of the head. The custom headset and application is designed to interact via Bluetooth on your tablet or smartphone.
Using two sensors gyroscopes and accelerometers, players will be prompted for head motion input during play. There are essentially 6 directions possible: up, down, left, right, tilt left, tilt right. One use example: some prompts ask for yes or no input where players can nod head up and down for "yes" and back and forth for "no." Where more options are available, players can "scroll" through options by tilting their heads left and right. More complex or subtle inputs such as circles, arcs, or otherwise may be possible with testing. I imagine a simple Arduino based microcontroller and sensors mounted in a headset, on a headband or on hair/ear clips will make this possible.
Using voice interaction, players can interact with vocal responses. I imagine using Amazon's Alexa or other voice interaction systems that work in Unity 3D.
Using positional trackers, players can interact with invisible spaces (think overlapping universes or alternate planes in a game maybe, physical memory & sound puzzles, or art experiences with location based sounds). Being akin to VR, I imagine using my HTC vive and a puck sensor to start possibly. Other possibilities might include Leap Motion, or Microsoft Kinect DK hardware. This is much less budget friendly, for developer or user, but has potential to be very interesting. This can be mocked up using HTC Vive without video, which begs the question, should I bother with the little mobile headset or just experiment with 3D audio using the VR headset? I think the answer is no, unless I can figure out a way to track less expensively with less hardware.
Ideation of User Experience:
Beyond being a platform designed with the blind in mind, any player can use Noddinggam to engage their minds or play while doing other things with their hands and eyes, be it cleaning, working, driving or even just relaxing under an umbrella on the beach.
Level of interactivity can be adjusted to activity, but also may depend on the book or game.
Choose-Your-Own-Adventure books lend towards this experience. Games like "Life is Strange" or Telltale type games might also be great candidates. Perhaps even sims or emulators. I would like to challenge developers to find more creative uses than these more obvious answers.
So I've been researching how I might go about creating the headset for Noddinggam. I've done the Google thing, the Pinterest thing, and the Instructables thing and have some inspirations...
First, the headphones...after scouring the places I know to go I found several plans for 3D printed headphones. Here are some ideas I found that I really liked...
I really liked the effort and reasoning that Shannon Ley put into the design from Instructables.com. So my thought at the moment is that I will be editing that design to hold the sensors, microcontroller and Bluetooth device. But first I have to get sensors working how I want them, and prototype the functionality...
I started thinking about this project months ago and bought an arduino primo core to play with, which is cool and all, but is now retired so I can't replicate the headset, and neither could anyone else. I also ran into a problem with getting it to register a twisting motion. And it required soldering to attach wireing, which I find annoying when I'm prototyping. At least it got me that much info, but I'm going to have to use it for some one-of-a-kind thing down the road because it wasn't cheap. I'm using LilyPad and Flora sensors for sewable projects, and find them easy to prototype with. So I decided to give those a try. I'm trying out Flora and other wearables from Adafruit.com...
And one or two of these...
I already have a few other items to fill the gaps like conductive thread and battery packs. So as soon as I get the new stuff from Adafruit, I'm looking forward to seeing if they do what I hope or if I'll need to revise.
As a work in progress, here's what I've done so far...
Misc. Stuff I already have:
Getting the hardware ready for experimentation...
The above image shows how to hook up the accelerometer to the Flora.
You'll also need Micro-B USB cable cable to give it power and load the programming, a good phone charger cable will probably work, my thick data capable / fast charging cable worked, not the thin slow power-only cable. If you want to be sure, buy one of theirs. This is where you plug it in...
Programming the Flora
I downloaded the library for the LSM9DS0 here...
and installed it using this tutorial...
In the Arduino App, I needed to make sure I had the Adafruit Flora board installed using this tutorial...
I used the "sensorapi" example mentioned in this tutorial, and voila I have accelerometer data.
My experiments so far have shown that I need two accelerometers to sense all six directions of head movement. I'm going to mess around with this a bit, maybe read up and see if it's better to handle the data on the Arduino side or in the phone.
Baby steps...I'm just learning. If you have suggestions or comments feel free to leave them below.
Still a work in progress.