ISAS is an eyes-free mobile system designed to give blind users a better sense of their surroundings. Although other systems (e.g., Humanware's Trekker and standard GPS tools) emphasize navigation from one specific location to another, typically accomplished by explicit turn-by-turn instructions, our goal is to use ambient audio to reveal the kind of information that visual cues such as neon signs provide to sighted users. Once users notice a point of interest, additional details are available on demand.
Our current design consists of the following modes of operation:
Walking mode: Users are not focused on the system, but as they walk, they receive cues as to what is around them, including points of interest and orientation aids. POIs can be rendered either in a circular sweep clockwise around the user (radar sweep) or else from near to far (shockwave sweep). The user can toggle between an ongoing "continuous sweep" that repeats in a loop as long as ISAS remains in Walking mode, and "single sweep" which plays one sweep (e.g., one 360 degree revolution for Radar) each time the screen is tapped. Holding a finger down on the screen pauses the sweep and provides more details on recently heard POIs.
Stop & Listen mode: After hearing something that piques their interest, or from a desire to find something specific, users can interact with the device to learn more about their surroundings. ISAS supports one-handed operation, so that the other hand remains free to hold a cane or guide dog. The user moves their finger on the screen to explore points of interest in front of them, as well as obtain the current street address, current direction, sensor error information and a summary of nearby POIs
The ISAS user experience is tightly tied to spatialized audio, preferably using only bone-conducting or open air headphones so as not to interfere with the natural sounds of the environment. Sounds appear to come from locations surrounding the user, thereby giving a sense of directionality and distance. This allows for parsimony of representation and less intrusive sound cues. Imagine the difference between a mechanical voice stating, "Restaurant, 50 meters, 60 degrees to your left" vs. a very short "Restaurant" spatialized in the correct direction.
ISAS currently runs on either an iPhone 4 (or later) or an Android Galaxy Nexus, and uses the smartphone's built-in compass, accelerometer, gyroscope and GPS hardware to determine the user's location and orientation. Nearby points of interest are retrieved automatically via the ISAS server. The appropriate sound scene is then rendered using the libpd library to run PureData (PD) patches.
This server has cached street and intersection information from Open Street Map (currently only for Québec) and La Société de transport de Montréal (STM), and does real-time queries via the Google Places API to find nearby POIs such as businesses, public buildings, parks and landmarks. These locations can be rendered in different ways, e.g., by relative direction ("front right") or by cardinal direction ("northeast"). Category names (e.g., "bar," "fast food", etc.), are spatialized to sound like they are coming from the actual location of the POI, giving a direct cue as to its direction and distance.
We have worked with both French and English organizations for the blind in Montreal, including the Institut Nazareth & Louis Braille (INLB) and the Montreal Association for the Blind (MAB) to test ISAS with a number of blind participants. These user tests have taken the form of informal walkabouts while soliciting feedback, more formal tests with specific tasks to complete using ISAS while on the streets of Montreal, and also longer-term deployments where blind individuals were loaned iPhone devices to use in their daily routines. Feedback has been generally positive for the system as a whole, but has also pointed out numerous usability and other issues that have been factored into the design. A paper summarizing the results across several of these tests is in preparation.
Walking straight
In addition to core environmental awareness functionality, we have also experimented with an additional mode designed to assist blind users walk in a straight line. By using the iPhone's gyroscope, we detect deviation from a straight path, and play sounds through headphones to guide the user back on course. Initial results (paper in preparation), indicate that performance with this mode is better than walking without the system, and is generally adequate to keep the user within a pedestrian walkway over a distance of 15m.
Smartphone sensor evaluation
Sensor reliability on smartphone devices continues to be an ongoing concern for practical deployment, especially in downtown areas with large buildings, where the sensors become erratic. Thus, we have focused considerable attention on evaluating the quality of the location and orientation sensors in current smartphones. Since ISAS depends heavily on these sensors, knowing when and how they fail is crucial to handling these issues in the application. We undertook an experiment with three smartphones: iPhone 4, iPhone 4s, and Android Samsung Galaxy Nexus. By repeatedly walking a prescribed path in two areas of Montreal with all three phone in various orientations and positions on the body, while continually logging location and compass/device orientation data, we can see how well the sensors perform vs. reality. In addition, we can see whether the devices' estimates of their own error are accurate. The results have been written up in a paper, Smartphone sensor reliability for augmented reality applications, presented at the Mobiquitous 2012 conference in Beijing, China.
These results have caused us to remove our original sensor fusion algorithm and to implement solutions such as "snapping" the user to the nearest street and to dynamically change how POIs are rendered depending on sensor accuracy. We expect the results of this study to be useful to others who are encountering the same issues while implementing augmented reality solutions on smartphones.
As part of the Impact Award program, CIRA also ran a "People's Choice Award" competition, which ISAS won by receiving the most votes from mesh12 attendees, placing first out of the four Impact Award winners.
Smartphone sensor reliability for augmented reality applications.
Blum, J.R., Greencorn, D., and Cooperstock, J.R. 2012,
9th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (Mobiquitous 2012), Beijing, China, December. Accept rate 31%.
Eyes-Free Environmental Awareness for Navigation.
El-Shimy, D.; Grond, F.; Olmos, A.; and Cooperstock, J. R. 2012. Springer Journal on Multimodal User Interfaces, Special Issue on Interactive Sonification.
Tagging mode: Users may record new content of their own, based on objects in their immediate vicinity, for example, "this is a great restaurant" or "there's a bad pothole in the sidewalk here". STATUS: 90% implemented; untested with users.
Exploring mode: An extension to Stop & Listen mode, users may query the system to find information about areas further away from their current position. STATUS: unimplemented.
User settings: Allows the end-user to easily change options such as filtering by category and shockwave vs. radar Walking modes. STATUS: implemented internally; not exposed to or tested with end-users.
Voice Recognition: An easier interface for additional category filtering options and user tagging, being driven through voice recognition rather than touchscreen and gesture interaction.
ISAS received the Best Paper Award at Mobiquitous 2011 in Copenhagen, Denmark for the paper,
What's around me? Spatialized audio augmented reality for blind users with a smartphone.
ISAS won the 2012 CIRA Impact Award in the Application Category.
We were interviewed after the awards ceremony at the Mesh 2012 conference in Toronto:
As part of the Impact Award program, CIRA also ran a "People's Choice Award" competition, which ISAS won by receiving the most votes from mesh12 attendees, placing first out of the four Impact Award winners.