Today at our monthly meetup, I talk how the technologies: Artificial intelligence, Augment reality, Machine Learning can help people with disability. I start to introduce the Smart City concept and why it is important this context.
Before to start explanation, here you can find my slides: https://www.slideshare.net/algraps/smart-cities-digital-transformation-helps-people
and my video: https://www.youtube.com/watch?v=_hxiBzFNdxM
About 36 million people worldwide are blind. For many of these people, navigating new spaces can be a cumbersome or frustrating experience as they listen and feel their way around their environment.
Our project seeks to utilize the spatial mapping power of the Microsoft HoloLens augmented reality device in order to help users navigate in unfamiliar environments. This could improve independence for blind people by allowing them to find and read objects such as bus stops without sighted assistance.
By making audible the 3D mesh of the user’s surroundings that these devices generate, we will create a soundscape that will let users hear the objects around them. Thus, visually impaired users will be able to hear the presence of obstacles like tables and chairs, or the lack of obstacles that represents a doorway, from up to five meters away.
In combining this capability with machine vision-powered text recognition in one hands-free headset, we hope to give blind users a new level of independence in unfamiliar spaces.
The proposed system includes a depth camera for acquiring the depth information of the surroundings, an ultrasonic rangefinder consisting of an ultrasonic sensor and a MCU (Microprogrammed Control Unit) for measuring the obstacle distance, an embedded CPU (Central Processing Unit) board acting as main processing module, which does such operations as depth image processing, data fusion, AR rendering, guiding sound synthesis, etc., a pair of AR glasses to display the visual enhancement information and an earphone to play the guiding sound.
The depth image acquired from the depth camera is processed by the depth-based way-finding algorithm which outputs several candidate moving directions. The multi-sensor fusion based obstacle avoiding algorithm then uses the ultrasonic measurement data to select an optimal moving direction from the candidates. The AR rendering utilizes one depth image to generate and render the binocular images as well as the moving direction to guide the user efficiently. The guiding sound synthesis takes the moving direction as the input to produce the auditory cue for guiding the totally blind people. Three kinds of auditory cues are developed and tested to allow the selection of the most suitable one under different scenarios.
The visual enhancement, which adopts the AR technique, is used for weak sighted people. In order to showing the guiding cue to the user based on the one depth image, the binocular parallax images are needed to generate. This was realized in Unity3D by adjusting the texture coordinates of the depth image. The rendering stereo images integrate the feasible direction for guiding the user.
Alessandro Graps' blog: 