Abstract
According to a released report of WHO(World Health Organization) in August 2014, 285 million people are estimated to be visually impaired worldwide, 39 million of them are blind and 246 million have low vision. [1] They have encountered a lot of problems and among those problems, the most urgent one is mobility.
The Smart Environment Explorer Stick (SEES) [2] project has been implemented since 2013 by SMIR group of the laboratory LIMOS. It has been focusing on implementing a real-time, low-cost and energy efficient smart stick, providing VIP (Visually Impaired People) with both navigation and environmental detection function.
This project has presented a solution of adaptive context-aware knowledge based object recognition in SEES project. With a single camera, several fast and reliable object detecting methods have been developed, combining color based method, shape based method and feature based method together, to give a fast and reliable solution for SEES project.
A combined detecting method has been developed in this project to detect the door. Filtered HSV (Hue-Saturation-Value) color area and edge line of the potential objects were both considered in this combined detecting method. For other objects such as doorplate, ORB (Oriented FAST and Rotated BRIEF, FAST, Features from accelerated segment test, BRIEF, Binary Robust Independent Elementary Features) method was used for its fast and accurate performance to detect the features, which were stored in a database. A confidence value was attached to each object detected. The confidence model was also used in data fusion with other indoor navigation results that came from other sensors, such as accelerometer and wheel encoder.
A Multi-resolution pyramid computing algorithm was implemented in the project. According to different situation, the system would switch between different detecting methods.
This project has provided fast and reliable object information to SEES project and helps improve the accuracy of VIP’s indoor position.
Keywords: Visually impaired people, Indoor navigation; Object recognition; Embedded system
Visually impaired people with his guide dog.
System design
Architecture
A set of thresholds and images database were generated by training model. The camera attached at the stick produced real-time images and checked if there were enough computing resources in Raspberry Pi board. If the answer was yes, the image processing of object detection would be implemented in local board. But if there was not enough computing resources, the images would be sent to the server, which was more powerful than
Raspberry Pi board. After detecting, the results of objects would be returned back to Raspberry Pi board through Wi-Fi.
Multi-solution pyramid algorithm
Different type of object detection called for different level accuracy. For example, the door detection method runs in a high frequency in the whole period of the software. If it took too much computing time, the detection would be slow and it could not be real-time, just like the previous method of SEES project. In this case, the door detection called for a fast and reliable computing method and decreased the size of original image was the easiest way.
For another example, door handle detection was implemented only when VIP reached his destination. The computing time was not the main problem in this detection. It demanded a higher accuracy method. In this case, after a fast detection of the door under lower-level accuracy (at a size of 4030 pixels), a rough door rectangle was detected. Normally, the door rectangle contained the door handle. With the given door rectangle under low level accuracy, a new rectangle was recomputed in the original image, which was at a size of 640480 pixels. After the switch of the rectangle, the matched part of the image was used as the input of the door handle detection.
Combined algorithm
Both color based method and shape based methods were combined to detect the object. We set door detection as an example. For a real door, its HSV color area should be larger than the threshold, and in addition, there should be a line in each edge of the area detected. The algorithm was shown in the figure.
Training application
A graphical interface was implemented in this project. The thresholds of the object detection were attached with the track bars. For door detection, there were 15 thresholds, including accuracy level, lower and upper values of HSV color space, lower and upper values of objects’ area. For door handle detection, there were 10 thresholds, including lower and upper value of HSV color space, just like door detection while the values were different. For direction detection, there were five thresholds, including lower and upper angles of the lines detected the max gap between two lines and so on.
Project results
Indoor position
The position of visually impaired people was computed by my partner Jean CONNIER
Door detection
the computing time of door detection under different resolutions.
Door handle detection
Door plate detection
I implemented three feature based method to detect the doorplate, SIFT, SURF, ORB, consider the computing time and the number of matched features, i finally chose ORB.
The computing time of SIFT, SURF and ORB were given in the figure below.