engleski

Autonomous agent localization in dynamic scenarios based on visual sensor data fusion

This thesis addresses the problem of the visual localization of mobile agents in challenging scenarios. As challenging scenarios, we consider the environments populated by humans, i.e., the environments in which humans and robots coexist and cooperate. A good example of such an environment is an automated warehouse where humans and robots cooperate to increase the efficiency of the warehouse operation. The safety requirements for such coexistence are divided into three levels: (i) redirection of robots to avoid human- robot encounters, (ii) warning of humans about the environment and possible encounters with robots, and (iii) immediate shutdown of robots in close proximity to humans. The most important safety level is solved by setting up ranging sensors on all agents (humans and robots) in the warehouse and shutting down all robots whose distance to a human becomes too small. Less stringent safety requirements apply to the other two levels, and their implementation is allowed to be more complex. Since the cost of the solution is one of the criteria for choosing one solution over another, the focus is put on approaches that maximize the use of existing infrastructure in warehouses and require minimal installation time. The solution converged to a set of wearable sensors worn on the operator’s Safety Vest which limits the processing power and power supply. The implementation of the modified Semi-Global Matching (SGM) method for disparity computation was developed as part of the safety level that informs the operator of his environment and warns him of potential robot encounters. The computationally intensive steps of the original SGM are improved for an image sequence by reusing the existing disparity values from the previous steps. Assuming that the scene is constant in time, the disparity information between successive steps is transformed with visual odometry and fused with the new disparity measurements within the Kalman filter framework. The improvement in the complex steps of the method and the efficient implementation with the SIMD instruction set and multithreading showed the overall improvement of the proposed solution over the original SGM method. In addition, the Kalman filter framework enabled the detection of moving objects between two consecutive steps in the disparity images because moving objects do not follow the method’s assumption of a static scene. Locating all agents in the warehouse is a requirement for a safety level responsible for avoiding encounters based on the redirection of robots. The locations of the robots are already known to the warehouse management system, since tasks to carry racks are assigned based on their current location. The remaining element, localization of humans, requires a special approach since the conditions in the warehouse do not satisfy the static environment assumption of all common localization approaches. Therefore, the implemented localization algorithm is based on two location cues: (i) visual odometry and (ii) detection of the existing warehouse’s fiducial ground markers. The location cues are fused within the graph optimization process which provides a globally correct location estimate at a constant frequency. The proposed localization method is robust to visual aliasing and changing environment conditions in the warehouse, and requires only a lightweight map of ground marker placement. Furthermore, localization in visually challenging scenarios is further improved by augmenting the camera sensors with non-visual sensors and providing a simple odometry uncertainty model to provide information of visual odometry estimate quality.

semi-global matching, Kalman filter, moving objects detection, stereo camera, UWB sensors, wearable sensors fusion, warehouse localization, visual odometry, graph optimization-based localization

nije evidentirano