Robot learning by mimicking humans with KINECT sensors

Wednesday, December 31, 1969 - 6:00pm
Faculty Advisor: Thomas Hoft, Department of Mathematics, St Thomas University

Problem Poser: Vassilios Morellas, Computer Science & Engineering, University of Minnesota

In this project, we will use numerical optimization and linear algebra techniques to devise representations of human poses via processing 3D information from multiple KINECT sensors. We will then map them to the body of a humanoid robot and program the robotic hardware to mimic the respective human poses.

Robots and robotic devices have become integral parts of our lives. They sweep floors, clean windows, perform surveillance, and assist in operating rooms. Future successful interactions will likely depend on enriching our communications repertoire to include important non-verbal modalities. Just as human interactions typically are not restricted only to simple commands or to simple statement/response dialogues, advances in sensor technologies and algorithmic procedures provide new ways to interact with robots. A natural human-robot coexistence must include the capacities to instruct robot companions and assistants by demonstration, to have them respond correctly to our gestures and expressions, and to have them recognize our intentions and our immediate situational needs.

Depth sensors such as the KINECT provide an excellent medium to realize robot learning by mapping human poses and gestures to equivalent robotic poses and gestures. Interpreting the KINECT sensor data is a nonlinear constrained optimization problem, known as inverse kinematics or motion retargeting. Current implementations use heuristic methods to interpret the data. We will apply a first-principles approach using established optimization and numerical linear algebra techniques to identify and map human poses.

Required Background: linear algebra, strong programming skills, scientific computing or numerical analysis

Useful background: MATLAB experience, numerical linear algebra, one of computer graphics, system integration or hardware programming