The document discusses reactive reaching and grasping capabilities on a humanoid robot. It describes using visual perception to detect objects and hands, and learning approaches like neural networks and genetic programming to develop filters for object detection. It also discusses frameworks for motion generation and manipulation, hand-eye coordination models, and how manipulation actions can extract information to improve perception through online continuous learning. The overall goal is to develop capabilities for object manipulation and improved eye-hand coordination.

1. Reactive Reaching and Grasping
on a Humanoid Robot
Jürgen ‘Juxi’ Leitner
Dalle Molle Institute for AI (IDSIA)
#ICINCO 2014

2. our iCub
humanoid

3. towards learning
object manipulation
http://robotics.idsia.ch/

4. IM-CLeVeR
project
http://robotics.idsia.ch/im-clever/

5. towards learning
object manipulation
http://robotics.idsia.ch/

6. visual
!
perception
thanks to G. Metta and IIT for this picture

7. detecting
!
objects
Harding et al., 2013
Leitner et al., ICDL 2012, ARS 2012, BICA 2012, CEC 2013

8. + min dilate avg INP INP INP
cartesian genetic
programming
[Leitner et al, 2012a/b, Harding et al., 2013]

9. learning
approach

10. !
icImage GreenTeaBoxDetector::runFilter() {
icImage node0 = InputImages[6];
icImage node1 = InputImages[1];
icImage node2 = node0.absdiff(node1);
icImage node5 = node2.SmoothBilateral(11);
icImage node12 = InputImages[0];
icImage node16 = node12.Sqrt();
icImage node33 = node16.erode(6);
icImage node34 = node33.log();
icImage node36 = node34.min(node5);
icImage node49 = node36.Normalize();
!
//cleanup
...
icImage out = node49.threshold(230.7218f);
return out;
}
detection

11. detect

12. detect

13. detection

14. cgp
approach

15. [Leitner et al, 2013]
visual
hand detection

16. detecting
hands

17. supervised learning
approach
BUT

18. FIAScollaboration
feature
segmentation
saliency
map

19. pre
segmentation

20. combined
approach

21. !
transferring
spatial perception

22. learning
setup

23. 9DOF
iCub
trainingset
bounding box
6 per eye
Cartesian
Coordinates
.
.
.

24. spatial perception
neural network
! fully connected
..
.
9DOF
iCub
bounding box
6 per eye
! fully connected
Cartesian
Coordinates
..
.

25. towards learning
object manipulation
http://robotics.idsia.ch/

26. MoBeE
framework Frank et al., ICINCO, 2012.

27. MoBeE
Frank et al., ICINCO, 2012.

28. Frank et al., ICINCO, 2012.

30. robot
motion
[Stollenga et al, IROS 2013]

31. gmeonteiorantion
Shakey 2013 Winner
Stollenga et al, 2013

32. MoBeEv2
[Frank et al., 2011,2012, 2013]

33. op-space forcing
hand/arm
CSWorld
CSHand
CSR/CSL
[Leitner et al, in prep]
-

34. towards learning
object manipulation
http://robotics.idsia.ch/

35. hand-eye
coordination

36. model

37. http://robotics.idsia.ch/

38. manipulation
for improved perception

39. manipulation
actions

40. extracting
information

41. improved
detection

42. !
icImage* BlueCupFilter::runFilter() {
icImage* node43 = InputImages[4];
icImage* node49 = node43->LocalAvg(15);
!
icImage* out = node49->threshold(81.532f);
return out;
}
detection

43. !
icImage* BlueCupFilter::runFilter() {
icImage* node0 = InputImages[4].Exp();
icImage* node5 = InputImages[0];
icImage* node16 = node0->Gabor(-8,14,1,13);
icImage* node17 = InputImages[4]->LocalAvg(6);
icImage* node18 = node16->Laplace(5);
icImage* node19 = node5->Sobel(13,9);
icImage* node24 = node17->Erode(5);
icImage* node28 = node19->Min(node18);
icImage* node29 = node28->Min(node24);
icImage* node41 = node29->LocalAvg(7);
icImage* node49 = node41->LocalMax(7);
!
icImage* out = node49->threshold(68.032f);
return out;
}
resulting
detection

44. tele
operation

45. tele
operation

46. !
fuwtourrek
improved eye-hand coordination
different object representations
online, continuous learning
vSLAM aerial robotics & for world model

47. for listening thanks
juxi@idsia.ch
http://Juxi.net/projects
http://dilbert.com/strips/comic/2013-10-24/