GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
victores2014thesis-presentation
1. Robot Imagination System
Author: Juan Carlos González Víctores
Advisors: Carlos Balaguer and Alberto Jardón
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
2. Robot Imagination System
Introduction
State of the Art
General Architecture
Perception
Inference
Execution
Experiments
Conclusions
1 of 45Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
2 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
4. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
3 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
5. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
3 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
6. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
3 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
7. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
4 of 45
A. No perceptual links between physical
characteristics and words.
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
8. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
4 of 45
A. No perceptual links between physical
characteristics and words.
B. Lack of inference capabilities for
interpreting combinations of words.
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
9. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
4 of 45
A. No perceptual links between physical
characteristics and words.
B. Lack of inference capabilities for
interpreting combinations of words.
C. No bindings between actions and their
effects, reprogramming required.
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
10. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
5 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
A. Develop a framework for linking
physical characteristics and words.
B. Enable inference mechanisms for
interpreting combinations of words.
C. Allow a robot to act according to the
desired effects on an object.
11. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
6 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
12. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
6 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
13. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
7 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
1. Imagination
2. Grounded Language
3. Spatial Language
4. Execution
5. Integration
14. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
7 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
1. Imagination
We define Robot Imagination as the process of
creating previously unexisting Knowledge K’ from
existing Knowledge K and Words W.
Where new Knowledge K’ can be used for a real
world application.
- Robot Imagination goes beyond Prediction.
15. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
7 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
1. Imagination
2. Grounded Language
3. Spatial Language
4. Execution
5. Integration
16. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
8 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
17. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
9 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
A. No perceptual links between physical
characteristics and words.
18. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
9 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
A. No perceptual links between physical
characteristics and words.
19. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
10 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
A. No perceptual links between physical
characteristics and words.
20. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
10 of 45
``The horizontal
purple rectangle
below the horizontal
green rectangle'‘
(DESCRIBER, 2002)
A. No perceptual links between physical
characteristics and words.
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
21. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
11 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
B. Lack of inference capabilities for
interpreting combinations of words.
22. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
11 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
B. Lack of inference capabilities for
interpreting combinations of words.
(AARON, 1989)
23. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
11 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
B. Lack of inference capabilities for
interpreting combinations of words.
(AARON, 1989) (AHA! Experience, 2011)
!
24. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
12 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
C. No bindings between actions and their
effects, reprogramming required.
25. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
12 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
C. No bindings between actions and their
effects, reprogramming required.
26. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
12 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
C. No bindings between actions and their
effects, reprogramming required.
27. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
13 of 45
Robot Imagination System
General Overview
Component Breakdown
State Machine
28. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
14 of 45
General Overview
Component Breakdown
State Machine
A. Perception
Imagination Core
State Machine
Execution Core
Visual OutputComputer Vision
Grounding Core
Speech OutputSpeech Recognition
B. Inference C. Execution
29. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
General Overview
Component Breakdown
State Machine
A. Perception
30. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Computer Vision
General Overview
Component Breakdown
State Machine
A. Perception
Computer Vision
31. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Computer Vision
General Overview
Component Breakdown
State Machine
• Speech Recognition
A. Perception
Computer Vision
Speech Recognition
32. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Speech Recognition
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Computer Vision
General Overview
Component Breakdown
State Machine
• Semantic Memory
A. Perception
Computer Vision
Speech Recognition
33. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Speech Recognition
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Computer Vision
General Overview
Component Breakdown
State Machine
A. Perception
Computer Vision
Speech Recognition
Grounding Core
• Semantic Memory
34. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
A. Perception
Computer Vision
Speech Recognition
State Machine
Grounding Core
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
General Overview
Component Breakdown
State Machine
• Speech Recognition
• Computer Vision
• Grounded Language
• Semantic Memory
35. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
A. Perception
Computer Vision
Speech Recognition
State Machine
Grounding Core
• Speech Recognition
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Semantic Memory
• Embodied Cognition
• Computer Vision
• Grounded Language
General Overview
Component Breakdown
State Machine
36. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Philosophy
• Statistical Learning
• Psychology
• Neuro-science (medical)
• Neuro-science (computer)
General Overview
Component Breakdown
State Machine
A. Perception
Computer Vision
Speech Recognition
Grounding Core
State Machine
Imagination Core
B. Inference
37. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
A. Perception
Computer Vision
Speech Recognition
Grounding Core
State Machine
Imagination Core
B. Inference C. Execution
Execution Core
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Correspondence Problem
• Robot Imitation
• Goal-Directed Actions
• Continuous Goal-Directed
Actions (CGDA)
General Overview
Component Breakdown
State Machine
38. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
General Overview
Component Breakdown
State Machine
A. Perception
Imagination Core
State Machine
Execution Core
Visual OutputComputer Vision
Grounding Core
Speech OutputSpeech Recognition
B. Inference C. Execution
39. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
16 of 45
General Overview
Component Breakdown
State Machine
40. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
16 of 45
General Overview
Component Breakdown
State Machine
41. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
16 of 45
General Overview
Component Breakdown
State Machine
42. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
43. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
44. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
45. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
46. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
47. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
48. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
49. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
17 of 45
Perception Block
Inference Block
Execution Block
A. Perception
Computer Vision
50. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
18 of 45
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
• 2D
• RGB/HSV or Canny Edge Detection
• Morphological Closing
• Find contours (blobs) to segment objects
• 3D
• Remove NaN and Voxelize
• RANSAC using plane model
• Remove planes (tables and walls)
• Euclidian Clustering to segment objects
• Feature extraction
• Centroid, area, rotation, min/max axis,
aspect ratio, rectangularity, solidity, arc,
radius, RGB, HSV.
A. Perception
Computer Vision
51. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
52. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
• Generated semantic subspaces:
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
53. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
• Generated semantic subspaces:
• Blue
0
Solidity
Hue
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
54. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
• Generated semantic subspaces:
• Blue
• Scissors
0
Solidity
Hue
0
Solidity
Hue
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
55. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
0
Solidity
Hue
0
Solidity
Hue
• Generated semantic subspaces:
• Blue
• Scissors
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
56. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
0
Solidity
Hue
0
Solidity
Hue
• Generated semantic subspaces:
• Blue
• Scissors
1
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
57. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
1
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
0
Solidity
Hue
0
Solidity
Hue
Hue
• Generated semantic subspaces:
• Blue
• Scissors
• Red
Solidity
0
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
58. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
1
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
0
Solidity
Hue
0
Solidity
Hue
Hue
• Generated semantic subspaces:
• Blue
• Scissors
• Red
Solidity
0
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
59. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
20 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Execution Block
60. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
• Enhanced Prediction Algorithm (RIS)
• Object Recognition (Euclidean Distance)
• Reconstruction (Evolutionary Computation)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
21 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Imagination Core
Execution Block
61. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Imagination Core
Execution Block
62. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Imagination Core
Execution Block
63. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Imagination Core
Execution Block
64. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
• Basic Prediction:
Perception Block
Inference Block
Imagination Core
Execution Block
65. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
Perception Block
Inference Block
Imagination Core
Execution Block
66. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
Perception Block
Inference Block
Imagination Core
Execution Block
67. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
Perception Block
Inference Block
Imagination Core
Execution Block
68. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
Perception Block
Inference Block
Imagination Core
Execution Block
69. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• Basic Prediction:
Perception Block
Inference Block
Imagination Core
Execution Block
70. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• Basic Prediction:
• COMPLETE
Perception Block
Inference Block
Imagination Core
Execution Block
71. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• Basic Prediction:
• COMPLETE
• RELEVANT
Perception Block
Inference Block
Imagination Core
Execution Block
72. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
1. Generalization
2. Intersection
3. Projection
4. Averaging
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
• COMPLETE
• RELEVANT
• BALANCED
Perception Block
Inference Block
Imagination Core
Execution Block
73. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
23 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why a hyperplane of order n-1 ?
Perception Block
Inference Block
Imagination Core
Execution Block
74. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
23 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why a hyperplane of order n-1 ?
• From the algebraic definition, if the points fit
to the model, it simultaneously captures all of
the linear dependencies and couplings in Rn.
Perception Block
Inference Block
Imagination Core
Execution Block
75. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
23 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why a hyperplane of order n-1 ?
• From the algebraic definition, if the points fit
to the model, it simultaneously captures all of
the linear dependencies and couplings in Rn.
• Hyperplanes are defined in all space, so the
“meanings” of words are also extended across,
and along, all the Feature Space.
Perception Block
Inference Block
Imagination Core
Execution Block
76. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
23 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why a hyperplane of order n-1 ?
• From the algebraic definition, if the points fit
to the model, it simultaneously captures all of
the linear dependencies and couplings in Rn.
• Hyperplanes are defined in all space, so the
“meanings” of words are also extended across,
and along, all the Feature Space.
• Practical benefit of always finding
intersections between these representations.
Perception Block
Inference Block
Imagination Core
Execution Block
77. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
24 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why are orthogonal projections used ?
Perception Block
Inference Block
Imagination Core
Execution Block
78. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
24 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why are orthogonal projections used ?
• Projections result on the valid solution
geometrical construct, and are closest to the
original data of a given point cloud.
Perception Block
Inference Block
Imagination Core
Execution Block
79. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
24 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why are orthogonal projections used ?
• Projections result on the valid solution
geometrical construct, and are closest to the
original data of a given point cloud.
• Promotes ranges of values that are common in
objects, which is useful when not all features are
specified by words.
Perception Block
Inference Block
Imagination Core
Execution Block
80. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Perception Block
Inference Block
Imagination Core
Execution Block
81. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Yes, the basic prediction algorithm may fail under
the following circumstances:
Perception Block
Inference Block
Imagination Core
Execution Block
82. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Yes, the basic prediction algorithm may fail under
the following circumstances:
• Context dependency (e.g. “green” color or state
of ripeness).
Perception Block
Inference Block
Imagination Core
Execution Block
83. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Yes, the basic prediction algorithm may fail under
the following circumstances:
• Context dependency (e.g. “green” color or state
of ripeness).
• Periodicity (e.g. “red” hue is 0º and 360º, a
hyperplane could be fit at 180º).
Perception Block
Inference Block
Imagination Core
Execution Block
84. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Yes, the basic prediction algorithm may fail under
the following circumstances:
• Context dependency (e.g. “green” color or state
of ripeness).
• Periodicity (e.g. “red” hue is 0º and 360º, a
hyperplane could be fit at 180º).
• Specificity (non-extendable meanings).
Perception Block
Inference Block
Imagination Core
Execution Block
85. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
26 of 45
Imagination Core
B. Inference
• Object Feature Prediction:
• Enhanced Prediction Algorithm (RIS)
• Enhanced Prediction:
1. Context Detector
2. Hyperspherical
Shape Detector
Perception Block
Inference Block
Imagination Core
Execution Block
86. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
27 of 45
Imagination Core
B. Inference
• Object Feature Prediction:
• Enhanced Prediction Algorithm (RIS)
• Enhanced Prediction:
1. Context Detector
2. Hyperspherical
Shape Detector
The Context Detector counts the accompanying
words of a query word within each cluster
Perception Block
Inference Block
Imagination Core
Execution Block
87. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
28 of 45
Imagination Core
B. Inference
• Object Feature Prediction:
• Enhanced Prediction Algorithm (RIS)
• Enhanced Prediction:
1. Context Detection
2. Hyperspherical
Shape Detector The Hyperspherical Shape Detector compares all
the sorted eigenvalues of a given point cloud
Perception Block
Inference Block
Imagination Core
Execution Block
88. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
29 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
89. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
30 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization (CGDA)
• Action Recognition (CGDA)
• Execution (Evolutionary Computation)
90. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
31 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization (CGDA)
91. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
31 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization (CGDA)
• Action Generalization
1. Split in Intervals
92. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
31 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization (CGDA)
• Action Generalization
1. Split in Intervals
2. Average in Intervals
93. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
31 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization
1. Split in Intervals
2. Average in Intervals
3. Linear Interpolation
• Action Generalization (CGDA)
94. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
GeneralizedTraj.e.g.Xpos
Query Traj. e.g. X pos
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
32 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Recognition (CGDA)
95. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
GeneralizedTraj.e.g.Xpos
Query Traj. e.g. X pos
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
32 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Recognition (CGDA)
• Action Recognition
1. Axis-by-axis comparison
96. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
GeneralizedTraj.e.g.Xpos
Query Traj. e.g. X pos
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
32 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Recognition (CGDA)
• Action Recognition
1. Axis-by-axis comparison
2. Dynamic Time Warping
97. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
GeneralizedTraj.e.g.Xpos
Query Traj. e.g. X pos
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
32 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Recognition (CGDA)
• Action Recognition
1. Axis-by-axis comparison
2. Dynamic Time Warping
3. Total cost = sum(DTW)
98. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
A. Perception
Imagination Core
State Machine
Execution Core
Visual OutputComputer Vision
Grounding Core
Speech OutputSpeech Recognition
B. Inference C. Execution
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
33 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
Each RIS component has been
implemented as a YARP module.
They are direct implementations of
the explained algorithms.
99. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
34 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
1. Populating the database: Every training sample has a description
100. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
34 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
1. Populating the database: Every training sample has a description,
e.g. the first image is labeled “top-left-dark-blue-fat-straight-box”
101. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”.
102. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”
103. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”
104. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”
105. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”
106. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
36 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
4.The drawing demonstration, where the robot exposes its mental model.
[J.G.Victores, S.Morante et Al., IROS 2013,Tokyo] [FASTCompany, 2013]
107. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
37 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
-The system is trained through speech with a blue visual marker.
-The component used for Speech Recognition requires a pre-defined
corpus of words to recognize.
108. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
38 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
The system can generalize a word with as little as 2 training
samples in the 2D case, and 3 training samples in the 3D case.
109. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
39 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
Testing the execution of the “waxing” action encoded as a
Continuous Goal-Oriented Action (CGDA).
110. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
40 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
[S.Morante, J.G.Victores et Al., ICRA 2013, Hong Kong]
111. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
41 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
112. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
42 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
113. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
43 of 45
Perception
- Framework for creating links between words and
physical characteristics
Inference
- Robot Imagination System for providing robots with
imagination skills through object features and semantics
Execution
- Presented a new way of encoding actions, which can
complement Programming by Demonstration
- Explained proposed algorithms for CGDA
generalization, recognition and execution
Progress beyond the State of Art
Future Lines of Research
Scientific Contributions
114. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
44 of 45
Progress beyond the State of Art
Future Lines of Research
Scientific Contributions
Origin of Data
- Sensory information of various nature (e.g. haptic,
rugosity) and from the Internet-of-Things.
Data Representation
- Human-inspired color space such as YUV. Use of SIFT and
SURF visual descriptors.
Techniques for Inference
- Feature selection (e.g. LASSO). Gaussian Mixture Models,
Non-linear PCA, Manifolds, Neural Networks.
Additional Functionalities
- Vector and dynamic information, auto-tuned parameters.
115. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
45 of 45
Progress beyond the State of Art
Future Lines of Research
Scientific Contributions
“Action Effect Generalization, Recognition and Execution
through Continuous Goal-Directed Actions”, IEEE ICRA,
Hong Kong, China, 2014.
“Towards Robot Imagination Through Object Feature
Inference”, IEEE/RSJ IROS,Tokyo, Japan, 2013.
“Personal Autonomy Rehabilitation in Home
Environments by a Portable Assistive Robot”, IEEE-
Systems, Man and Cybernetics, Part C, 2011.
Over 27 conference articles, 4 top-quartile journal papers,
3 book chapters, a patent, and artistic works have been
published by the author within the duration of this Thesis.
116. Robot Imagination System
Author: Juan Carlos González Víctores
Advisors: Carlos Balaguer and Alberto Jardón
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014