SlideShare une entreprise Scribd logo
1  sur  116
Télécharger pour lire hors ligne
Robot Imagination System
Author: Juan Carlos González Víctores
Advisors: Carlos Balaguer and Alberto Jardón
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Robot Imagination System
 Introduction
 State of the Art
 General Architecture
 Perception
 Inference
 Execution
 Experiments
 Conclusions
1 of 45Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
2 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
3 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
3 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
3 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
4 of 45
A. No perceptual links between physical
characteristics and words.
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
4 of 45
A. No perceptual links between physical
characteristics and words.
B. Lack of inference capabilities for
interpreting combinations of words.
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
4 of 45
A. No perceptual links between physical
characteristics and words.
B. Lack of inference capabilities for
interpreting combinations of words.
C. No bindings between actions and their
effects, reprogramming required.
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
5 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
A. Develop a framework for linking
physical characteristics and words.
B. Enable inference mechanisms for
interpreting combinations of words.
C. Allow a robot to act according to the
desired effects on an object.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
6 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
6 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
7 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
1. Imagination
2. Grounded Language
3. Spatial Language
4. Execution
5. Integration
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
7 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
1. Imagination
We define Robot Imagination as the process of
creating previously unexisting Knowledge K’ from
existing Knowledge K and Words W.
Where new Knowledge K’ can be used for a real
world application.
- Robot Imagination goes beyond Prediction.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
7 of 45
Origin of this Thesis
Main Objectives
Target Scenario
Expected Novelties
1. Imagination
2. Grounded Language
3. Spatial Language
4. Execution
5. Integration
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
8 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
9 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
A. No perceptual links between physical
characteristics and words.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
9 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
A. No perceptual links between physical
characteristics and words.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
10 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
A. No perceptual links between physical
characteristics and words.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
10 of 45
``The horizontal
purple rectangle
below the horizontal
green rectangle'‘
(DESCRIBER, 2002)
A. No perceptual links between physical
characteristics and words.
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
11 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
B. Lack of inference capabilities for
interpreting combinations of words.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
11 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
B. Lack of inference capabilities for
interpreting combinations of words.
(AARON, 1989)
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
11 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
B. Lack of inference capabilities for
interpreting combinations of words.
(AARON, 1989) (AHA! Experience, 2011)
!
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
12 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
C. No bindings between actions and their
effects, reprogramming required.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
12 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
C. No bindings between actions and their
effects, reprogramming required.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
12 of 45
Artificial Cognitive Systems
The Symbol Grounding Problem
Mental Models and Mental Imagery
Robot Task Execution
C. No bindings between actions and their
effects, reprogramming required.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
13 of 45
Robot Imagination System
General Overview
Component Breakdown
State Machine
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
14 of 45
General Overview
Component Breakdown
State Machine
A. Perception
Imagination Core
State Machine
Execution Core
Visual OutputComputer Vision
Grounding Core
Speech OutputSpeech Recognition
B. Inference C. Execution
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
General Overview
Component Breakdown
State Machine
A. Perception
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Computer Vision
General Overview
Component Breakdown
State Machine
A. Perception
Computer Vision
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Computer Vision
General Overview
Component Breakdown
State Machine
• Speech Recognition
A. Perception
Computer Vision
Speech Recognition
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Speech Recognition
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Computer Vision
General Overview
Component Breakdown
State Machine
• Semantic Memory
A. Perception
Computer Vision
Speech Recognition
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Speech Recognition
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Computer Vision
General Overview
Component Breakdown
State Machine
A. Perception
Computer Vision
Speech Recognition
Grounding Core
• Semantic Memory
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
A. Perception
Computer Vision
Speech Recognition
State Machine
Grounding Core
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
General Overview
Component Breakdown
State Machine
• Speech Recognition
• Computer Vision
• Grounded Language
• Semantic Memory
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
A. Perception
Computer Vision
Speech Recognition
State Machine
Grounding Core
• Speech Recognition
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Semantic Memory
• Embodied Cognition
• Computer Vision
• Grounded Language
General Overview
Component Breakdown
State Machine
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Philosophy
• Statistical Learning
• Psychology
• Neuro-science (medical)
• Neuro-science (computer)
General Overview
Component Breakdown
State Machine
A. Perception
Computer Vision
Speech Recognition
Grounding Core
State Machine
Imagination Core
B. Inference
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
A. Perception
Computer Vision
Speech Recognition
Grounding Core
State Machine
Imagination Core
B. Inference C. Execution
Execution Core
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
• Correspondence Problem
• Robot Imitation
• Goal-Directed Actions
• Continuous Goal-Directed
Actions (CGDA)
General Overview
Component Breakdown
State Machine
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
15 of 45
General Overview
Component Breakdown
State Machine
A. Perception
Imagination Core
State Machine
Execution Core
Visual OutputComputer Vision
Grounding Core
Speech OutputSpeech Recognition
B. Inference C. Execution
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
16 of 45
General Overview
Component Breakdown
State Machine
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
16 of 45
General Overview
Component Breakdown
State Machine
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
16 of 45
General Overview
Component Breakdown
State Machine
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Say: "I am ready.
Please tell me."
Store as %2.
Switch %2.
Say: "Oh, sorry.
Could you
please repeat?"
Say: "What?
Was it correct,
yes or no?"
Say: "Okay, perfect.
I've grounded this
information on %1."
Store as %1.
Say: "I understood %1.
Is that correct?".
Switch %1.
Execute touch action
with parameters %1.
Execute draw action
with parameters %1.
Say: "Sorry, I do not
know what that is. Let's
start over again."
Imagine a %1, find
most similar object, and
return its coordinates.
Imagine a %1, and
return coordinates for
drawing a sketch.
Store extracted
features as %1 in
semantic memory.
"No"
Other
Listen
Listen
Listen Listen
"Yes"
Other
"Touch a ..."
"Draw a ..."
"This is ..."
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
General Overview
Component Breakdown
State Machine
16 of 45
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
17 of 45
Perception Block
Inference Block
Execution Block
A. Perception
Computer Vision
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
18 of 45
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
• 2D
• RGB/HSV or Canny Edge Detection
• Morphological Closing
• Find contours (blobs) to segment objects
• 3D
• Remove NaN and Voxelize
• RANSAC using plane model
• Remove planes (tables and walls)
• Euclidian Clustering to segment objects
• Feature extraction
• Centroid, area, rotation, min/max axis,
aspect ratio, rectangularity, solidity, arc,
radius, RGB, HSV.
A. Perception
Computer Vision
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
• Generated semantic subspaces:
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
• Generated semantic subspaces:
• Blue
0
Solidity
Hue
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
• Generated semantic subspaces:
• Blue
• Scissors
0
Solidity
Hue
0
Solidity
Hue
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
0
Solidity
Hue
0
Solidity
Hue
• Generated semantic subspaces:
• Blue
• Scissors
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
0
Solidity
Hue
0
Solidity
Hue
• Generated semantic subspaces:
• Blue
• Scissors
1
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
1
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
0
Solidity
Hue
0
Solidity
Hue
Hue
• Generated semantic subspaces:
• Blue
• Scissors
• Red
Solidity
0
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
1
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
19 of 45
0
Solidity
Hue
0
Solidity
Hue
Hue
• Generated semantic subspaces:
• Blue
• Scissors
• Red
Solidity
0
Grounding Core
A. Perception
Perception Block
Computer Vision
Grounding Core
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
20 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
• Enhanced Prediction Algorithm (RIS)
• Object Recognition (Euclidean Distance)
• Reconstruction (Evolutionary Computation)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
21 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
• Basic Prediction:
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• Basic Prediction:
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• Basic Prediction:
• COMPLETE
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• Basic Prediction:
• COMPLETE
• RELEVANT
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
1. Generalization
2. Intersection
3. Projection
4. Averaging
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
22 of 45
Imagination Core
B. Inference
• COMPLETE
• RELEVANT
• BALANCED
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
23 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why a hyperplane of order n-1 ?
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
23 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why a hyperplane of order n-1 ?
• From the algebraic definition, if the points fit
to the model, it simultaneously captures all of
the linear dependencies and couplings in Rn.
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
23 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why a hyperplane of order n-1 ?
• From the algebraic definition, if the points fit
to the model, it simultaneously captures all of
the linear dependencies and couplings in Rn.
• Hyperplanes are defined in all space, so the
“meanings” of words are also extended across,
and along, all the Feature Space.
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
23 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why a hyperplane of order n-1 ?
• From the algebraic definition, if the points fit
to the model, it simultaneously captures all of
the linear dependencies and couplings in Rn.
• Hyperplanes are defined in all space, so the
“meanings” of words are also extended across,
and along, all the Feature Space.
• Practical benefit of always finding
intersections between these representations.
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
24 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why are orthogonal projections used ?
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
24 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why are orthogonal projections used ?
• Projections result on the valid solution
geometrical construct, and are closest to the
original data of a given point cloud.
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
24 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Why are orthogonal projections used ?
• Projections result on the valid solution
geometrical construct, and are closest to the
original data of a given point cloud.
• Promotes ranges of values that are common in
objects, which is useful when not all features are
specified by words.
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Yes, the basic prediction algorithm may fail under
the following circumstances:
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Yes, the basic prediction algorithm may fail under
the following circumstances:
• Context dependency (e.g. “green” color or state
of ripeness).
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Yes, the basic prediction algorithm may fail under
the following circumstances:
• Context dependency (e.g. “green” color or state
of ripeness).
• Periodicity (e.g. “red” hue is 0º and 360º, a
hyperplane could be fit at 180º).
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
• Basic Prediction:
• Object Feature Prediction:
• Basic Prediction Algorithm (RIS)
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
25 of 45
Imagination Core
B. Inference
1. Generalization
2. Intersection
3. Projection
4. Averaging
• COMPLETE
• RELEVANT
• BALANCED
Does the algorithm have any known limitations?
Yes, the basic prediction algorithm may fail under
the following circumstances:
• Context dependency (e.g. “green” color or state
of ripeness).
• Periodicity (e.g. “red” hue is 0º and 360º, a
hyperplane could be fit at 180º).
• Specificity (non-extendable meanings).
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
26 of 45
Imagination Core
B. Inference
• Object Feature Prediction:
• Enhanced Prediction Algorithm (RIS)
• Enhanced Prediction:
1. Context Detector
2. Hyperspherical
Shape Detector
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
27 of 45
Imagination Core
B. Inference
• Object Feature Prediction:
• Enhanced Prediction Algorithm (RIS)
• Enhanced Prediction:
1. Context Detector
2. Hyperspherical
Shape Detector
The Context Detector counts the accompanying
words of a query word within each cluster
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
28 of 45
Imagination Core
B. Inference
• Object Feature Prediction:
• Enhanced Prediction Algorithm (RIS)
• Enhanced Prediction:
1. Context Detection
2. Hyperspherical
Shape Detector The Hyperspherical Shape Detector compares all
the sorted eigenvalues of a given point cloud
Perception Block
Inference Block
Imagination Core
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
29 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
30 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization (CGDA)
• Action Recognition (CGDA)
• Execution (Evolutionary Computation)
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
31 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization (CGDA)
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
31 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization (CGDA)
• Action Generalization
1. Split in Intervals
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
31 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization (CGDA)
• Action Generalization
1. Split in Intervals
2. Average in Intervals
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
31 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Generalization
1. Split in Intervals
2. Average in Intervals
3. Linear Interpolation
• Action Generalization (CGDA)
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
GeneralizedTraj.e.g.Xpos
Query Traj. e.g. X pos
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
32 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Recognition (CGDA)
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
GeneralizedTraj.e.g.Xpos
Query Traj. e.g. X pos
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
32 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Recognition (CGDA)
• Action Recognition
1. Axis-by-axis comparison
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
GeneralizedTraj.e.g.Xpos
Query Traj. e.g. X pos
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
32 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Recognition (CGDA)
• Action Recognition
1. Axis-by-axis comparison
2. Dynamic Time Warping
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
GeneralizedTraj.e.g.Xpos
Query Traj. e.g. X pos
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
32 of 45
Execution Core
C. Execution
Perception Block
Inference Block
Execution Block
Execution Core
• Action Recognition (CGDA)
• Action Recognition
1. Axis-by-axis comparison
2. Dynamic Time Warping
3. Total cost = sum(DTW)
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
A. Perception
Imagination Core
State Machine
Execution Core
Visual OutputComputer Vision
Grounding Core
Speech OutputSpeech Recognition
B. Inference C. Execution
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
33 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
Each RIS component has been
implemented as a YARP module.
They are direct implementations of
the explained algorithms.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
34 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
1. Populating the database: Every training sample has a description
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
34 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
1. Populating the database: Every training sample has a description,
e.g. the first image is labeled “top-left-dark-blue-fat-straight-box”
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
3. The mental models are created through Object Reconstruction (using
evolutionary computation) aiming at the predicted features, e.g. the first
image responds to the query “bottom right”, the second to “bottom left”,
the third to “top right”, and the last to “top blue”.
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
35 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
2. Queries are summited upon the populated database,
e.g. “bottom right” or “bottom left”
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
36 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
4.The drawing demonstration, where the robot exposes its mental model.
[J.G.Victores, S.Morante et Al., IROS 2013,Tokyo] [FASTCompany, 2013]
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
37 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
-The system is trained through speech with a blue visual marker.
-The component used for Speech Recognition requires a pre-defined
corpus of words to recognize.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
38 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
The system can generalize a word with as little as 2 training
samples in the 2D case, and 3 training samples in the 3D case.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
39 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
Testing the execution of the “waxing” action encoded as a
Continuous Goal-Oriented Action (CGDA).
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
40 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
[S.Morante, J.G.Victores et Al., ICRA 2013, Hong Kong]
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
41 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
42 of 45
Basic Prediction Algorithm: Drawing
Enhanced Prediction: Spatial Language
Execution Core: Waxing
The Token Test
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
43 of 45
Perception
- Framework for creating links between words and
physical characteristics
Inference
- Robot Imagination System for providing robots with
imagination skills through object features and semantics
Execution
- Presented a new way of encoding actions, which can
complement Programming by Demonstration
- Explained proposed algorithms for CGDA
generalization, recognition and execution
Progress beyond the State of Art
Future Lines of Research
Scientific Contributions
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
44 of 45
Progress beyond the State of Art
Future Lines of Research
Scientific Contributions
Origin of Data
- Sensory information of various nature (e.g. haptic,
rugosity) and from the Internet-of-Things.
Data Representation
- Human-inspired color space such as YUV. Use of SIFT and
SURF visual descriptors.
Techniques for Inference
- Feature selection (e.g. LASSO). Gaussian Mixture Models,
Non-linear PCA, Manifolds, Neural Networks.
Additional Functionalities
- Vector and dynamic information, auto-tuned parameters.
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
Introduction
State of the Art
Global Architecture
Perception, Inference and Execution
Experiments
Conclusions
45 of 45
Progress beyond the State of Art
Future Lines of Research
Scientific Contributions
“Action Effect Generalization, Recognition and Execution
through Continuous Goal-Directed Actions”, IEEE ICRA,
Hong Kong, China, 2014.
“Towards Robot Imagination Through Object Feature
Inference”, IEEE/RSJ IROS,Tokyo, Japan, 2013.
“Personal Autonomy Rehabilitation in Home
Environments by a Portable Assistive Robot”, IEEE-
Systems, Man and Cybernetics, Part C, 2011.
Over 27 conference articles, 4 top-quartile journal papers,
3 book chapters, a patent, and artistic works have been
published by the author within the duration of this Thesis.
Robot Imagination System
Author: Juan Carlos González Víctores
Advisors: Carlos Balaguer and Alberto Jardón
Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014

Contenu connexe

Similaire à victores2014thesis-presentation

Design Insights for the Next Wave Ontology Authoring Tools
Design Insights for the Next Wave Ontology Authoring ToolsDesign Insights for the Next Wave Ontology Authoring Tools
Design Insights for the Next Wave Ontology Authoring ToolsMarkel Vigo
 
Better neuroimaging data processing: driven by evidence, open communities, an...
Better neuroimaging data processing: driven by evidence, open communities, an...Better neuroimaging data processing: driven by evidence, open communities, an...
Better neuroimaging data processing: driven by evidence, open communities, an...Gael Varoquaux
 
Invited Tutorial - Cognitive Design for Artificial Minds AI*IA 2022
Invited Tutorial - Cognitive Design for Artificial Minds AI*IA 2022Invited Tutorial - Cognitive Design for Artificial Minds AI*IA 2022
Invited Tutorial - Cognitive Design for Artificial Minds AI*IA 2022Antonio Lieto
 
Virtual/ Physical Co-Existing Design (CapX)
Virtual/ Physical Co-Existing Design (CapX)Virtual/ Physical Co-Existing Design (CapX)
Virtual/ Physical Co-Existing Design (CapX)Kai-Tzu Lu
 
User experience design portfolio, Harry Brenton
User experience design portfolio, Harry Brenton User experience design portfolio, Harry Brenton
User experience design portfolio, Harry Brenton Harry Brenton
 
Virtual/ Physical Co-Existing Design _Capturing Space Interactive Device
Virtual/ Physical Co-Existing Design_Capturing Space Interactive Device   Virtual/ Physical Co-Existing Design_Capturing Space Interactive Device
Virtual/ Physical Co-Existing Design _Capturing Space Interactive Device Kai-Tzu Lu
 
Jia-Bin Huang's Curriculum Vitae
Jia-Bin Huang's Curriculum VitaeJia-Bin Huang's Curriculum Vitae
Jia-Bin Huang's Curriculum VitaeJia-Bin Huang
 
Artificial Intelligence power point presentation document
Artificial Intelligence power point presentation documentArtificial Intelligence power point presentation document
Artificial Intelligence power point presentation documentDavid Raj Kanthi
 
An investigation into the physical build and psychological aspects of an inte...
An investigation into the physical build and psychological aspects of an inte...An investigation into the physical build and psychological aspects of an inte...
An investigation into the physical build and psychological aspects of an inte...Jessica Navarro
 
Final_Semester_Project _Report
Final_Semester_Project _ReportFinal_Semester_Project _Report
Final_Semester_Project _ReportSriram Raghavan
 
Engineering the software of robotic systems - 4 - Current results [ICSE 2017 ...
Engineering the software of robotic systems - 4 - Current results [ICSE 2017 ...Engineering the software of robotic systems - 4 - Current results [ICSE 2017 ...
Engineering the software of robotic systems - 4 - Current results [ICSE 2017 ...Ivano Malavolta
 
Towards a framework for making applications provenance aware: UML2PROV
Towards a framework for making applications provenance aware: UML2PROVTowards a framework for making applications provenance aware: UML2PROV
Towards a framework for making applications provenance aware: UML2PROVUniversidad de La Rioja
 
OVERALL PERFORMANCE EVALUATION OF ENGINEERING STUDENTS USING FUZZY LOGIC
OVERALL PERFORMANCE EVALUATION OF ENGINEERING STUDENTS USING FUZZY LOGICOVERALL PERFORMANCE EVALUATION OF ENGINEERING STUDENTS USING FUZZY LOGIC
OVERALL PERFORMANCE EVALUATION OF ENGINEERING STUDENTS USING FUZZY LOGICIJCI JOURNAL
 
Robotic hand prototype as a didactic model.
Robotic hand prototype as a didactic model.Robotic hand prototype as a didactic model.
Robotic hand prototype as a didactic model.IRJET Journal
 
Opps approch of software development
Opps approch of software developmentOpps approch of software development
Opps approch of software developmentRaja Babu
 

Similaire à victores2014thesis-presentation (20)

Design Insights for the Next Wave Ontology Authoring Tools
Design Insights for the Next Wave Ontology Authoring ToolsDesign Insights for the Next Wave Ontology Authoring Tools
Design Insights for the Next Wave Ontology Authoring Tools
 
Better neuroimaging data processing: driven by evidence, open communities, an...
Better neuroimaging data processing: driven by evidence, open communities, an...Better neuroimaging data processing: driven by evidence, open communities, an...
Better neuroimaging data processing: driven by evidence, open communities, an...
 
Invited Tutorial - Cognitive Design for Artificial Minds AI*IA 2022
Invited Tutorial - Cognitive Design for Artificial Minds AI*IA 2022Invited Tutorial - Cognitive Design for Artificial Minds AI*IA 2022
Invited Tutorial - Cognitive Design for Artificial Minds AI*IA 2022
 
Cap xpresent
Cap xpresentCap xpresent
Cap xpresent
 
Virtual/ Physical Co-Existing Design (CapX)
Virtual/ Physical Co-Existing Design (CapX)Virtual/ Physical Co-Existing Design (CapX)
Virtual/ Physical Co-Existing Design (CapX)
 
User experience design portfolio, Harry Brenton
User experience design portfolio, Harry Brenton User experience design portfolio, Harry Brenton
User experience design portfolio, Harry Brenton
 
Virtual/ Physical Co-Existing Design _Capturing Space Interactive Device
Virtual/ Physical Co-Existing Design_Capturing Space Interactive Device   Virtual/ Physical Co-Existing Design_Capturing Space Interactive Device
Virtual/ Physical Co-Existing Design _Capturing Space Interactive Device
 
Jia-Bin Huang's Curriculum Vitae
Jia-Bin Huang's Curriculum VitaeJia-Bin Huang's Curriculum Vitae
Jia-Bin Huang's Curriculum Vitae
 
Artificial Intelligence power point presentation document
Artificial Intelligence power point presentation documentArtificial Intelligence power point presentation document
Artificial Intelligence power point presentation document
 
report
reportreport
report
 
An investigation into the physical build and psychological aspects of an inte...
An investigation into the physical build and psychological aspects of an inte...An investigation into the physical build and psychological aspects of an inte...
An investigation into the physical build and psychological aspects of an inte...
 
Final_Semester_Project _Report
Final_Semester_Project _ReportFinal_Semester_Project _Report
Final_Semester_Project _Report
 
Engineering the software of robotic systems - 4 - Current results [ICSE 2017 ...
Engineering the software of robotic systems - 4 - Current results [ICSE 2017 ...Engineering the software of robotic systems - 4 - Current results [ICSE 2017 ...
Engineering the software of robotic systems - 4 - Current results [ICSE 2017 ...
 
Cv
CvCv
Cv
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentation
 
Towards a framework for making applications provenance aware: UML2PROV
Towards a framework for making applications provenance aware: UML2PROVTowards a framework for making applications provenance aware: UML2PROV
Towards a framework for making applications provenance aware: UML2PROV
 
CV_CEB_2017_en
CV_CEB_2017_enCV_CEB_2017_en
CV_CEB_2017_en
 
OVERALL PERFORMANCE EVALUATION OF ENGINEERING STUDENTS USING FUZZY LOGIC
OVERALL PERFORMANCE EVALUATION OF ENGINEERING STUDENTS USING FUZZY LOGICOVERALL PERFORMANCE EVALUATION OF ENGINEERING STUDENTS USING FUZZY LOGIC
OVERALL PERFORMANCE EVALUATION OF ENGINEERING STUDENTS USING FUZZY LOGIC
 
Robotic hand prototype as a didactic model.
Robotic hand prototype as a didactic model.Robotic hand prototype as a didactic model.
Robotic hand prototype as a didactic model.
 
Opps approch of software development
Opps approch of software developmentOpps approch of software development
Opps approch of software development
 

Plus de Juan G. Victores

victores2017xgnitive-techfest-presentation
victores2017xgnitive-techfest-presentationvictores2017xgnitive-techfest-presentation
victores2017xgnitive-techfest-presentationJuan G. Victores
 
estevez2016future-presentation
estevez2016future-presentationestevez2016future-presentation
estevez2016future-presentationJuan G. Victores
 
victores2013semantic-presentation
victores2013semantic-presentationvictores2013semantic-presentation
victores2013semantic-presentationJuan G. Victores
 
victores2013towards-presentation
victores2013towards-presentationvictores2013towards-presentation
victores2013towards-presentationJuan G. Victores
 
morante2014on-presentation
morante2014on-presentationmorante2014on-presentation
morante2014on-presentationJuan G. Victores
 
morante2014action-presentation
morante2014action-presentationmorante2014action-presentation
morante2014action-presentationJuan G. Victores
 
morante2015automatic-presentation
morante2015automatic-presentationmorante2015automatic-presentation
morante2015automatic-presentationJuan G. Victores
 

Plus de Juan G. Victores (8)

victores2017xgnitive-techfest-presentation
victores2017xgnitive-techfest-presentationvictores2017xgnitive-techfest-presentation
victores2017xgnitive-techfest-presentation
 
estevez2016future-presentation
estevez2016future-presentationestevez2016future-presentation
estevez2016future-presentation
 
victores2013semantic-presentation
victores2013semantic-presentationvictores2013semantic-presentation
victores2013semantic-presentation
 
estevez2016towards-poster
estevez2016towards-posterestevez2016towards-poster
estevez2016towards-poster
 
victores2013towards-presentation
victores2013towards-presentationvictores2013towards-presentation
victores2013towards-presentation
 
morante2014on-presentation
morante2014on-presentationmorante2014on-presentation
morante2014on-presentation
 
morante2014action-presentation
morante2014action-presentationmorante2014action-presentation
morante2014action-presentation
 
morante2015automatic-presentation
morante2015automatic-presentationmorante2015automatic-presentation
morante2015automatic-presentation
 

Dernier

THE ROLE OF PHARMACOGNOSY IN TRADITIONAL AND MODERN SYSTEM OF MEDICINE.pptx
THE ROLE OF PHARMACOGNOSY IN TRADITIONAL AND MODERN SYSTEM OF MEDICINE.pptxTHE ROLE OF PHARMACOGNOSY IN TRADITIONAL AND MODERN SYSTEM OF MEDICINE.pptx
THE ROLE OF PHARMACOGNOSY IN TRADITIONAL AND MODERN SYSTEM OF MEDICINE.pptxNandakishor Bhaurao Deshmukh
 
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptxECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptxmaryFF1
 
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)riyaescorts54
 
basic entomology with insect anatomy and taxonomy
basic entomology with insect anatomy and taxonomybasic entomology with insect anatomy and taxonomy
basic entomology with insect anatomy and taxonomyDrAnita Sharma
 
Dubai Calls Girl Lisa O525547819 Lexi Call Girls In Dubai
Dubai Calls Girl Lisa O525547819 Lexi Call Girls In DubaiDubai Calls Girl Lisa O525547819 Lexi Call Girls In Dubai
Dubai Calls Girl Lisa O525547819 Lexi Call Girls In Dubaikojalkojal131
 
STOPPED FLOW METHOD & APPLICATION MURUGAVENI B.pptx
STOPPED FLOW METHOD & APPLICATION MURUGAVENI B.pptxSTOPPED FLOW METHOD & APPLICATION MURUGAVENI B.pptx
STOPPED FLOW METHOD & APPLICATION MURUGAVENI B.pptxMurugaveni B
 
User Guide: Pulsar™ Weather Station (Columbia Weather Systems)
User Guide: Pulsar™ Weather Station (Columbia Weather Systems)User Guide: Pulsar™ Weather Station (Columbia Weather Systems)
User Guide: Pulsar™ Weather Station (Columbia Weather Systems)Columbia Weather Systems
 
LIGHT-PHENOMENA-BY-CABUALDIONALDOPANOGANCADIENTE-CONDEZA (1).pptx
LIGHT-PHENOMENA-BY-CABUALDIONALDOPANOGANCADIENTE-CONDEZA (1).pptxLIGHT-PHENOMENA-BY-CABUALDIONALDOPANOGANCADIENTE-CONDEZA (1).pptx
LIGHT-PHENOMENA-BY-CABUALDIONALDOPANOGANCADIENTE-CONDEZA (1).pptxmalonesandreagweneth
 
Call Girls in Majnu Ka Tilla Delhi 🔝9711014705🔝 Genuine
Call Girls in Majnu Ka Tilla Delhi 🔝9711014705🔝 GenuineCall Girls in Majnu Ka Tilla Delhi 🔝9711014705🔝 Genuine
Call Girls in Majnu Ka Tilla Delhi 🔝9711014705🔝 Genuinethapagita
 
Pests of castor_Binomics_Identification_Dr.UPR.pdf
Pests of castor_Binomics_Identification_Dr.UPR.pdfPests of castor_Binomics_Identification_Dr.UPR.pdf
Pests of castor_Binomics_Identification_Dr.UPR.pdfPirithiRaju
 
Pests of safflower_Binomics_Identification_Dr.UPR.pdf
Pests of safflower_Binomics_Identification_Dr.UPR.pdfPests of safflower_Binomics_Identification_Dr.UPR.pdf
Pests of safflower_Binomics_Identification_Dr.UPR.pdfPirithiRaju
 
Best Call Girls In Sector 29 Gurgaon❤️8860477959 EscorTs Service In 24/7 Delh...
Best Call Girls In Sector 29 Gurgaon❤️8860477959 EscorTs Service In 24/7 Delh...Best Call Girls In Sector 29 Gurgaon❤️8860477959 EscorTs Service In 24/7 Delh...
Best Call Girls In Sector 29 Gurgaon❤️8860477959 EscorTs Service In 24/7 Delh...lizamodels9
 
《Queensland毕业文凭-昆士兰大学毕业证成绩单》
《Queensland毕业文凭-昆士兰大学毕业证成绩单》《Queensland毕业文凭-昆士兰大学毕业证成绩单》
《Queensland毕业文凭-昆士兰大学毕业证成绩单》rnrncn29
 
Servosystem Theory / Cybernetic Theory by Petrovic
Servosystem Theory / Cybernetic Theory by PetrovicServosystem Theory / Cybernetic Theory by Petrovic
Servosystem Theory / Cybernetic Theory by PetrovicAditi Jain
 
Pests of Blackgram, greengram, cowpea_Dr.UPR.pdf
Pests of Blackgram, greengram, cowpea_Dr.UPR.pdfPests of Blackgram, greengram, cowpea_Dr.UPR.pdf
Pests of Blackgram, greengram, cowpea_Dr.UPR.pdfPirithiRaju
 
Bioteknologi kelas 10 kumer smapsa .pptx
Bioteknologi kelas 10 kumer smapsa .pptxBioteknologi kelas 10 kumer smapsa .pptx
Bioteknologi kelas 10 kumer smapsa .pptx023NiWayanAnggiSriWa
 
BIOETHICS IN RECOMBINANT DNA TECHNOLOGY.
BIOETHICS IN RECOMBINANT DNA TECHNOLOGY.BIOETHICS IN RECOMBINANT DNA TECHNOLOGY.
BIOETHICS IN RECOMBINANT DNA TECHNOLOGY.PraveenaKalaiselvan1
 
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxGENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxRitchAndruAgustin
 

Dernier (20)

THE ROLE OF PHARMACOGNOSY IN TRADITIONAL AND MODERN SYSTEM OF MEDICINE.pptx
THE ROLE OF PHARMACOGNOSY IN TRADITIONAL AND MODERN SYSTEM OF MEDICINE.pptxTHE ROLE OF PHARMACOGNOSY IN TRADITIONAL AND MODERN SYSTEM OF MEDICINE.pptx
THE ROLE OF PHARMACOGNOSY IN TRADITIONAL AND MODERN SYSTEM OF MEDICINE.pptx
 
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptxECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
ECG Graph Monitoring with AD8232 ECG Sensor & Arduino.pptx
 
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
(9818099198) Call Girls In Noida Sector 14 (NOIDA ESCORTS)
 
Volatile Oils Pharmacognosy And Phytochemistry -I
Volatile Oils Pharmacognosy And Phytochemistry -IVolatile Oils Pharmacognosy And Phytochemistry -I
Volatile Oils Pharmacognosy And Phytochemistry -I
 
basic entomology with insect anatomy and taxonomy
basic entomology with insect anatomy and taxonomybasic entomology with insect anatomy and taxonomy
basic entomology with insect anatomy and taxonomy
 
Dubai Calls Girl Lisa O525547819 Lexi Call Girls In Dubai
Dubai Calls Girl Lisa O525547819 Lexi Call Girls In DubaiDubai Calls Girl Lisa O525547819 Lexi Call Girls In Dubai
Dubai Calls Girl Lisa O525547819 Lexi Call Girls In Dubai
 
STOPPED FLOW METHOD & APPLICATION MURUGAVENI B.pptx
STOPPED FLOW METHOD & APPLICATION MURUGAVENI B.pptxSTOPPED FLOW METHOD & APPLICATION MURUGAVENI B.pptx
STOPPED FLOW METHOD & APPLICATION MURUGAVENI B.pptx
 
User Guide: Pulsar™ Weather Station (Columbia Weather Systems)
User Guide: Pulsar™ Weather Station (Columbia Weather Systems)User Guide: Pulsar™ Weather Station (Columbia Weather Systems)
User Guide: Pulsar™ Weather Station (Columbia Weather Systems)
 
LIGHT-PHENOMENA-BY-CABUALDIONALDOPANOGANCADIENTE-CONDEZA (1).pptx
LIGHT-PHENOMENA-BY-CABUALDIONALDOPANOGANCADIENTE-CONDEZA (1).pptxLIGHT-PHENOMENA-BY-CABUALDIONALDOPANOGANCADIENTE-CONDEZA (1).pptx
LIGHT-PHENOMENA-BY-CABUALDIONALDOPANOGANCADIENTE-CONDEZA (1).pptx
 
Call Girls in Majnu Ka Tilla Delhi 🔝9711014705🔝 Genuine
Call Girls in Majnu Ka Tilla Delhi 🔝9711014705🔝 GenuineCall Girls in Majnu Ka Tilla Delhi 🔝9711014705🔝 Genuine
Call Girls in Majnu Ka Tilla Delhi 🔝9711014705🔝 Genuine
 
Pests of castor_Binomics_Identification_Dr.UPR.pdf
Pests of castor_Binomics_Identification_Dr.UPR.pdfPests of castor_Binomics_Identification_Dr.UPR.pdf
Pests of castor_Binomics_Identification_Dr.UPR.pdf
 
Pests of safflower_Binomics_Identification_Dr.UPR.pdf
Pests of safflower_Binomics_Identification_Dr.UPR.pdfPests of safflower_Binomics_Identification_Dr.UPR.pdf
Pests of safflower_Binomics_Identification_Dr.UPR.pdf
 
Best Call Girls In Sector 29 Gurgaon❤️8860477959 EscorTs Service In 24/7 Delh...
Best Call Girls In Sector 29 Gurgaon❤️8860477959 EscorTs Service In 24/7 Delh...Best Call Girls In Sector 29 Gurgaon❤️8860477959 EscorTs Service In 24/7 Delh...
Best Call Girls In Sector 29 Gurgaon❤️8860477959 EscorTs Service In 24/7 Delh...
 
Let’s Say Someone Did Drop the Bomb. Then What?
Let’s Say Someone Did Drop the Bomb. Then What?Let’s Say Someone Did Drop the Bomb. Then What?
Let’s Say Someone Did Drop the Bomb. Then What?
 
《Queensland毕业文凭-昆士兰大学毕业证成绩单》
《Queensland毕业文凭-昆士兰大学毕业证成绩单》《Queensland毕业文凭-昆士兰大学毕业证成绩单》
《Queensland毕业文凭-昆士兰大学毕业证成绩单》
 
Servosystem Theory / Cybernetic Theory by Petrovic
Servosystem Theory / Cybernetic Theory by PetrovicServosystem Theory / Cybernetic Theory by Petrovic
Servosystem Theory / Cybernetic Theory by Petrovic
 
Pests of Blackgram, greengram, cowpea_Dr.UPR.pdf
Pests of Blackgram, greengram, cowpea_Dr.UPR.pdfPests of Blackgram, greengram, cowpea_Dr.UPR.pdf
Pests of Blackgram, greengram, cowpea_Dr.UPR.pdf
 
Bioteknologi kelas 10 kumer smapsa .pptx
Bioteknologi kelas 10 kumer smapsa .pptxBioteknologi kelas 10 kumer smapsa .pptx
Bioteknologi kelas 10 kumer smapsa .pptx
 
BIOETHICS IN RECOMBINANT DNA TECHNOLOGY.
BIOETHICS IN RECOMBINANT DNA TECHNOLOGY.BIOETHICS IN RECOMBINANT DNA TECHNOLOGY.
BIOETHICS IN RECOMBINANT DNA TECHNOLOGY.
 
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxGENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
 

victores2014thesis-presentation

  • 1. Robot Imagination System Author: Juan Carlos González Víctores Advisors: Carlos Balaguer and Alberto Jardón Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
  • 2. Robot Imagination System  Introduction  State of the Art  General Architecture  Perception  Inference  Execution  Experiments  Conclusions 1 of 45Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014
  • 3. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 2 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 4. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 3 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 5. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 3 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 6. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 3 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 7. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 4 of 45 A. No perceptual links between physical characteristics and words. Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 8. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 4 of 45 A. No perceptual links between physical characteristics and words. B. Lack of inference capabilities for interpreting combinations of words. Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 9. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 4 of 45 A. No perceptual links between physical characteristics and words. B. Lack of inference capabilities for interpreting combinations of words. C. No bindings between actions and their effects, reprogramming required. Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 10. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 5 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties A. Develop a framework for linking physical characteristics and words. B. Enable inference mechanisms for interpreting combinations of words. C. Allow a robot to act according to the desired effects on an object.
  • 11. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 6 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 12. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 6 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties
  • 13. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 7 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties 1. Imagination 2. Grounded Language 3. Spatial Language 4. Execution 5. Integration
  • 14. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 7 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties 1. Imagination We define Robot Imagination as the process of creating previously unexisting Knowledge K’ from existing Knowledge K and Words W. Where new Knowledge K’ can be used for a real world application. - Robot Imagination goes beyond Prediction.
  • 15. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 7 of 45 Origin of this Thesis Main Objectives Target Scenario Expected Novelties 1. Imagination 2. Grounded Language 3. Spatial Language 4. Execution 5. Integration
  • 16. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 8 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution
  • 17. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 9 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution A. No perceptual links between physical characteristics and words.
  • 18. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 9 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution A. No perceptual links between physical characteristics and words.
  • 19. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 10 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution A. No perceptual links between physical characteristics and words.
  • 20. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 10 of 45 ``The horizontal purple rectangle below the horizontal green rectangle'‘ (DESCRIBER, 2002) A. No perceptual links between physical characteristics and words. Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution
  • 21. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 11 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution B. Lack of inference capabilities for interpreting combinations of words.
  • 22. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 11 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution B. Lack of inference capabilities for interpreting combinations of words. (AARON, 1989)
  • 23. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 11 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution B. Lack of inference capabilities for interpreting combinations of words. (AARON, 1989) (AHA! Experience, 2011) !
  • 24. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 12 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution C. No bindings between actions and their effects, reprogramming required.
  • 25. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 12 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution C. No bindings between actions and their effects, reprogramming required.
  • 26. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 12 of 45 Artificial Cognitive Systems The Symbol Grounding Problem Mental Models and Mental Imagery Robot Task Execution C. No bindings between actions and their effects, reprogramming required.
  • 27. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 13 of 45 Robot Imagination System General Overview Component Breakdown State Machine
  • 28. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 14 of 45 General Overview Component Breakdown State Machine A. Perception Imagination Core State Machine Execution Core Visual OutputComputer Vision Grounding Core Speech OutputSpeech Recognition B. Inference C. Execution
  • 29. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 General Overview Component Breakdown State Machine A. Perception
  • 30. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 • Computer Vision General Overview Component Breakdown State Machine A. Perception Computer Vision
  • 31. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 • Computer Vision General Overview Component Breakdown State Machine • Speech Recognition A. Perception Computer Vision Speech Recognition
  • 32. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Speech Recognition Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 • Computer Vision General Overview Component Breakdown State Machine • Semantic Memory A. Perception Computer Vision Speech Recognition
  • 33. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Speech Recognition Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 • Computer Vision General Overview Component Breakdown State Machine A. Perception Computer Vision Speech Recognition Grounding Core • Semantic Memory
  • 34. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 A. Perception Computer Vision Speech Recognition State Machine Grounding Core Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 General Overview Component Breakdown State Machine • Speech Recognition • Computer Vision • Grounded Language • Semantic Memory
  • 35. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 A. Perception Computer Vision Speech Recognition State Machine Grounding Core • Speech Recognition Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 • Semantic Memory • Embodied Cognition • Computer Vision • Grounded Language General Overview Component Breakdown State Machine
  • 36. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 • Philosophy • Statistical Learning • Psychology • Neuro-science (medical) • Neuro-science (computer) General Overview Component Breakdown State Machine A. Perception Computer Vision Speech Recognition Grounding Core State Machine Imagination Core B. Inference
  • 37. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 A. Perception Computer Vision Speech Recognition Grounding Core State Machine Imagination Core B. Inference C. Execution Execution Core Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 • Correspondence Problem • Robot Imitation • Goal-Directed Actions • Continuous Goal-Directed Actions (CGDA) General Overview Component Breakdown State Machine
  • 38. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 15 of 45 General Overview Component Breakdown State Machine A. Perception Imagination Core State Machine Execution Core Visual OutputComputer Vision Grounding Core Speech OutputSpeech Recognition B. Inference C. Execution
  • 39. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 16 of 45 General Overview Component Breakdown State Machine
  • 40. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 16 of 45 General Overview Component Breakdown State Machine
  • 41. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 16 of 45 General Overview Component Breakdown State Machine
  • 42. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions General Overview Component Breakdown State Machine 16 of 45
  • 43. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions General Overview Component Breakdown State Machine 16 of 45
  • 44. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions General Overview Component Breakdown State Machine 16 of 45
  • 45. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions General Overview Component Breakdown State Machine 16 of 45
  • 46. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions General Overview Component Breakdown State Machine 16 of 45
  • 47. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions General Overview Component Breakdown State Machine 16 of 45
  • 48. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Say: "I am ready. Please tell me." Store as %2. Switch %2. Say: "Oh, sorry. Could you please repeat?" Say: "What? Was it correct, yes or no?" Say: "Okay, perfect. I've grounded this information on %1." Store as %1. Say: "I understood %1. Is that correct?". Switch %1. Execute touch action with parameters %1. Execute draw action with parameters %1. Say: "Sorry, I do not know what that is. Let's start over again." Imagine a %1, find most similar object, and return its coordinates. Imagine a %1, and return coordinates for drawing a sketch. Store extracted features as %1 in semantic memory. "No" Other Listen Listen Listen Listen "Yes" Other "Touch a ..." "Draw a ..." "This is ..." Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions General Overview Component Breakdown State Machine 16 of 45
  • 49. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 17 of 45 Perception Block Inference Block Execution Block A. Perception Computer Vision
  • 50. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 18 of 45 Perception Block Computer Vision Grounding Core Inference Block Execution Block • 2D • RGB/HSV or Canny Edge Detection • Morphological Closing • Find contours (blobs) to segment objects • 3D • Remove NaN and Voxelize • RANSAC using plane model • Remove planes (tables and walls) • Euclidian Clustering to segment objects • Feature extraction • Centroid, area, rotation, min/max axis, aspect ratio, rectangularity, solidity, arc, radius, RGB, HSV. A. Perception Computer Vision
  • 51. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 19 of 45 Grounding Core A. Perception Perception Block Computer Vision Grounding Core Inference Block Execution Block
  • 52. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 19 of 45 • Generated semantic subspaces: Grounding Core A. Perception Perception Block Computer Vision Grounding Core Inference Block Execution Block
  • 53. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 19 of 45 • Generated semantic subspaces: • Blue 0 Solidity Hue Grounding Core A. Perception Perception Block Computer Vision Grounding Core Inference Block Execution Block
  • 54. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 19 of 45 • Generated semantic subspaces: • Blue • Scissors 0 Solidity Hue 0 Solidity Hue Grounding Core A. Perception Perception Block Computer Vision Grounding Core Inference Block Execution Block
  • 55. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 19 of 45 0 Solidity Hue 0 Solidity Hue • Generated semantic subspaces: • Blue • Scissors Grounding Core A. Perception Perception Block Computer Vision Grounding Core Inference Block Execution Block
  • 56. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 19 of 45 0 Solidity Hue 0 Solidity Hue • Generated semantic subspaces: • Blue • Scissors 1 Grounding Core A. Perception Perception Block Computer Vision Grounding Core Inference Block Execution Block
  • 57. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 1 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 19 of 45 0 Solidity Hue 0 Solidity Hue Hue • Generated semantic subspaces: • Blue • Scissors • Red Solidity 0 Grounding Core A. Perception Perception Block Computer Vision Grounding Core Inference Block Execution Block
  • 58. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 1 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 19 of 45 0 Solidity Hue 0 Solidity Hue Hue • Generated semantic subspaces: • Blue • Scissors • Red Solidity 0 Grounding Core A. Perception Perception Block Computer Vision Grounding Core Inference Block Execution Block
  • 59. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 20 of 45 Imagination Core B. Inference Perception Block Inference Block Execution Block
  • 60. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Object Feature Prediction: • Basic Prediction Algorithm (RIS) • Enhanced Prediction Algorithm (RIS) • Object Recognition (Euclidean Distance) • Reconstruction (Evolutionary Computation) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 21 of 45 Imagination Core B. Inference Perception Block Inference Block Imagination Core Execution Block
  • 61. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference Perception Block Inference Block Imagination Core Execution Block
  • 62. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference Perception Block Inference Block Imagination Core Execution Block
  • 63. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference Perception Block Inference Block Imagination Core Execution Block
  • 64. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference 1. Generalization • Basic Prediction: Perception Block Inference Block Imagination Core Execution Block
  • 65. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection Perception Block Inference Block Imagination Core Execution Block
  • 66. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection Perception Block Inference Block Imagination Core Execution Block
  • 67. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection Perception Block Inference Block Imagination Core Execution Block
  • 68. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection Perception Block Inference Block Imagination Core Execution Block
  • 69. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • Basic Prediction: Perception Block Inference Block Imagination Core Execution Block
  • 70. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • Basic Prediction: • COMPLETE Perception Block Inference Block Imagination Core Execution Block
  • 71. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • Basic Prediction: • COMPLETE • RELEVANT Perception Block Inference Block Imagination Core Execution Block
  • 72. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 1. Generalization 2. Intersection 3. Projection 4. Averaging • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 22 of 45 Imagination Core B. Inference • COMPLETE • RELEVANT • BALANCED Perception Block Inference Block Imagination Core Execution Block
  • 73. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 23 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Why a hyperplane of order n-1 ? Perception Block Inference Block Imagination Core Execution Block
  • 74. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 23 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Why a hyperplane of order n-1 ? • From the algebraic definition, if the points fit to the model, it simultaneously captures all of the linear dependencies and couplings in Rn. Perception Block Inference Block Imagination Core Execution Block
  • 75. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 23 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Why a hyperplane of order n-1 ? • From the algebraic definition, if the points fit to the model, it simultaneously captures all of the linear dependencies and couplings in Rn. • Hyperplanes are defined in all space, so the “meanings” of words are also extended across, and along, all the Feature Space. Perception Block Inference Block Imagination Core Execution Block
  • 76. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 23 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Why a hyperplane of order n-1 ? • From the algebraic definition, if the points fit to the model, it simultaneously captures all of the linear dependencies and couplings in Rn. • Hyperplanes are defined in all space, so the “meanings” of words are also extended across, and along, all the Feature Space. • Practical benefit of always finding intersections between these representations. Perception Block Inference Block Imagination Core Execution Block
  • 77. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 24 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Why are orthogonal projections used ? Perception Block Inference Block Imagination Core Execution Block
  • 78. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 24 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Why are orthogonal projections used ? • Projections result on the valid solution geometrical construct, and are closest to the original data of a given point cloud. Perception Block Inference Block Imagination Core Execution Block
  • 79. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 24 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Why are orthogonal projections used ? • Projections result on the valid solution geometrical construct, and are closest to the original data of a given point cloud. • Promotes ranges of values that are common in objects, which is useful when not all features are specified by words. Perception Block Inference Block Imagination Core Execution Block
  • 80. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 25 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Does the algorithm have any known limitations? Perception Block Inference Block Imagination Core Execution Block
  • 81. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 25 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Does the algorithm have any known limitations? Yes, the basic prediction algorithm may fail under the following circumstances: Perception Block Inference Block Imagination Core Execution Block
  • 82. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 25 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Does the algorithm have any known limitations? Yes, the basic prediction algorithm may fail under the following circumstances: • Context dependency (e.g. “green” color or state of ripeness). Perception Block Inference Block Imagination Core Execution Block
  • 83. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 25 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Does the algorithm have any known limitations? Yes, the basic prediction algorithm may fail under the following circumstances: • Context dependency (e.g. “green” color or state of ripeness). • Periodicity (e.g. “red” hue is 0º and 360º, a hyperplane could be fit at 180º). Perception Block Inference Block Imagination Core Execution Block
  • 84. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 • Basic Prediction: • Object Feature Prediction: • Basic Prediction Algorithm (RIS) Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 25 of 45 Imagination Core B. Inference 1. Generalization 2. Intersection 3. Projection 4. Averaging • COMPLETE • RELEVANT • BALANCED Does the algorithm have any known limitations? Yes, the basic prediction algorithm may fail under the following circumstances: • Context dependency (e.g. “green” color or state of ripeness). • Periodicity (e.g. “red” hue is 0º and 360º, a hyperplane could be fit at 180º). • Specificity (non-extendable meanings). Perception Block Inference Block Imagination Core Execution Block
  • 85. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 26 of 45 Imagination Core B. Inference • Object Feature Prediction: • Enhanced Prediction Algorithm (RIS) • Enhanced Prediction: 1. Context Detector 2. Hyperspherical Shape Detector Perception Block Inference Block Imagination Core Execution Block
  • 86. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 27 of 45 Imagination Core B. Inference • Object Feature Prediction: • Enhanced Prediction Algorithm (RIS) • Enhanced Prediction: 1. Context Detector 2. Hyperspherical Shape Detector The Context Detector counts the accompanying words of a query word within each cluster Perception Block Inference Block Imagination Core Execution Block
  • 87. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 28 of 45 Imagination Core B. Inference • Object Feature Prediction: • Enhanced Prediction Algorithm (RIS) • Enhanced Prediction: 1. Context Detection 2. Hyperspherical Shape Detector The Hyperspherical Shape Detector compares all the sorted eigenvalues of a given point cloud Perception Block Inference Block Imagination Core Execution Block
  • 88. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 29 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block
  • 89. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 30 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Generalization (CGDA) • Action Recognition (CGDA) • Execution (Evolutionary Computation)
  • 90. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 31 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Generalization (CGDA)
  • 91. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 31 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Generalization (CGDA) • Action Generalization 1. Split in Intervals
  • 92. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 31 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Generalization (CGDA) • Action Generalization 1. Split in Intervals 2. Average in Intervals
  • 93. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 31 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Generalization 1. Split in Intervals 2. Average in Intervals 3. Linear Interpolation • Action Generalization (CGDA)
  • 94. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 GeneralizedTraj.e.g.Xpos Query Traj. e.g. X pos Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 32 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Recognition (CGDA)
  • 95. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 GeneralizedTraj.e.g.Xpos Query Traj. e.g. X pos Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 32 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Recognition (CGDA) • Action Recognition 1. Axis-by-axis comparison
  • 96. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 GeneralizedTraj.e.g.Xpos Query Traj. e.g. X pos Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 32 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Recognition (CGDA) • Action Recognition 1. Axis-by-axis comparison 2. Dynamic Time Warping
  • 97. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 GeneralizedTraj.e.g.Xpos Query Traj. e.g. X pos Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 32 of 45 Execution Core C. Execution Perception Block Inference Block Execution Block Execution Core • Action Recognition (CGDA) • Action Recognition 1. Axis-by-axis comparison 2. Dynamic Time Warping 3. Total cost = sum(DTW)
  • 98. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 A. Perception Imagination Core State Machine Execution Core Visual OutputComputer Vision Grounding Core Speech OutputSpeech Recognition B. Inference C. Execution Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 33 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test Each RIS component has been implemented as a YARP module. They are direct implementations of the explained algorithms.
  • 99. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 34 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test 1. Populating the database: Every training sample has a description
  • 100. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 34 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test 1. Populating the database: Every training sample has a description, e.g. the first image is labeled “top-left-dark-blue-fat-straight-box”
  • 101. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 35 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test 3. The mental models are created through Object Reconstruction (using evolutionary computation) aiming at the predicted features, e.g. the first image responds to the query “bottom right”, the second to “bottom left”, the third to “top right”, and the last to “top blue”. 2. Queries are summited upon the populated database, e.g. “bottom right” or “bottom left”.
  • 102. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 3. The mental models are created through Object Reconstruction (using evolutionary computation) aiming at the predicted features, e.g. the first image responds to the query “bottom right”, the second to “bottom left”, the third to “top right”, and the last to “top blue”. Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 35 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test 2. Queries are summited upon the populated database, e.g. “bottom right” or “bottom left”
  • 103. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 3. The mental models are created through Object Reconstruction (using evolutionary computation) aiming at the predicted features, e.g. the first image responds to the query “bottom right”, the second to “bottom left”, the third to “top right”, and the last to “top blue”. Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 35 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test 2. Queries are summited upon the populated database, e.g. “bottom right” or “bottom left”
  • 104. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 3. The mental models are created through Object Reconstruction (using evolutionary computation) aiming at the predicted features, e.g. the first image responds to the query “bottom right”, the second to “bottom left”, the third to “top right”, and the last to “top blue”. Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 35 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test 2. Queries are summited upon the populated database, e.g. “bottom right” or “bottom left”
  • 105. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 3. The mental models are created through Object Reconstruction (using evolutionary computation) aiming at the predicted features, e.g. the first image responds to the query “bottom right”, the second to “bottom left”, the third to “top right”, and the last to “top blue”. Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 35 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test 2. Queries are summited upon the populated database, e.g. “bottom right” or “bottom left”
  • 106. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 36 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test 4.The drawing demonstration, where the robot exposes its mental model. [J.G.Victores, S.Morante et Al., IROS 2013,Tokyo] [FASTCompany, 2013]
  • 107. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 37 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test -The system is trained through speech with a blue visual marker. -The component used for Speech Recognition requires a pre-defined corpus of words to recognize.
  • 108. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 38 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test The system can generalize a word with as little as 2 training samples in the 2D case, and 3 training samples in the 3D case.
  • 109. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 39 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test Testing the execution of the “waxing” action encoded as a Continuous Goal-Oriented Action (CGDA).
  • 110. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 40 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test [S.Morante, J.G.Victores et Al., ICRA 2013, Hong Kong]
  • 111. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 41 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test
  • 112. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 42 of 45 Basic Prediction Algorithm: Drawing Enhanced Prediction: Spatial Language Execution Core: Waxing The Token Test
  • 113. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 43 of 45 Perception - Framework for creating links between words and physical characteristics Inference - Robot Imagination System for providing robots with imagination skills through object features and semantics Execution - Presented a new way of encoding actions, which can complement Programming by Demonstration - Explained proposed algorithms for CGDA generalization, recognition and execution Progress beyond the State of Art Future Lines of Research Scientific Contributions
  • 114. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 44 of 45 Progress beyond the State of Art Future Lines of Research Scientific Contributions Origin of Data - Sensory information of various nature (e.g. haptic, rugosity) and from the Internet-of-Things. Data Representation - Human-inspired color space such as YUV. Use of SIFT and SURF visual descriptors. Techniques for Inference - Feature selection (e.g. LASSO). Gaussian Mixture Models, Non-linear PCA, Manifolds, Neural Networks. Additional Functionalities - Vector and dynamic information, auto-tuned parameters.
  • 115. Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014 Introduction State of the Art Global Architecture Perception, Inference and Execution Experiments Conclusions 45 of 45 Progress beyond the State of Art Future Lines of Research Scientific Contributions “Action Effect Generalization, Recognition and Execution through Continuous Goal-Directed Actions”, IEEE ICRA, Hong Kong, China, 2014. “Towards Robot Imagination Through Object Feature Inference”, IEEE/RSJ IROS,Tokyo, Japan, 2013. “Personal Autonomy Rehabilitation in Home Environments by a Portable Assistive Robot”, IEEE- Systems, Man and Cybernetics, Part C, 2011. Over 27 conference articles, 4 top-quartile journal papers, 3 book chapters, a patent, and artistic works have been published by the author within the duration of this Thesis.
  • 116. Robot Imagination System Author: Juan Carlos González Víctores Advisors: Carlos Balaguer and Alberto Jardón Doctorado en Ingeniería Eléctrica, Electrónica y Automática – Julio 2014