The term 'multimodal interactions' (MMI) on the HCI field refers to the situation where a system offers many ways of interaction. This talk combines an academic subject with the daily JS we already know. It is an effort to bring together two, at first, different worlds looking for a win-win situation. On the one hand, academia could enjoy getting a novel approach to a particular problem thanks to our awesome JS. On the other, the industry can get new ways of interaction which can be applied on a variety of context and/or products. In short, the talk contains: - A quick introduction to MMI (Multimodal Interactions) - A particular novel approach to add support for MMI on Web Apps. - Short list of related open problems or possible research lines to pursuit.

2. Multimodal Interactions
The what, the why and the how
Diego Paez, Despegar

3. About Me
Diego Paez => DEKΔ
G E U T
co-founder
JS developer with a
passion for HCI ♥
Currently working at:

4. AGENDA
➔ Definitions
➔ Foundations
➔ An alternative approach
➔ Future work

5. What is a
multimodal
interaction (MMI)?
#The what

6. Human
Computer
Interaction
Feedback&

7. Senses
5 major senses: sight, hearing, touch,
taste and smell. But there are more; eg:
Proprioception.
You Real
World
Interaction

8. Highway 401 by Nayuki
Senses

9. But, why?
#The why

10. Why?
➔ Lowering barriers to HCI. Eg: gestures
research.
➔ Accessibility [1]
➔ Killer feature
➔ IoT

11. Foundations
#The how

12. Architecture proposed
by Dumas et al [2]
1
2
3

13. An approach
#The how

14. Plusultra Gyes

15. Proposed Architecture
Modalities
(HW, distributed)
Pub/sub pattern
used to separate
layers
Web Clients
(distributed)
1
2
3

16. Proposed Architecture

17. Getting started
npm install plusultra
Docker-compose up
mkdir myMultimodalDeviceDriver
cd myMultimodalDeviceDriver
npm install gyes

18. Creating a MM Device Driver
var Inherits = require( 'inherits' );
var ModalityChannel = require( 'gyes' ).ModalityDriver;
module.exports = MyModality;
function MyModality( opts ){
// constructor code...
}
Inherits(MyModality, ModalityChannel);
MyModality.prototype.fission = function( time ){ … };
MyModality.prototype.newEvent = function( data ){ … };

19. Inside the app
// lets create a new plusultra client
plusultraClient = new gyes( app_key, socketURI, options );
plusultraClient.authenticate( app_key );
// create a new modality (this is like metadata)
var myMod = new gyes.Modality( modName, 'input', {} );
// instance a driver for the modality
var modDriver = new myMultimodalDeviceDriver( options );
myMod.use( modDriver );
// finally add the modality into the system
plusultraClient.addModality( app_key, airMod );

20. Future Work
aka: Help is welcome!

21. ➔ Experimenting with WebRTC to enable P2P interactions.
➔ Define Interpretations as components. W3C MMI. React can
help?
➔ Improve API. Follow a reactive/functional approach. Cycle/rxjs?
Future Work

22. Project repos
➔ Plusultra: https://github.com/dpaez/plusultra
➔ Gyes: https://github.com/dpaez/gyes
➔ Demo app (shapes): https://github.com/dpaez/shapes_app
➔ Gitter: https://gitter.im/gyes_mmi/Lobby

23. Thanks!
Who is the hipster?
Follow me @carax
Github @dpaez
Get the GEUT experience: geutstudio.com / Some cool code @geut