PPT used during my speech during NASSCOM's BRAINS Event in Hyderabad, Sep 2019. This covers the emerging trends in Inferencing for Artificial Intelligence. PPT discusses about Edge Computing, VPUs, TPUs, GPUs etc.

1. © 2019 MUST India
Trends in Inference for AI
N. Gopi Krishna
Vice President
MUST Research

2. © 2019 MUST India
• Inference = (noun) a conclusion reached on the basis of evidence and reasoning.
• Etymologically, the word means to "carry forward"
• Studied from the times of Aristotle (Atleast)
Inference – Dictionary meaning

3. © 2019 MUST India
• Inference Engine is a component of the system that applies logical rules to the knowledge base to
deduce new information.
• Traditional
• Expert Systems
• Forward Chaining, Backward Chaining etc.
• Deep Learning world
• The hardware where the DL Model is executed on new data to obtain the result
Inference in AI

4. © 2019 MUST India
When does it happen?
Past
Data
Training
Process
AI Model
New
Data
AI Model Result

5. © 2019 MUST India
• Naive approach
• Run everything on a computer
• Becomes a problem when inference is a complex DL model involving CNNs, RNNs etc
• Why?
• Cost, Time, Complexity
• Maintenance (which again is cost and time)
• Alternates?
• Meet the business :-)
Inference – How is it done

6. © 2019 MUST India
• Dedicated Servers in a server farm
• Cloud
• IaaS
• GPU as a Service
• AI as a Service
• TPU (TPU on Edge)
• On the device
• Edge Computing
• Dedicated hardware for specific tasks
• Intel NCS, Open VINO
• Qualcomm SNPE
• NVIDIA Jetson, EGX
• FPGA, ASIC, DSP
Inference Approaches

7. © 2019 MUST India
• A parallel computing platform and programming model
• Developed by NVIDIA for general computing on GPUs.
• Dramatically speeds up computing applications by harnessing the power of GPUs.
• Developers can code in C, C++, Fortran, Python, Matlab etc and achieve parallelism using keywords
• Compiled using nvcc etc available from CUDA Toolkit
CUDA

8. © 2019 MUST India
• Visual Inference and Network Optimization toolkit
• distributed by Intel
• Contains tools for converting a model written in Tensorflow, Caffe, into formats that can be
understood by Intel hardware like Neural Computing Stick etc
Intel OpenVINO

9. © 2019 MUST India
• SnapdragonTM Neural Processing Engine by Qualcomm
• A set of tools which help in converting models built in Tensorflow, Caffe to formats that can be
understood by Qualcomm’s Snapdragon hardware
Qualcomm SNPE

10. © 2019 MUST India
• OpenCL
• Vulkan
• Direct 3D12
• Mantle (foundation of Vulkan)
• Metal from iOS
Manage the complexity

11. © 2019 MUST India
Thanks
N. Gopi Krishna
Vice President
MUST Research