Infrared simulation and processing on Nvidia platforms
•
0 j'aime•48 vues
Simulation and processing of Infrared images and video on Nvidia Platforms
Recommandé
Contenu connexe
Similaire à Infrared simulation and processing on Nvidia platforms
Similaire à Infrared simulation and processing on Nvidia platforms (20)
Plus de Yoss Cohen
Plus de Yoss Cohen (20)
Dernier
Dernier (20)
Infrared simulation and processing on Nvidia platforms
- 1. Infrared simulation and processing On Nvidia Platforms
- 2. Neuronics • Developing projects in • Computer vision & AI • 3D generation and simulation • Nvidia Edge and server platforms and SDKs • Defense / Military • UAV / Autonomous vehicles • Autonomous vassals • robots • Medical COMPUTER VISION & AI NVIDIA PLATFORMS 3D GENERATION & SIMULATION
- 3. Agenda •Solve IR camera processing challenges
- 5. Camera connection •Camera Link -> Capture card •GMSL – Fiber ( for distanced camera) •USB •GIGE •HDMI •RAW Transfer
- 6. Noise •DSNU - Dark Signal Non-Uniformity ~1% bias •PRNU - Photo-Response Non-Uniformity ~1% gain variability •Thermal noise •Sensor •Gain •Temp
- 7. Data problems •Capture & Label Data – expensive / not possible •Augmentation - Sensitive •Just changing brightness like in RGB images wont work! •Simulation •3D World simulation (Real, Blender, Omniverse, Unity) •Thermal assignment (simple) • Glitter simulation / Ray tracing •Atmospheric simulation (Frequency, distance, angle)
- 8. Simulation – Basic 3D World simulation Temperature Map Sensor Simulation Distance Map Glitter Map Shared config
- 9. Sensor Simulation – Advance Temperature to electrons Lens Shading PSF ) Point spread function( Motion Blur PRNU / DSNU NUC Simulation Shot Noise Electrons to sensor out Temperature map Distance Map Glitter Map Per frame exposure Distance Map Wave Gliter to sensor out The simulation is not standalone since exposure is dynamic and changed by the processing pipe
- 10. Frame Processor Inverse PSFs Fix NUC Exposure estimation To sensor simulation
- 11. Target Acquisition Double Filter (Hadar) Morph operations Target to track association Kalman filtering Main target selection + reporting
- 12. IR Recording and transfer •We DON’T compress IR sensor output and pixel targets •Store in raw format 8/10/12 bit!!! •Sometimes we have problem in streaming it •Lossless JPEG, Lossless AV1 •Process at the edge - > Jetsons!!! •Stream Raw over fiber using NVIDIA Rivermax
- 13. Nvidia RiverMax •Nvidia architecture for stream 10s-100s of GB direct to GPU •Used to stream RAW 8/10/12bit video 4K •Broadcasting quality
- 14. NVIDIA Video Analytics flow Sensors Actions Capture and Decode Pre-processing & Batching AI Inference Tracking Composition Business Rules and Analytics … … H.265
- 15. Processing Pipe requirements • Process stream/s of video (and audio/other data synchronously) • Block based for each unit/function • Easy connection between camera / socket / encoder / decoder / file / parser • Ability to reuse and enhance existing modules (“Inheritance” + open source) • Fix all the threads / buffers on its own • Runs on Linux
- 16. Edge Processing Architecture •DeepStream Nvidia Standard processing architecture – Not used •Python CPP Processing •Basic filtering using OpenCV/CUDA , RAPIDs • Double Filter – DoG • Inverse PSF • Neural Networks • Segmentation • Binary Operations •Tracking – MTT with Kalman/EKF/UKF