Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
•
0 j'aime•2,507 vues
Warp 10 Platform presentation done at a Criteo Beer & Tech event on 2016-02-03
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
Similaire à Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03 (20)
Plus de Mathias Herberts
Plus de Mathias Herberts (8)
Dernier
Dernier (20)
Warp 10 Platform Presentation - Criteo Beer & Tech 2016-02-03
- 1. Warp 10 Mathias Herberts - @herberts @warp10io – @CityzenData
- 2. Introduction
- 3. Our singular history First focus on Smart Fabric Then all verticals with sensors
- 4. Architecture
- 14. Security
- 15. Privacy / Integrity / Authorization
- 16. Ingesting Data
- 17. Warp 10 manages Geo Time Series® +
- 18. 1411725015746208/48:-4.558/23000 water.temp{sensorid=42,sensornet=X} 16.9 TIMESTAMP [LOCATION] [ELEVATION] CLASS LABELS VALUE Timestamp confgurable from ms to ns precision Location stored with 32 bits precision for latitude and longitude Elevation is a 64 bits LONG, assumed to be expressed in mm Class can be any valid UTF-8 sequence (URL % Encoded) – type of sensor Labels are immutable K/V pairs of UTF-8 sequences (URL % encoded) – id in class Attributes are mutable K/V pairs which can be associated with a Geo Time Series (GTS) Values can be LONG, DOUBLE, BOOLEAN or UTF-8 STRING
- 19. Pushing data is a HTTP request away! POST /api/v0/update HTTP/1.1 Host: warp.cityzendata.net Transfer-Encoding: chunked Content-Type: text/plain ← we also support application/gzip with a gzipped body X-Warp10-Token: MySecretToken X-Warp10-Now: 1442558287045000 Connection: close 1442558287045000// linux.proc.vmstat.pgmajfault{cell=prod,dc=rbx4,hname=prod-6,rack=55c14} 15429 1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0 1442558287045000// linux.proc.vmstat.pgrefill_dma32{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 992 1442558287045000// linux.proc.vmstat.pgrefill_normal{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 1321759 1442558287045000// linux.proc.vmstat.pgrefill_movable{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0 =1442558287046000// 0 =1442558287047000// 0 =1442558287048000// 0 // foo{label=value} 'String%20value' =T+1000// bar{label=value} T =T-1000// foobar{label=value} 1234.5
- 20. Or pushed on a WebSocket UPDATE MySecretToken 1442558287045000// linux.proc.vmstat.pgrefill_dma{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0 1442558287045000// linux.proc.vmstat.pgrefill_dma32{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 992 1442558287045000// linux.proc.vmstat.pgrefill_normal{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 1321759 1442558287045000// linux.proc.vmstat.pgrefill_movable{cell=prod,dc=rbx4,hname=prod-2,rack=55c14} 0 =1442558287046000// 0 =1442558287047000// 0 =1442558287048000// 0 // foo{label=value} 'String%20value' =T+1000// bar{label=value} T =T-1000// foobar{label=value} 1234.5
- 23. A stack based language designed for working with Geo Time Series®
- 24. A feature rich language 600+functions 5high level frameworks BUCKETIZE MAP REDUCE APPLY FILTER Loops, Conditionals, Server Side Macros, Java UDFs
- 25. Support for alternate scripting languages JSR 223
- 26. No SDK, simply use HTTP and JSON libs POST /dist/api/v0/exec HTTP/1.1 Host: warp.cityzendata.net Content-Length: 358 content-type: application/x-www-form-urlencoded; charset=UTF-8 Referer: https://api0.cityzendata.net/widgets/static/ 'lMYI.....Kpnl9Lk' 'token' STORE NOW 'now' STORE 'PT2h' DURATION 'duration' STORE // Fetch raw data [ $token '~.*ingress.*requests' { 'cell' 'prod' } $now $duration ] FETCH // Compute QPS [ SWAP mapper.rate 1 0 0 ] MAP 'Hello' NOW
- 27. Build complex systems WEBCALL allows to issue HTTP calls from within WarpScript UPDATE enables storing new data from within WarpScript META can set attributes for specifc Geo Time Series® Runner can schedule WarpScript programs periodically Right within Warp 10
- 28. Visualizing Data
- 29. Quantum, a WarpScript execution widget
- 30. Quantum-viz graphing widget <!doctype html> <html> <head> <meta name="viewport" content="width=device-width, minimum-scale=1.0, initial-scale=1.0, user-scalable=yes"> <script src="https://api0.cityzendata.net/widgets/quantumviz/webcomponentsjs/webcomponents.js"></script> <link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/polymer/polymer.html"> <link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/warp10-quantumviz/warp10-quantumviz.html"> <body> <warp10-quantumviz width="500" height="400" show-axis="true" tooltip="true" line-width="2" reload="0" host="https://warp.cityzendata.net/dist" > NEWGTS 1 720 <% DUP 'i' STORE 10000000 * NaN NaN NaN $i TORADIANS COS ADDVALUE %> FOR [ SWAP ] 'gts' STORE [ { 'color' '#00d4ff' 'key' 'Sine' } ] 'params' STORE { 'interpolate' 'linear' } 'globalParams' STORE { 'gts' $gts 'params' $params 'globalParams' $globalParams } </warp10-quantumviz> </body> </html>
- 31. Integration
- 32. WarpScript <3 Processing 800 'width' STORE 800 'height' STORE $width $height '2D' PGraphics 400.0 'maxspeed' STORE 40000.0 'maxalt' STORE 3.0 2.0 2.0 @orbit/heatmap/kernel/triangular 'kernel' STORE @orbit/heatmap/palette/classic 'palette' STORE 'MULTIPLY' PblendMode 'CENTER' PimageMode 'TOKEN''token' STORE [ $token '~(ALT|CAS)' {} NOW -2000000 ] FETCH DUP 0 GET LASTTICK 'now' STORE [ SWAP bucketizer.last $now STU 0 ] BUCKETIZE // Create heatmap <% 7 GET LIST-> DROP 'CAS' STORE 'ALT' STORE <% $CAS ISNULL NOT $ALT ISNULL NOT && %> <% $kernel $CAS $maxspeed / $width * $ALT $maxalt / 1.0 SWAP - $height * Pimage %> IFT 0 NaN NaN NaN NULL %> MACROREDUCER 'GRAPHER' STORE [ SWAP [] $GRAPHER ] REDUCE DROP // Colorize Ppixels <% DROP Palpha $palette SWAP GET %> LMAP PupdatePixels Pencode Pdecode $width $height '2D' PGraphics // Do the grid PnoFill 0 0 $width 1 - $height 1 - Prect 2.0 PstrokeWeight 200.0 Pcolor Pstroke 250.0 $maxspeed / $width * DUP 0 SWAP $height Pline 0 10000 $maxalt / 1.0 SWAP - $height * DUP $width SWAP Pline SWAP 0 0 Pimage Pencode
- 33. Quantum-image image display widget <!doctype html> <html> <head> <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css"> <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css"> <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script> <script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.5/js/bootstrap.min.js"></script> <script src="https://api0.cityzendata.net/widgets/quantumviz/webcomponentsjs/webcomponents.js"></script> <link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/polymer/polymer.html"> <link rel="import" href="https://api0.cityzendata.net/widgets/quantumviz/warp10-quantumviz/quantumviz-warpscript-image.html"> <body> <warp10-img width="300" height="300" reload="0" host="https://warp.cityzendata.net/dist"> 200 'width' CSTORE 200 'height' CSTORE $width $height '2D' Pgraphics Ppixels <% DROP DROP RAND 0xFFFFFFFF * TOLONG %> LMAP PupdatePixels Pencode </warp10-img> </body> </html>
- 34. Use any JavaScript library
- 36. Plasma and Mobius Stream raw data (Plasma) or WarpScript results (Mobius)
- 37. Use WarpScript from other tools
- 38. Use Cases
- 39. Provide, replace or complement these Time series databases Data historians IoT analytics platforms OpenTSDB / KairosDB / InfluxDB / Prometheus OSISoft Pi / Proprietary solutions / Sensor data processing solution Monitoring / alerting platform Time series manipulation library
- 41. contact @ cityzendata . com