Tutorial on Point Cloud Compression and standardisation given at IEEE VCIP 2017 in december. I provide the techniques for point cloud compression and the designed quality metrics and codecs in my PhD at CWI. I detail the standardisation activity on point cloud compression that I started in 2014 and that started in 2017 involving all mobile device makers like Apple, Huawei, Sony, Samsung and Nokia.
1. Point Cloud Compression Tutorial
Rufael Mekuria (PhD), Unified Streaming
and point cloud compression
IEEE Visual Communications and Image Processing 2017
St. Petersburg Florida
Invited Tutorial
rufael@unified-streaming.com
Point Cloud
Compression
2. About Unified Streaming
Unified Streaming
Creator of video streaming software behind many of the large scale early deployments
of HTTP Streaming (BBC iplayer etc.., NPO) www.unified-streaming.com key
implementer of technologies like DASH, HLS, CMAF et… used by CDN, Telco, cloud,
broadcaster etc.
MPEG affiliate, DASH-IF Member
DASH, CMAF, Network based media processing
5G Video streaming pioneer
Involved in EU H2020 project Superfluidity with companies like Intel, RedHat and
Nokia. Helped many telco/CDN on advanced streaming architectures
Point Cloud
Compression
3. About Me
Academic Background
MSEE, Delft 2011, PhD CWI/Vrije Universiteit Amsterdam jan’17
25+ peered reviewed papers, best paper in ACM MMSys’13, 5+ invited papers in
conferences like SPIE, ICASSP, etc…. Invited talks: UIUC, Huawei, Google etc…
Professional Background
TNO (2010-2011), CWI 2011-2016, Unified Streaming 2016-date, PCC (2017-)
Active in large scale EU Funded projects: Superfluidity H2020, Reverie FP7, developed one
of the first large scale tele-immersive system combining real and CGI content
Standardization Work
Chaired and pioneered PCC in MPEG (since jan.2014) CfP was responded by 9 companies
in oct. 2017 including all major mobile device vendors (CfP response referred to as
“Historic” by MPEG convenor).
Point Cloud
Compression
4. About this Tutorial
Before the break:
Introduction to this (re-) emerging research area, point clouds for VR, point
cloud compression architecture
After the break:
Standardization activity on point cloud compression in MPEG
Applications and requirements, quality assessments
State of art and research challenges
Copyrights & Acknowledgement:
Some of the images and parts of this tutorial fall under copyright of
respective contributors/authors as noted to MPEG CfP. I do not name the
contributors explicitly for now. Many opinions in this work are derived from
standardization meetings representing rough industry consensus. I
acknowledge 8i, Microsoft, Technicolor, Ford, Mitshubishi, et al. for
contribution of test data. I thank Technicolor for large contributions to this
body of work.
Point Cloud
Compression
5. A collection of points
Not related to each other
Typically no order
Typically no local topology (no mesh!)
Each is point is the given of
a position (X,Y,Z)
a color (R,G,B) or (Y,U,V)
possibly other things like transparency, time of
acquisition, etc.
Point Cloud
Compression
Point cloud content from Microsoft
research laboratory Donated to MPEG
Point Cloud Format
6. .ply files =
an example raw data format for point cloud
This is an example raw point coud file format
(compare to YUV for video coding)
How many points?
Static case up to several tens of millions,
depending on the application
Dynamic case ~1 million per frame, 30 fps
Probably more is needed for good VR
Format
Geometry XYZ
Fixed precision for VR applications
Float still often used
Colors RGB
As usual, integer 8/10 bits.
Possibly other attributes
(not present in the ply file here on the left)
plyformat ascii 1.0
element vertex 764940
property float x
property float y
property float z
property uchar red
property uchar green
property uchar blue
end_header
211 63 63 127 98 73
213 63 61 134 109 87
212 62 63 122 97 75
212 63 62 129 102 79
212 63 63 124 98 76
213 62 63 122 98 76
213 63 62 128 104 81
213 63 63 124 99 78
215 61 63 120 97 76
214 63 60 141 117 95
214 63 61 135 111 89
215 63 60 144 120 97
215 63 61 133 109 87
214 62 62 126 102 80
214 62 63 122 98 77
214 63 62 128 104 82
etc.
one point X Y Z R G B
no order! Swapping points
does not change the data
Point Cloud
Compression
7. An application of point cloud: free-view point (6DoF) for sport
Scene model
360°/omnidirectional background
reshaping depending on viewpoint
3D object
occlusion, parallax (in HMD)
position relatively to the background
Free-view path
viewer body position freely chosen on the free-view path
+ free head movement (in HMD)
360°
background
3D objects
free-
view
path
https://www.youtube.com/watch?v=Q-LNA9KlHhw
Point Cloud
Compression
8. Why Point Cloud? (for VR)
No occlusions
all angle of views are acceptable => parallax
free-view point (=6 DoF VR) is natively supported
Fine topology
volumes, hairs, fur can be represented by a point cloud
naturally captured by sensors without heavy processing
usually deduced from depth and/or disparity from multi-view
capture
10. Real and Virtual Engagement in Realistic Immersive Environments
Point Cloud Compression/transmission:
Immersive Communications (2014) in Reverie FP7
Highly realistic representation for immersive communications reveriefp7.eu
Human is reconstructed as a photo realistic 3D Cloud (or mesh) of Points in a 3D space!
Challenges: Low bit rate, real-time encoding, color coding, inter frame coding, scalability
11. Point Clouds for high-end AR/VR
Key Requirements for a 3DoF+/6DoF VR/AR format
Support of stereo imaging with view dependent parallax
360 video ruled out
Universal applicability
Effective handling of occlusions
2D plus depth ruled out
Easy acquisition & Rendering
Candidates: (Super-) Multi-view, Point Cloud, (Mesh)
Comparing MV and PC
Multi-view
+ Easy acquisition and existing compression technology
- Number of views constraints occlusions and viewing angle
Combining with CGI models difficult – requires methods similar to point cloud creation
Point Cloud
+ Most versatile – works for live-action acquisitions as well as CGI. Composition of scenes is easy (just “cat” point clouds)
+ Occlusions only depend on acquisition technology
o Known but comparably complex approaches for acquisition
- No efficient compression technology yet
12. How to capture point clouds?
Multi camera setups
Depth estimation/measurement
Color/feature detection
3D reconstruction
3D Modelling
15. Algorithms for point cloud reconstruction
- From multiple depth images (Reverie project)
- From rigs with stereo cameras (microsoft)
- From multiple images (3D culture cloud)
- Mobile devices (new sony experia for example)
- Systems like 8i or owlii
- Systems for point cloud capture will be key if pcc will be an impotrant medium
For delivery
16. How to render point clouds?
Giving size to points
Splats, rectangles, cubes (=3D pixels)
Trade-off size vs. texture high frequency
Meshing, and using illumation techniques
A demo using PCC contents (and renderer)
8i content
Technicolor based rendering
18. How to compress point clouds: examples of known technologies
How to compress PC geometry ?
octree-based
occupancy data to be entropy coded using prediction
intra local prediction (local plane, etc.)
inter-prediction (motion)
image based
depth coding but unable to handle occlusions
global/local
global envelope for surfacic objects, then geometrical residual
similar to mesh + height
How to compress PC colors ?
octree-based
palette in an octree, prediction and residual coding
wavelets on trees
image-based
projection on planes/surfaces, compression using a 2D video
codec
local projection and tiling/packing in a unique image to avoid
loss of occlusions due to global projections
block-based (mimicking video compression)
3D blocks, push points in a corner, prediction, 3D-DCT,
quantization and entropy coding
inter prediction with 3D motion vectors
19. Comparison with other format compression
Performance comparison with other formats
in bit per pixel/points to be displayed (bpp)
Compression format (lossy, good quality) bpp
2D flat UHD, intra ~0.25-0.5
2D flat UHD, inter ~0.025-0.1
2D + smooth depth 2D flat + 25% (for parallax)
Multi-View 2D*nb views*75%
Point Cloud geo ~1-3, texture ~0.5-2
Mesh geo ~0.25, texture ~0.5
Light-field unknown yet
Challenges ahead
improve PC compression efficiency
in particular for inter coding
find a robust method to assess compressed geometry and
texture quality
Industry wants fast adoption, re-use of existing hardware
infrastructure (e.g. HEVC, AVC) will be important
21. Design of PCC Anchor Point Cloud
Compression
- Design for Reverie Immersive online platform
- Replicant (3D Point Cloud Reconstruction) by CERTH-ITI
- Real-time requirements
- Different quality representations (for rendering at different distances)
- Publication:
R. Mekuria, K. Blom and P. Cesar, "Design, Implementation, and Evaluation of a Point Cloud
Codec for Tele-Immersive Video," in IEEE Transactions on Circuits and Systems for Video Technology,
vol. 27, no. 4, pp. 828-842, April 2017.
doi: 10.1109/TCSVT.2016.2543039
- Experimental work, codec used as anchor for MPEG PCC
22. Reverie Immersive Framwork
Media Router Media Router
Large Scale Tele-Immersive Architecture
User Analysis
Scene structuring &
NavigationScene structuring &
NavigationServer
Composition
Renderers
Content
Decoding
Stream
Synchronization
3D Capturing
3D
Reconstruction
Scalable Content
Coding
Streaming
User Generated
Objects
Network
Monitoring
Session Manager Session Set-up
Social
Network
Media
Coordinator
Avatar
Embodiment
Media Router
Stream Selector
& Media Objects
Audio Streams
Perception
Cognition
Animation
Avatar Reasoning
Media Stream
Demux
Visual Streams
Client
Media Router
Network
Monitoring
Spatial & Audio
Composition
23. Real and Virtual Engagement in Realistic Immersive Environments
Point Cloud Compression: Immersive
Communications
Replicant (immersive communications) reveriefp7.eu
Human is reconstructed as a photo realistic 3D Cloud (or mesh) of Points in a 3D space!
Challenges: Low bit rate, real-time encoding, color coding, inter frame coding, scalability
25. Point Cloud Geometry Compression
recursive sub-divsions: complexity O(2^(N))
Limit level N and differentially code of points in larger leafs for real-time coding [Kammerl12]
Prediction of subdivisions based on the previous level [Schnabel06, Huang06]
Context Adaptive Entropy encoding [Schnabel06, Huang06]
range coding [Kammerl12]
10000000 10000000
[Kammerl12] Kammerl, J.; Blodow, N.; Rusu, R.B.; Gedikli, S.; Beetz, M.; Steinbach, E., "Real-
time compression of point cloud streams," Robotics and Automation (ICRA), 2012 IEEE
International Conference on , vol., no., pp.778,785, 14-18 May 2012
[Schnabel06] Ruwen Schnabel and Reinhard Klein. 2006. Octree-based point-cloud
compression. In Proceedings of the 3rd Eurographics / IEEE VGTC conference on Point-Based
Graphics (SPBG'06),
[Huang06]Huang Yan Huang, Jingliang Peng, C.-C. Jay Kuo, and M. Gopi. 2006. Octree-based
progressive geometry coding of point clouds. In Proceedings of the 3rd Eurographics / IEEE
VGTC conference on Point-Based Graphics (SPBG'06)
Point Cloud
Compression
26. Real and Virtual Engagement in Realistic Immersive Environments
Point Cloud (Color) Attribute Compression
DPCM [Kammerl12]
Colorization [Huang06]
Octree Based [schnabel06]
Based on Graph Transform [Zhang14]
Mapping to jpeg image grid ??
Table 1 scan order for mapping octree centroid colours to image grid (8x10 sample block)
0 1 2 3 4 5 6 7 64 …
15 14 13 12 11 10 9 8 79 …
16 17 18 19 20 21 22 23 80 …
31 30 29 28 27 26 25 24 95 …
32 33 34 35 36 37 38 39 … …
47 46 45 44 43 42 41 40 … …
48 49 50 51 52 53 54 55 … …
63 62 61 60 59 58 57 56 … …
0
0.5
1
1.5
2
2.5
3
dense octree (11) sparse octree (9)
bitrate(bytesperoutput
vertex/voxel)
Achieved Compression Ratio
original (5 bits colors)
proposed (jpeg mapping)
Overall coding gain compared to legacy point cloud codec available in PCL
(at comparable objective quality)
Traverse
octree
Write color
attributes to an
image grid
JPEG
Encode
27. Real and Virtual Engagement in Realistic Immersive Environments
Inter Predictive Coding of Octree
Compressed Point Clouds
Basic Algorithm Overview
1. Input Cloud I and Input Cloud P
2. Align bounding box Cloud I and P
3. Compute octrees of I and P at level N – M
4. Find common leafs at level N-M occupied in I and P
5. Try to predict the vertices in leafs in P from leafs in I
6. If P can be predicted, code vertices as a rigid transform on the input
vertices
7. Code all other vertices in P via an intra coding scheme
Algorithm Details
1. Color variance and point count used to decide wether or not to do the
prediction
2. Compute prediction via ICP procedure (iterative closest points)
3. Compress the rigid transform as a quaternion or 2 vectors
4. Use an octree coding upto level N to code the points that cannot be
predicted
28. Real and Virtual Engagement in Realistic Immersive Environments
Bounding Box Alignment
I input
cloud
compute BB
compute BB
P input
cloud
expand BB I
normalize I on
BB_IE
normalize P on
BB_IE
P Coding of P
I Coding of I
expand BB P
normalize P on
BB_PE
I Coding of P
BB_P fits BB_IE
BB_I
BB_P
BB_IE
BB_IE
N
BB_IE
Y
BB_PE
29. Real and Virtual Engagement in Realistic Immersive Environments
Inter prediction Algorithm
normalized
I Cloud
normalized
P Cloud
Generate Macroblocks of I and P
For each Macroblock in P
corr macroblock in I ?
color_var < TRESH
icp converged ?
encode rigid transform
fittness < thresh ?
Intra Encoding of Macro Block
Intra Encoding of Macro Block
Intra Encoding of Macro Block
store key, rigid tf, color_offset
I coded
part
p coded
part
Compute rigid transform via ICP
between corr Macroblocks I and P
N
Y
Y
N
Y
N
N
Y
# of points ok ?
1.
2.
3.
4.
5.
6.
7.
8. 15 bytes
N
Y
N-M
Octree !!
30. Real and Virtual Engagement in Realistic Immersive Environments
Results: Bounding Box Alignment
Alex at 12 fps, Alex at 24 fps
Bounding box factor Alignment % (12
fps)
Alignment % (24
fps)
5 % 44 65
10 % 68 76
15 % 72 83
20 % 74 83
25 % 82 88
33. Real and Virtual Engagement in Realistic Immersive Environments
Results Inter Prediction
For Alex, Chris and Dimitrios > 65 % of blocks are shared and
~ 30 percent can be predicted
Jog and ski don’t work well because outlier points mess up
the bounding box algorithm
0
1
2
3
4
5
alex chris dim
Bytesperoutputvertex
Dataset (ITI/CERTH, 24 fps)
bit rate gain predictive coding
intra pred
60
65
70
75
80
alex chris dim
Quality[dB]
Dataset (ITI/CERTH 24 fps)
Quality Geometry [dB]
intra pred
0
200
400
600
800
alex chris dim
Encodertime[ms]
Dataset
Encoder Time intra
pred
34. Real and Virtual Engagement in Realistic Immersive Environments
PCC Reference Software (m36527)
http://wg11.sc29.org/svn/repos/MPEG-04/Part16-
Animation_Framework_eXtension_(AFX)/trunk/3Dgraphics/3DG-
PCC/trunk/pcc-mp3dg (official mpeg version)
https://github.com/RufaelDev/pcc-mp3dg (open source)
Tested: Win7, Mac, Ubuntu Linux
Cloud Codec V2 compression_eval Quality
35. Real and Virtual Engagement in Realistic Immersive Environments
Octree Based Point Cloud
Table 2 – MOS on the perceived quality
36. Real and Virtual Engagement in Realistic Immersive Environments
Coding Color Attributes with JPEG
Table 2 – MOS on the perceived quality
37. Real and Virtual Engagement in Realistic Immersive Environments
Temporal Prediction via ICP
Table 2 – MOS on the perceived quality
39. Standardization of PCC Outline
Point Cloud
Compression
- PCC Exploration in MPEG
- Use Cases / Requirements
- Test Datasets
- Quality Assessment
- Common Test Conditions
- Rendering
- Anchor
- Call for Proposals (10 responses, october 2017)
40. MPEG Standardization
Point Cloud
Compression
- Industry consortium for developing standardized specification under
ISO/IEC
- Meets 3-4 times per year
- Joint work with ITU on video coding (AVC, HEVC)
- Much used standards in the media industry
- Still relatively strong attendance (around 300 per meeting)
- PCC AhG group under 3D Graphics groups (MPEG only)
- Next Meeting in Gwanju, Korea, in January
41. Exploration on PCC/3DGC
Point Cloud
Compression
- Started in 2014 in San Diego Meeting
- Naturalistic content
- Virtual Reality use case (not yet popular back then)
- Techniques for point clouds in addition to meshes
- Evaluation methodology (objective, subjective metrics)
- Anchor codec framework
- Work done as PhD student at CWI
- Contributions from Reverie EU project
- Discussion with major industries in 2016 on the requirements and
methodology (e.g. Apple, Nokia, Huawei, Technicolor, 8i, owlii, Orange,
Samsung, Orange, Canon, Sony, Panasonic etc..)
- Call for Proposals in Jan. 2017, (10 responses, october 2017)
- Industry standard specification expected by mid 2019
42. Table 2 – MOS on the perceived quality
Point cloudImages
Point Cloud Compression in
VR Ecosystem (1)
Point Cloud
Compression
43. Use Cases (1)
Point Cloud
Compression
- High Quality Broadcast of content (volumetric video)
- Tele-immersive mixed reality (real-time volumetric video), both AR/VR
like
- Cultural Heritage, 3D photography, CAD scans
- Mobile mapping data
- Large scale geographic information
- https://mpeg.chiariglione.org/standards/exploration/point-cloud-
compression/use-cases-point-cloud-compression
45. Use Cases (3)
Point Cloud
Compression
3D Recon
Struction
Software
Reconstructed 3D
Human
3D Point Cloud
Representation
3D Source
Encoding
IP
Network
Multi Depth
Camera
Capture
Or other 3D
Capture
Real-Time 3D
Rendering
Composition
In Virtual World
Packetization
&
Transmission
3D Source
Decoding
N RGB + Depth
Images
Or other sensor
data
Reception
&
Synchronization
Figure 1: Transmission pipeline for conferencing with 3D geometry
Key requirements for this application are:
Lossy compression with bit-rate control is needed
low complexity and/or support for real-time encoding/
decoding is needed
error resilience to transmission errors is needed
color attributes coding is needed for realistic rendering
material/appearance related attributes coding is needed
to code additional attributes to support the rendering
view dependence: for streaming optimization view
dependence can be used to optimize the transmission process
Typical point clouds in this use case have the following
characteristics:
Between 100,000 and 10,000,000 points to represent
a reconstructed human color attributes with 8-10 bits
per color component Normals and or material properties
to support the rendering using a shader
46. Use Cases (4)
Point Cloud
Compression
Key requirements for this application are:
Lossy compression with bit-rate control is needed
error resilience to transmission errors is needed
color attributes coding is needed for realistic rendering
material/appearance related attributes coding
is needed to code additional attributes to support the rendering
view dependence: the view dependence can be used to
optimize the encoding process
Typical point clouds in this use case have the following characteristics:
Between 100,000 and 10,000,000 points to represent
closeby objects in the scene, Color attributes with 8-10 bits
per color component Global parameters defining the spatial
constraints of the rendering viewport
47. Use Cases (5)
Point Cloud
Compression
Figure: Cultural heritage in the 3D Cloud project
Key requirements for this application are:
Progressive coding to enable increasing quality.
Color attributes coding is needed, preferable 8-12 bits
per component
Generic attributes coding such as for material properties.
Lossless is important to enable the best representation
when possible
Typical point clouds in this use case have the following
characteristics:
1 Million upto billions of points (e.g. [13])
Color attributes of 8-12 bits per color component
Can contain multiple clusters/groups of points
48. Use Cases (6)
Point Cloud
Compression
Key requirements for this application are:
a) High precision is needed to support
localization needs
b) Low complexity and/or support for
real-time encoding/decoding is needed
c) Low delay is needed for real-time
communication of dynamic parts of the map
d) Region selectivity is important to
maintain and access the map data
e) Color attributes coding is needed for
realistic rendering and visualization
f) Additional attributes coding for
reflectance and other scene properties
Typical point clouds in this use case have the
following characteristics:
a) Millions to billions of 3D points with up
to 1cm precision
b) Color attributes with 8-12 bits per
color component
c) Normals and/or reflectance properties
as additional attributes
49. Use Cases (7)
Point Cloud
Compression
How to adress the different use case requirements and Applications ?
Cat 1: static point clouds for cultural heritage and other related 3D
photography applications
Cat 2: for tele-immersive conferencing and broadcasts (volumetric
video)
Cat 3: mobile mapping data acquired via non-stationary lidar scanners
(last use case)
Each category has its own test conditions and test data, it is key to
chose your category of interest when you like to do research on point
cloud compression
50. Data sets cat 1
Point Cloud
Compression
Red and black
Egyptian Mask
House without a
roof
Frog
Facade
Loot
53. Quality Assessment
Point Cloud
Compression
- Symmetric metrics adapted from mesh compression (symmetric rms,
symmetric hausdorff)
- point cloud PSNR by using logarithmic scale, instead of peak signal
value, average nearest neighbour distance is used to determine the
peak value
- Subjective testing with a renderer and a camera path based on 60
seconds of uncompressed renderered videos
54. Objective Metric
Point Cloud
Compression
Decoded coud
Original cloud
Symmetric distances:
the max of D to O and O to D
PSNR using scalar based on peak value (based on the size of the bounding cube)
𝑃𝑆𝑁𝑅 = 10 log10
3𝑝2
𝑀𝑆𝐸
𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 =
1
√𝐾
||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2𝑣 𝑙 ∈𝑉𝑜𝑟 ,
(4)
𝑑 𝑠𝑦𝑚 _𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥( 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑 𝑟𝑚𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 ) (5)
𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥 𝑣 𝑙 ∈𝑉𝑜𝑟 ,
( ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2 ) (6)
𝑑 𝑠𝑦𝑚 _ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎 𝑥 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 (7)
𝑝𝑠𝑛𝑟𝑔𝑒𝑜𝑚 = 10𝑙𝑜𝑔10 ( | 𝑚𝑎𝑥 𝑥,𝑦,𝑧 𝑉𝑑𝑒𝑔 |2
2
/ (𝑑 𝑠𝑦𝑚 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2
) (8)
𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 =
1
√𝐾
||𝑦(𝑣𝑙) − 𝑦(𝑣 𝑛𝑛 _𝑑𝑒𝑔 )||2𝑣 𝑙 ∈𝑉𝑜𝑟 ,
(9)
𝑝𝑠𝑛𝑟𝑦 = 10𝑙𝑜𝑔10 ( |𝑚𝑎𝑥 𝑦 𝑉𝑑𝑒𝑔 |2
2
/ (𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2
) (10)
Tool for computing metric available in MPEG, and as open source with the anchor codec:
https://github.com/RufaelDev/pcc-mp3dg
55. Subjective Testing
Point Cloud
Compression
- 60 seconds video
- (secret) Camera path
- No reference method
- Renderer provided by technicolor, no lighting computations
- Uncompressed video on HD screen
56. Common Test conditions cat 1
Point Cloud
Compression
Class R1 R2 R3 R4
A 0.4 1.25 3.5 6.8
B 0.61 2 4.5 8
C 0.07 0.25 1.0 1.6
Table 1 Target rates in bits per point (geometry + color), for each point cloud class
Four rate points lossy geometry + lossy colors, measured in bits per input point
bit-rates determined using the anchor codec based on octree
57. Common Test conditions cat 2
Point Cloud
Compression
Test Dataset
Queen 3 5* 15* 30 55*
Loot 3.5* 5 8* 16 27*
Redandblack 3.5* 6 9* 18 30*
Soldier 3.5* 6 11* 20 37.1*
Longdress 3.9* 6 13* 27 42.7*
Table 1 Target bitrates for Category 2 in Mbit/s
Five rate points lossy geometry + lossy colors, measured in bits per input point,
Both all intra and random access conditions
58. Common Test conditions cat 3
Point Cloud
Compression
1. Two test sub-conditions are considered for Category 3, with
target rates provided in terms of bits per input point (bpp):
Geometry only: The target rates are 24, 12, 8, 4, 2, 1, 0.5 bpp.
Geometry & Attributes: The target rates are 32, 16, 12, 6, 3, 1.5,
0.75 bpp. These rates are inclusive of geometry and all associated
attribute data. It is mandatory for submissions to this condition for
Category 3 to encode both reflectance and color attributes, if
present in the input point cloud.
2. No subjective testing
This application deals with non visual data and there is no subjective testing
involved
59. Call for Proposals Response
Overview
Point Cloud
Compression
- 9 companies including Unified Streaming
- HEVC based approaches (4)
- Pure point cloud based approaches (e.g. octree based, 4)
- Mesh based approach, convert to mesh and compress the mesh (1)
- 3 in cat 1, 9 in cat 2 and 1 in cat 3
- HEVC/video based approach for cat 2, octree approach for cat 1 and cat 3
- 8 Mbit’s achieved for transparent cat2 coding
- Information is based on respective input contributions and independent
validation done by mines telecom, CWI and GBA Tech
60. Octree Based approaches cat 1/2
Point Cloud
Compression
1. Re-coloring (after up-sampling decoded geometry)
2. Region adaptive hierarchical transform (RaHT) transform
3. Improving the colors by avoiding 4:2:0 subsampling in image
mapping, interpolation
4. Octree Self similarity coding
5. Proposed attribute coding with Laplacian Sparsity Optimized
Graph Transforms
6. Inter prediction using trajectories (octree based coding in
category 2)
63. Geometry coding Point Cloud
Compression
1. Surface approximation at higher levels in the octree based on
triangles that can be used for upsampling
2. Differential coding of vertices based in a gof
3. RAHT transform for attributes (superseded by HEVC based
coding)
64. Octree Self similarity leaf nodes Point Cloud
Compression
Diff = 𝑘=0
63
𝑋𝑂𝑅 𝑙𝑛𝑜𝑑𝑒𝐴 𝑘 , 𝑙𝑛𝑜𝑑𝑒𝐵[𝑘]
If (Diff <= TH)
Merge the two nodes
Else
Handle the two nodes independently
65. Color Palette and HEVC
based techniques in octree
Point Cloud
Compression
- Reduce the number of colors by k-means (vector quantization)
- Code colors by mapping the point attributes to an HEVC grid
66. Laplacian sparsity optimized
attribute coding with GFT
Point Cloud
Compression
1. Facilitate the Graph transform
2. Best weights, best subgraphs and no isolated components in the
graph
3. Kd-tree based partitioning
4. Graph transform (based on graph constrict and eigen value
composition of laplacian)
5. Experiments assessing attribute coding performance checking
influence of parameters sigma and tau (offline training restricting
the ranges (0.1)
67. HEVC Based approach cat 2
Point Cloud
Compression
1. Brute force mapping .ply file to HEVC image (bottom place)
2. Pure projection based (middle)
3. Projection/hybrid octree HEVC based approach (best of rest)
4. Patch based mapping to HEVC (winning proposal)
68. Direct HEVC Mapping approach
Point Cloud
Compression
1. Sort the points
2. Directly compress using HEVC codec
HEVC 10-Bit
Lossless
Encoding
3D-Model Point Cloud
(Geometry only)
Compressed
Bitstream
Sort and store
into 2D video
frame
Encoder
HEVC 10-Bit
Decoding
3D-Model Point Cloud
(Geometry only)
Compressed
Bitstream
Read point
cloud data from
2D video frame
Decoder
69. Direct HEVC approach after sort
Point Cloud
Compression
1. Sort the points, write to image
2. Directly compress using HEVC codec 10 bit profile
HEVC 10-Bit
Lossless
Encoding
3D-Model Point Cloud
(Geometry only)
Compressed
Bitstream
Sort and store
into 2D video
frame
Encoder
HEVC 10-Bit
Decoding
3D-Model Point Cloud
(Geometry only)
Compressed
Bitstream
Read point
cloud data from
2D video frame
Decoder
70. Direct HEVC approach after sort
Point Cloud
Compression
1. Better performance at low bit rates
2. Many artefacts due to occlusions, seems limited
3. Not applicable to category 1
71. Direct HEVC approach after sort
Point Cloud
Compression
HEVC 10-Bit
Encoding
Mux
3D-Model
Point Cloud
Compressed
Stream
Sort and
store into
2D video
frame HEVC 8-Bit
Encoding
Encoder
Binary Map
Row/Column
Subsampling
Geometry
Color
72. Projection HEVC Based
Point Cloud
Compression
2D encoding
3D -> 2D
projection
Texture
plane(s)
Geometry
plane(s)
2D decoding
Texture
plane(s)
Geometry
plane(s)
2D -> 3D
projection
Input PLY
sequence
Output PLY
sequence
Bitstream
Projection metadata
Projection metadata
1. Projection to images
2. Meta data
82. HEVC projection + Octree (4) Point Cloud
Compression
Best results in subjective and objective evaluations
test model for cat2 with significant industry support
83. Mesh based approach cat 2
Point Cloud
Compression
Good results for simple applications
84. MPEG Decision
Point Cloud
Compression
- Test models based on octree for cat1 and cat3 content
- Test model based on HEVC + Patch for cat 2 (dynamic)
- Mesh based could move to a different activity (TBD)
85. Research directions
Point Cloud
Compression
- HEVC mapping related issues: occlusion points, inter prediction
Should be improved, difficult datasets without fixed normals enabling patches
- Attribute and large scale point cloud data (tree coding,
Graph transforms), out of core processing
- Mesh based geometry coding
- At some point the transport and storage formats will need attention
- Requirements of specific applications and use cases, especially
cat 3. mobile mapping should be further improved.
86. Industry directions
Point Cloud
Compression
- Rapid need for PCC in AR/VR frameworks of mobile devices
- Adoption application standards (khronos, gltf, but also ATSC, DVB)
- Mapping geospatial market is interesting for large players, but slightly
out of traditional MPEG domain
- Software and implementation aspects will become key
Notes de l'éditeur
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.
For comparison purpose we set the SC3DMC coders to code with 8 bits and differential coding options. We compared the orignal captured meshes to the meshes decoded using a tool to measure the symmetric distance between the surfaces. This metric is often known as haussdorf distance. A set of live captured models compressed and decoded shows that the qualities are comparible. Note that lower value implies better quality.