Tutorial on Point Cloud Compression and standardisation given at IEEE VCIP 2017 in december. I provide the techniques for point cloud compression and the designed quality metrics and codecs in my PhD at CWI. I detail the standardisation activity on point cloud compression that I started in 2014 and that started in 2017 involving all mobile device makers like Apple, Huawei, Sony, Samsung and Nokia.

1. Point Cloud Compression Tutorial
Rufael Mekuria (PhD), Unified Streaming
and point cloud compression
IEEE Visual Communications and Image Processing 2017
St. Petersburg Florida
Invited Tutorial
rufael@unified-streaming.com
Point Cloud
Compression

2. About Unified Streaming
Unified Streaming
Creator of video streaming software behind many of the large scale early deployments
of HTTP Streaming (BBC iplayer etc.., NPO) www.unified-streaming.com key
implementer of technologies like DASH, HLS, CMAF et… used by CDN, Telco, cloud,
broadcaster etc.
MPEG affiliate, DASH-IF Member
DASH, CMAF, Network based media processing
5G Video streaming pioneer
Involved in EU H2020 project Superfluidity with companies like Intel, RedHat and
Nokia. Helped many telco/CDN on advanced streaming architectures
Point Cloud
Compression

3. About Me
Academic Background
MSEE, Delft 2011, PhD CWI/Vrije Universiteit Amsterdam jan’17
25+ peered reviewed papers, best paper in ACM MMSys’13, 5+ invited papers in
conferences like SPIE, ICASSP, etc…. Invited talks: UIUC, Huawei, Google etc…
Professional Background
TNO (2010-2011), CWI 2011-2016, Unified Streaming 2016-date, PCC (2017-)
Active in large scale EU Funded projects: Superfluidity H2020, Reverie FP7, developed one
of the first large scale tele-immersive system combining real and CGI content
Standardization Work
Chaired and pioneered PCC in MPEG (since jan.2014) CfP was responded by 9 companies
in oct. 2017 including all major mobile device vendors (CfP response referred to as
“Historic” by MPEG convenor).
Point Cloud
Compression

4. About this Tutorial
Before the break:
Introduction to this (re-) emerging research area, point clouds for VR, point
cloud compression architecture
After the break:
Standardization activity on point cloud compression in MPEG
Applications and requirements, quality assessments
State of art and research challenges
Copyrights & Acknowledgement:
Some of the images and parts of this tutorial fall under copyright of
respective contributors/authors as noted to MPEG CfP. I do not name the
contributors explicitly for now. Many opinions in this work are derived from
standardization meetings representing rough industry consensus. I
acknowledge 8i, Microsoft, Technicolor, Ford, Mitshubishi, et al. for
contribution of test data. I thank Technicolor for large contributions to this
body of work.
Point Cloud
Compression

5.  A collection of points
 Not related to each other
 Typically no order
 Typically no local topology (no mesh!)
 Each is point is the given of
 a position (X,Y,Z)
 a color (R,G,B) or (Y,U,V)
 possibly other things like transparency, time of
acquisition, etc.
Point Cloud
Compression
Point cloud content from Microsoft
research laboratory Donated to MPEG
Point Cloud Format

6. .ply files =
an example raw data format for point cloud
 This is an example raw point coud file format
(compare to YUV for video coding)
 How many points?
 Static case up to several tens of millions,
depending on the application
 Dynamic case ~1 million per frame, 30 fps
 Probably more is needed for good VR
 Format
 Geometry XYZ
 Fixed precision for VR applications
 Float still often used
 Colors RGB
 As usual, integer 8/10 bits.
 Possibly other attributes
 (not present in the ply file here on the left)
plyformat ascii 1.0
element vertex 764940
property float x
property float y
property float z
property uchar red
property uchar green
property uchar blue
end_header
211 63 63 127 98 73
213 63 61 134 109 87
212 62 63 122 97 75
212 63 62 129 102 79
212 63 63 124 98 76
213 62 63 122 98 76
213 63 62 128 104 81
213 63 63 124 99 78
215 61 63 120 97 76
214 63 60 141 117 95
214 63 61 135 111 89
215 63 60 144 120 97
215 63 61 133 109 87
214 62 62 126 102 80
214 62 63 122 98 77
214 63 62 128 104 82
etc.
one point X Y Z R G B
no order! Swapping points
does not change the data
Point Cloud
Compression

7. An application of point cloud: free-view point (6DoF) for sport
 Scene model
 360°/omnidirectional background
 reshaping depending on viewpoint
 3D object
 occlusion, parallax (in HMD)
 position relatively to the background
 Free-view path
 viewer body position freely chosen on the free-view path
 + free head movement (in HMD)
360°
background
3D objects
free-
view
path
https://www.youtube.com/watch?v=Q-LNA9KlHhw
Point Cloud
Compression

8. Why Point Cloud? (for VR)
 No occlusions
 all angle of views are acceptable => parallax
 free-view point (=6 DoF VR) is natively supported
 Fine topology
 volumes, hairs, fur can be represented by a point cloud
 naturally captured by sensors without heavy processing
 usually deduced from depth and/or disparity from multi-view
capture

9. Point Cloud Representation in
Microsoft holoportation

10. Real and Virtual Engagement in Realistic Immersive Environments
Point Cloud Compression/transmission:
Immersive Communications (2014) in Reverie FP7
Highly realistic representation for immersive communications reveriefp7.eu
Human is reconstructed as a photo realistic 3D Cloud (or mesh) of Points in a 3D space!
Challenges: Low bit rate, real-time encoding, color coding, inter frame coding, scalability

11. Point Clouds for high-end AR/VR
 Key Requirements for a 3DoF+/6DoF VR/AR format
 Support of stereo imaging with view dependent parallax
 360 video ruled out
 Universal applicability
 Effective handling of occlusions
 2D plus depth ruled out
 Easy acquisition & Rendering
 Candidates: (Super-) Multi-view, Point Cloud, (Mesh)
 Comparing MV and PC
 Multi-view
+ Easy acquisition and existing compression technology
- Number of views constraints occlusions and viewing angle
 Combining with CGI models difficult – requires methods similar to point cloud creation
 Point Cloud
+ Most versatile – works for live-action acquisitions as well as CGI. Composition of scenes is easy (just “cat” point clouds)
+ Occlusions only depend on acquisition technology
o Known but comparably complex approaches for acquisition
- No efficient compression technology yet

12. How to capture point clouds?
 Multi camera setups
 Depth estimation/measurement
 Color/feature detection
 3D reconstruction
 3D Modelling

13. High Quality (studio level?) content

14. Low Quality Setup (3D skype ?)

15. Algorithms for point cloud reconstruction
- From multiple depth images (Reverie project)
- From rigs with stereo cameras (microsoft)
- From multiple images (3D culture cloud)
- Mobile devices (new sony experia for example)
- Systems like 8i or owlii
- Systems for point cloud capture will be key if pcc will be an impotrant medium
For delivery

16. How to render point clouds?
 Giving size to points
 Splats, rectangles, cubes (=3D pixels)
 Trade-off size vs. texture high frequency
 Meshing, and using illumation techniques
 A demo using PCC contents (and renderer)
 8i content
 Technicolor based rendering

17. Rendering Demonstration MPEG Data and rendering, data by 8i

18. How to compress point clouds: examples of known technologies
 How to compress PC geometry ?
 octree-based
 occupancy data to be entropy coded using prediction
 intra local prediction (local plane, etc.)
 inter-prediction (motion)
 image based
 depth coding but unable to handle occlusions
 global/local
 global envelope for surfacic objects, then geometrical residual
 similar to mesh + height
 How to compress PC colors ?
 octree-based
 palette in an octree, prediction and residual coding
 wavelets on trees
 image-based
 projection on planes/surfaces, compression using a 2D video
codec
 local projection and tiling/packing in a unique image to avoid
loss of occlusions due to global projections
 block-based (mimicking video compression)
 3D blocks, push points in a corner, prediction, 3D-DCT,
quantization and entropy coding
 inter prediction with 3D motion vectors

19. Comparison with other format compression
 Performance comparison with other formats
 in bit per pixel/points to be displayed (bpp)
Compression format (lossy, good quality) bpp
2D flat UHD, intra ~0.25-0.5
2D flat UHD, inter ~0.025-0.1
2D + smooth depth 2D flat + 25% (for parallax)
Multi-View 2D*nb views*75%
Point Cloud geo ~1-3, texture ~0.5-2
Mesh geo ~0.25, texture ~0.5
Light-field unknown yet
 Challenges ahead
 improve PC compression efficiency
 in particular for inter coding
 find a robust method to assess compressed geometry and
texture quality
 Industry wants fast adoption, re-use of existing hardware
infrastructure (e.g. HEVC, AVC) will be important

20. Questions (round 1) ??
Point Cloud
Compression

21. Design of PCC Anchor Point Cloud
Compression
- Design for Reverie Immersive online platform
- Replicant (3D Point Cloud Reconstruction) by CERTH-ITI
- Real-time requirements
- Different quality representations (for rendering at different distances)
- Publication:
R. Mekuria, K. Blom and P. Cesar, "Design, Implementation, and Evaluation of a Point Cloud
Codec for Tele-Immersive Video," in IEEE Transactions on Circuits and Systems for Video Technology,
vol. 27, no. 4, pp. 828-842, April 2017.
doi: 10.1109/TCSVT.2016.2543039
- Experimental work, codec used as anchor for MPEG PCC

22. Reverie Immersive Framwork
Media Router Media Router
Large Scale Tele-Immersive Architecture
User Analysis
Scene structuring &
NavigationScene structuring &
NavigationServer
Composition
Renderers
Content
Decoding
Stream
Synchronization
3D Capturing
3D
Reconstruction
Scalable Content
Coding
Streaming
User Generated
Objects
Network
Monitoring
Session Manager Session Set-up
Social
Network
Media
Coordinator
Avatar
Embodiment
Media Router
Stream Selector
& Media Objects
Audio Streams
Perception
Cognition
Animation
Avatar Reasoning
Media Stream
Demux
Visual Streams
Client
Media Router
Network
Monitoring
Spatial & Audio
Composition

23. Real and Virtual Engagement in Realistic Immersive Environments
Point Cloud Compression: Immersive
Communications
Replicant (immersive communications) reveriefp7.eu
Human is reconstructed as a photo realistic 3D Cloud (or mesh) of Points in a 3D space!
Challenges: Low bit rate, real-time encoding, color coding, inter frame coding, scalability

24. Architecture Hybrid Octree
point cloud codec
Point Cloud
Compression

25. Point Cloud Geometry Compression
recursive sub-divsions: complexity O(2^(N))
Limit level N and differentially code of points in larger leafs for real-time coding [Kammerl12]
Prediction of subdivisions based on the previous level [Schnabel06, Huang06]
Context Adaptive Entropy encoding [Schnabel06, Huang06]
range coding [Kammerl12]
10000000 10000000
[Kammerl12] Kammerl, J.; Blodow, N.; Rusu, R.B.; Gedikli, S.; Beetz, M.; Steinbach, E., "Real-
time compression of point cloud streams," Robotics and Automation (ICRA), 2012 IEEE
International Conference on , vol., no., pp.778,785, 14-18 May 2012
[Schnabel06] Ruwen Schnabel and Reinhard Klein. 2006. Octree-based point-cloud
compression. In Proceedings of the 3rd Eurographics / IEEE VGTC conference on Point-Based
Graphics (SPBG'06),
[Huang06]Huang Yan Huang, Jingliang Peng, C.-C. Jay Kuo, and M. Gopi. 2006. Octree-based
progressive geometry coding of point clouds. In Proceedings of the 3rd Eurographics / IEEE
VGTC conference on Point-Based Graphics (SPBG'06)
Point Cloud
Compression

26. Real and Virtual Engagement in Realistic Immersive Environments
Point Cloud (Color) Attribute Compression
DPCM [Kammerl12]
Colorization [Huang06]
Octree Based [schnabel06]
Based on Graph Transform [Zhang14]
Mapping to jpeg image grid ??
Table 1 scan order for mapping octree centroid colours to image grid (8x10 sample block)
0 1 2 3 4 5 6 7 64 …
15 14 13 12 11 10 9 8 79 …
16 17 18 19 20 21 22 23 80 …
31 30 29 28 27 26 25 24 95 …
32 33 34 35 36 37 38 39 … …
47 46 45 44 43 42 41 40 … …
48 49 50 51 52 53 54 55 … …
63 62 61 60 59 58 57 56 … …
0
0.5
1
1.5
2
2.5
3
dense octree (11) sparse octree (9)
bitrate(bytesperoutput
vertex/voxel)
Achieved Compression Ratio
original (5 bits colors)
proposed (jpeg mapping)
Overall coding gain compared to legacy point cloud codec available in PCL
(at comparable objective quality)
Traverse
octree
Write color
attributes to an
image grid
JPEG
Encode

27. Real and Virtual Engagement in Realistic Immersive Environments
Inter Predictive Coding of Octree
Compressed Point Clouds
Basic Algorithm Overview
1. Input Cloud I and Input Cloud P
2. Align bounding box Cloud I and P
3. Compute octrees of I and P at level N – M
4. Find common leafs at level N-M occupied in I and P
5. Try to predict the vertices in leafs in P from leafs in I
6. If P can be predicted, code vertices as a rigid transform on the input
vertices
7. Code all other vertices in P via an intra coding scheme
Algorithm Details
1. Color variance and point count used to decide wether or not to do the
prediction
2. Compute prediction via ICP procedure (iterative closest points)
3. Compress the rigid transform as a quaternion or 2 vectors
4. Use an octree coding upto level N to code the points that cannot be
predicted

28. Real and Virtual Engagement in Realistic Immersive Environments
Bounding Box Alignment
I input
cloud
compute BB
compute BB
P input
cloud
expand BB I
normalize I on
BB_IE
normalize P on
BB_IE
P Coding of P
I Coding of I
expand BB P
normalize P on
BB_PE
I Coding of P
BB_P fits BB_IE
BB_I
BB_P
BB_IE
BB_IE
N
BB_IE
Y
BB_PE

29. Real and Virtual Engagement in Realistic Immersive Environments
Inter prediction Algorithm
normalized
I Cloud
normalized
P Cloud
Generate Macroblocks of I and P
For each Macroblock in P
corr macroblock in I ?
color_var < TRESH
icp converged ?
encode rigid transform
fittness < thresh ?
Intra Encoding of Macro Block
Intra Encoding of Macro Block
Intra Encoding of Macro Block
store key, rigid tf, color_offset
I coded
part
p coded
part
Compute rigid transform via ICP
between corr Macroblocks I and P
N
Y
Y
N
Y
N
N
Y
# of points ok ?
1.
2.
3.
4.
5.
6.
7.
8. 15 bytes
N
Y
N-M
Octree !!

30. Real and Virtual Engagement in Realistic Immersive Environments
Results: Bounding Box Alignment
Alex at 12 fps, Alex at 24 fps
Bounding box factor Alignment % (12
fps)
Alignment % (24
fps)
5 % 44 65
10 % 68 76
15 % 72 83
20 % 74 83
25 % 82 88

31. Real and Virtual Engagement in Realistic Immersive Environments
Shared Macroblocks, convergence
percentage
Alex at 24 fps (100 frames) Alex at 12 fps (50 frames), bounding box factor 20 % 11 level octree
Dataset % shared
macroblocks
% ICP
converged
% points p coded
(apr.)
Alex (12 fps) 65 % 32 % ~ 25 %
Alex (24 fps) 75 % 35 % ~ 30 %
Christos (12 fps) 66 % 34 % ~ 30 %
Christos (24 fps) 78 % 40 % ~ 35 %
Dimitrios (12 fps) 69,9% 39 % ~ 35 %
Dimitrios (24 fps) 78 % 42 % ~ 40 %

32. Real and Virtual Engagement in Realistic Immersive Environments
Point Cloud Compression, Quality
Assessment (m36529)
Table 2 – MOS on the perceived quality
d_sym_rms Symmetric rms distance betw. Clouds (5)
d_sym_haussdorf Symmetric Haussdorf dist. betw. clouds (7)
psnr_geom PSNR(vertexpositions) (8)
psnr_y PSNR(colorsY) (10)
psnr_u PSNR (colors U) (as 10 for u)
psnr_v PSNR (colors V) (as 10 for v)
𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 =
1
√𝐾
||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2𝑣 𝑙 ∈𝑉𝑜𝑟 ,
(4)
𝑑 𝑠𝑦𝑚 _𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥⁡( 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑 𝑟𝑚𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 ) (5)
𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥 𝑣 𝑙 ∈𝑉𝑜𝑟 ,
( ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2 ) (6)
𝑑 𝑠𝑦𝑚 _ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎 𝑥 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 (7)
𝑝𝑠𝑛𝑟𝑔𝑒𝑜𝑚 = 10𝑙𝑜𝑔10 ( | 𝑚𝑎𝑥 𝑥,𝑦,𝑧 𝑉𝑑𝑒𝑔 |2
2
/ (𝑑 𝑠𝑦𝑚 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2
) (8)
𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 =
1
√𝐾
||𝑦(𝑣𝑙) − 𝑦(𝑣 𝑛𝑛 _𝑑𝑒𝑔 )||2𝑣 𝑙 ∈𝑉𝑜𝑟 ,
(9)
𝑝𝑠𝑛𝑟𝑦 = 10𝑙𝑜𝑔10 ( |𝑚𝑎𝑥 𝑦 𝑉𝑑𝑒𝑔 |2
2
/ (𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2
) (10)
We compare the output point to the nearest input point and vice versa!

33. Real and Virtual Engagement in Realistic Immersive Environments
Results Inter Prediction
For Alex, Chris and Dimitrios > 65 % of blocks are shared and
~ 30 percent can be predicted
Jog and ski don’t work well because outlier points mess up
the bounding box algorithm
0
1
2
3
4
5
alex chris dim
Bytesperoutputvertex
Dataset (ITI/CERTH, 24 fps)
bit rate gain predictive coding
intra pred
60
65
70
75
80
alex chris dim
Quality[dB]
Dataset (ITI/CERTH 24 fps)
Quality Geometry [dB]
intra pred
0
200
400
600
800
alex chris dim
Encodertime[ms]
Dataset
Encoder Time intra
pred

34. Real and Virtual Engagement in Realistic Immersive Environments
PCC Reference Software (m36527)
http://wg11.sc29.org/svn/repos/MPEG-04/Part16-
Animation_Framework_eXtension_(AFX)/trunk/3Dgraphics/3DG-
PCC/trunk/pcc-mp3dg (official mpeg version)
https://github.com/RufaelDev/pcc-mp3dg (open source)
Tested: Win7, Mac, Ubuntu Linux
Cloud Codec V2 compression_eval Quality

35. Real and Virtual Engagement in Realistic Immersive Environments
Octree Based Point Cloud
Table 2 – MOS on the perceived quality

36. Real and Virtual Engagement in Realistic Immersive Environments
Coding Color Attributes with JPEG
Table 2 – MOS on the perceived quality

37. Real and Virtual Engagement in Realistic Immersive Environments
Temporal Prediction via ICP
Table 2 – MOS on the perceived quality

38. Questions (round 2) ??
Point Cloud
Compression

39. Standardization of PCC Outline
Point Cloud
Compression
- PCC Exploration in MPEG
- Use Cases / Requirements
- Test Datasets
- Quality Assessment
- Common Test Conditions
- Rendering
- Anchor
- Call for Proposals (10 responses, october 2017)

40. MPEG Standardization
Point Cloud
Compression
- Industry consortium for developing standardized specification under
ISO/IEC
- Meets 3-4 times per year
- Joint work with ITU on video coding (AVC, HEVC)
- Much used standards in the media industry
- Still relatively strong attendance (around 300 per meeting)
- PCC AhG group under 3D Graphics groups (MPEG only)
- Next Meeting in Gwanju, Korea, in January

41. Exploration on PCC/3DGC
Point Cloud
Compression
- Started in 2014 in San Diego Meeting
- Naturalistic content
- Virtual Reality use case (not yet popular back then)
- Techniques for point clouds in addition to meshes
- Evaluation methodology (objective, subjective metrics)
- Anchor codec framework
- Work done as PhD student at CWI
- Contributions from Reverie EU project
- Discussion with major industries in 2016 on the requirements and
methodology (e.g. Apple, Nokia, Huawei, Technicolor, 8i, owlii, Orange,
Samsung, Orange, Canon, Sony, Panasonic etc..)
- Call for Proposals in Jan. 2017, (10 responses, october 2017)
- Industry standard specification expected by mid 2019

42. Table 2 – MOS on the perceived quality
Point cloudImages
Point Cloud Compression in
VR Ecosystem (1)
Point Cloud
Compression

43. Use Cases (1)
Point Cloud
Compression
- High Quality Broadcast of content (volumetric video)
- Tele-immersive mixed reality (real-time volumetric video), both AR/VR
like
- Cultural Heritage, 3D photography, CAD scans
- Mobile mapping data
- Large scale geographic information
- https://mpeg.chiariglione.org/standards/exploration/point-cloud-
compression/use-cases-point-cloud-compression

44. Use cases (2)
Gis: Lidar: http://www.cadalyst.com/cadalyst/gis-tech-news-99-13394
Robotics: lidar, kinect: www.pointclouds.org
GIS: Aereal Imaging
http://www.flightlinegeographics.com/3D_Point_Cloud.html
Cultural heritage (images): Culture3D Cloud Mixed Reality

45. Use Cases (3)
Point Cloud
Compression
3D Recon
Struction
Software
Reconstructed 3D
Human
3D Point Cloud
Representation
3D Source
Encoding
IP
Network
Multi Depth
Camera
Capture
Or other 3D
Capture
Real-Time 3D
Rendering
Composition
In Virtual World
Packetization
&
Transmission
3D Source
Decoding
N RGB + Depth
Images
Or other sensor
data
Reception
&
Synchronization
Figure 1: Transmission pipeline for conferencing with 3D geometry
Key requirements for this application are:
Lossy compression with bit-rate control is needed
low complexity and/or support for real-time encoding/
decoding is needed
error resilience to transmission errors is needed
color attributes coding is needed for realistic rendering
material/appearance related attributes coding is needed
to code additional attributes to support the rendering
view dependence: for streaming optimization view
dependence can be used to optimize the transmission process
Typical point clouds in this use case have the following
characteristics:
Between 100,000 and 10,000,000 points to represent
a reconstructed human color attributes with 8-10 bits
per color component Normals and or material properties
to support the rendering using a shader

46. Use Cases (4)
Point Cloud
Compression
Key requirements for this application are:
Lossy compression with bit-rate control is needed
error resilience to transmission errors is needed
color attributes coding is needed for realistic rendering
material/appearance related attributes coding
is needed to code additional attributes to support the rendering
view dependence: the view dependence can be used to
optimize the encoding process
Typical point clouds in this use case have the following characteristics:
Between 100,000 and 10,000,000 points to represent
closeby objects in the scene, Color attributes with 8-10 bits
per color component Global parameters defining the spatial
constraints of the rendering viewport

47. Use Cases (5)
Point Cloud
Compression
Figure: Cultural heritage in the 3D Cloud project
Key requirements for this application are:
Progressive coding to enable increasing quality.
Color attributes coding is needed, preferable 8-12 bits
per component
Generic attributes coding such as for material properties.
Lossless is important to enable the best representation
when possible
Typical point clouds in this use case have the following
characteristics:
1 Million upto billions of points (e.g. [13])
Color attributes of 8-12 bits per color component
Can contain multiple clusters/groups of points

48. Use Cases (6)
Point Cloud
Compression
Key requirements for this application are:
a) High precision is needed to support
localization needs
b) Low complexity and/or support for
real-time encoding/decoding is needed
c) Low delay is needed for real-time
communication of dynamic parts of the map
d) Region selectivity is important to
maintain and access the map data
e) Color attributes coding is needed for
realistic rendering and visualization
f) Additional attributes coding for
reflectance and other scene properties
Typical point clouds in this use case have the
following characteristics:
a) Millions to billions of 3D points with up
to 1cm precision
b) Color attributes with 8-12 bits per
color component
c) Normals and/or reflectance properties
as additional attributes

49. Use Cases (7)
Point Cloud
Compression
How to adress the different use case requirements and Applications ?
Cat 1: static point clouds for cultural heritage and other related 3D
photography applications
Cat 2: for tele-immersive conferencing and broadcasts (volumetric
video)
Cat 3: mobile mapping data acquired via non-stationary lidar scanners
(last use case)
Each category has its own test conditions and test data, it is key to
chose your category of interest when you like to do research on point
cloud compression

50. Data sets cat 1
Point Cloud
Compression
Red and black
Egyptian Mask
House without a
roof
Frog
Facade
Loot

51. Datasets cat 2
Point Cloud
Compression
Red and black
loot
Long
dress

52. Data sets cat 3
Point Cloud
Compression

53. Quality Assessment
Point Cloud
Compression
- Symmetric metrics adapted from mesh compression (symmetric rms,
symmetric hausdorff)
- point cloud PSNR by using logarithmic scale, instead of peak signal
value, average nearest neighbour distance is used to determine the
peak value
- Subjective testing with a renderer and a camera path based on 60
seconds of uncompressed renderered videos

54. Objective Metric
Point Cloud
Compression
Decoded coud
Original cloud
Symmetric distances:
the max of D to O and O to D
PSNR using scalar based on peak value (based on the size of the bounding cube)
𝑃𝑆𝑁𝑅 = 10 log10
3𝑝2
𝑀𝑆𝐸
𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 =
1
√𝐾
||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2𝑣 𝑙 ∈𝑉𝑜𝑟 ,
(4)
𝑑 𝑠𝑦𝑚 _𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥⁡( 𝑑 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑 𝑟𝑚𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 ) (5)
𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎𝑥 𝑣 𝑙 ∈𝑉𝑜𝑟 ,
( ||𝑣𝑙 − 𝑣 𝑛𝑛 _𝑑𝑒𝑔 ||2 ) (6)
𝑑 𝑠𝑦𝑚 _ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 = 𝑚𝑎 𝑥 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 , 𝑑ℎ 𝑎𝑢𝑠𝑠 𝑉𝑑𝑒𝑔 , 𝑉𝑜𝑟 (7)
𝑝𝑠𝑛𝑟𝑔𝑒𝑜𝑚 = 10𝑙𝑜𝑔10 ( | 𝑚𝑎𝑥 𝑥,𝑦,𝑧 𝑉𝑑𝑒𝑔 |2
2
/ (𝑑 𝑠𝑦𝑚 𝑟𝑚𝑠 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2
) (8)
𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 =
1
√𝐾
||𝑦(𝑣𝑙) − 𝑦(𝑣 𝑛𝑛 _𝑑𝑒𝑔 )||2𝑣 𝑙 ∈𝑉𝑜𝑟 ,
(9)
𝑝𝑠𝑛𝑟𝑦 = 10𝑙𝑜𝑔10 ( |𝑚𝑎𝑥 𝑦 𝑉𝑑𝑒𝑔 |2
2
/ (𝑑 𝑦 𝑉𝑜𝑟 , 𝑉𝑑𝑒𝑔 )2
) (10)
Tool for computing metric available in MPEG, and as open source with the anchor codec:
https://github.com/RufaelDev/pcc-mp3dg

55. Subjective Testing
Point Cloud
Compression
- 60 seconds video
- (secret) Camera path
- No reference method
- Renderer provided by technicolor, no lighting computations
- Uncompressed video on HD screen

56. Common Test conditions cat 1
Point Cloud
Compression
Class R1 R2 R3 R4
A 0.4 1.25 3.5 6.8
B 0.61 2 4.5 8
C 0.07 0.25 1.0 1.6
Table 1 Target rates in bits per point (geometry + color), for each point cloud class
Four rate points lossy geometry + lossy colors, measured in bits per input point
bit-rates determined using the anchor codec based on octree

57. Common Test conditions cat 2
Point Cloud
Compression
Test Dataset
Queen 3 5* 15* 30 55*
Loot 3.5* 5 8* 16 27*
Redandblack 3.5* 6 9* 18 30*
Soldier 3.5* 6 11* 20 37.1*
Longdress 3.9* 6 13* 27 42.7*
Table 1 Target bitrates for Category 2 in Mbit/s
Five rate points lossy geometry + lossy colors, measured in bits per input point,
Both all intra and random access conditions

58. Common Test conditions cat 3
Point Cloud
Compression
1. Two test sub-conditions are considered for Category 3, with
target rates provided in terms of bits per input point (bpp):
Geometry only: The target rates are 24, 12, 8, 4, 2, 1, 0.5 bpp.
Geometry & Attributes: The target rates are 32, 16, 12, 6, 3, 1.5,
0.75 bpp. These rates are inclusive of geometry and all associated
attribute data. It is mandatory for submissions to this condition for
Category 3 to encode both reflectance and color attributes, if
present in the input point cloud.
2. No subjective testing
This application deals with non visual data and there is no subjective testing
involved

59. Call for Proposals Response
Overview
Point Cloud
Compression
- 9 companies including Unified Streaming
- HEVC based approaches (4)
- Pure point cloud based approaches (e.g. octree based, 4)
- Mesh based approach, convert to mesh and compress the mesh (1)
- 3 in cat 1, 9 in cat 2 and 1 in cat 3
- HEVC/video based approach for cat 2, octree approach for cat 1 and cat 3
- 8 Mbit’s achieved for transparent cat2 coding
- Information is based on respective input contributions and independent
validation done by mines telecom, CWI and GBA Tech

60. Octree Based approaches cat 1/2
Point Cloud
Compression
1. Re-coloring (after up-sampling decoded geometry)
2. Region adaptive hierarchical transform (RaHT) transform
3. Improving the colors by avoiding 4:2:0 subsampling in image
mapping, interpolation
4. Octree Self similarity coding
5. Proposed attribute coding with Laplacian Sparsity Optimized
Graph Transforms
6. Inter prediction using trajectories (octree based coding in
category 2)

61. Re-coloring and upsampling
Point Cloud
Compression

62. Color Coding by mapping to HEVC Point Cloud
Compression

63. Geometry coding Point Cloud
Compression
1. Surface approximation at higher levels in the octree based on
triangles that can be used for upsampling
2. Differential coding of vertices based in a gof
3. RAHT transform for attributes (superseded by HEVC based
coding)

64. Octree Self similarity leaf nodes Point Cloud
Compression
Diff = 𝑘=0
63
𝑋𝑂𝑅 𝑙𝑛𝑜𝑑𝑒𝐴 𝑘 , 𝑙𝑛𝑜𝑑𝑒𝐵[𝑘]
If (Diff <= TH)
Merge the two nodes
Else
Handle the two nodes independently

65. Color Palette and HEVC
based techniques in octree
Point Cloud
Compression
- Reduce the number of colors by k-means (vector quantization)
- Code colors by mapping the point attributes to an HEVC grid

66. Laplacian sparsity optimized
attribute coding with GFT
Point Cloud
Compression
1. Facilitate the Graph transform
2. Best weights, best subgraphs and no isolated components in the
graph
3. Kd-tree based partitioning
4. Graph transform (based on graph constrict and eigen value
composition of laplacian)
5. Experiments assessing attribute coding performance checking
influence of parameters sigma and tau (offline training restricting
the ranges (0.1)

67. HEVC Based approach cat 2
Point Cloud
Compression
1. Brute force mapping .ply file to HEVC image (bottom place)
2. Pure projection based (middle)
3. Projection/hybrid octree HEVC based approach (best of rest)
4. Patch based mapping to HEVC (winning proposal)

68. Direct HEVC Mapping approach
Point Cloud
Compression
1. Sort the points
2. Directly compress using HEVC codec
HEVC 10-Bit
Lossless
Encoding
3D-Model Point Cloud
(Geometry only)
Compressed
Bitstream
Sort and store
into 2D video
frame
Encoder
HEVC 10-Bit
Decoding
3D-Model Point Cloud
(Geometry only)
Compressed
Bitstream
Read point
cloud data from
2D video frame
Decoder

69. Direct HEVC approach after sort
Point Cloud
Compression
1. Sort the points, write to image
2. Directly compress using HEVC codec 10 bit profile
HEVC 10-Bit
Lossless
Encoding
3D-Model Point Cloud
(Geometry only)
Compressed
Bitstream
Sort and store
into 2D video
frame
Encoder
HEVC 10-Bit
Decoding
3D-Model Point Cloud
(Geometry only)
Compressed
Bitstream
Read point
cloud data from
2D video frame
Decoder

70. Direct HEVC approach after sort
Point Cloud
Compression
1. Better performance at low bit rates
2. Many artefacts due to occlusions, seems limited
3. Not applicable to category 1

71. Direct HEVC approach after sort
Point Cloud
Compression
HEVC 10-Bit
Encoding
Mux
3D-Model
Point Cloud
Compressed
Stream
Sort and
store into
2D video
frame HEVC 8-Bit
Encoding
Encoder
Binary Map
Row/Column
Subsampling
Geometry
Color

72. Projection HEVC Based
Point Cloud
Compression
2D encoding
3D -> 2D
projection
Texture
plane(s)
Geometry
plane(s)
2D decoding
Texture
plane(s)
Geometry
plane(s)
2D -> 3D
projection
Input PLY
sequence
Output PLY
sequence
Bitstream
Projection metadata
Projection metadata
1. Projection to images
2. Meta data

73. Projection approach + HEVC
Point Cloud
Compression
Problems:
1. Occlusions
2. Invalid points

74. HEVC projection + Octree (1)
Point Cloud
Compression

75. HEVC projection + Octree (2)
Point Cloud
Compression
Enable fine grained projections by using octree and binary flags

76. LCU: Code points
As octree instead of
projection
HEVC projection + Octree (3) Point Cloud
Compression

77. HEVC projection + Octree (4) Point Cloud
Compression

78. HEVC projection + Octree (5) Point Cloud
Compression

79. HEVC projection + Patches Point Cloud
Compression
Decomposition
into patches
Packing
Geometry image
Generation
Texture image
Generation
Attribute image
generation
Video
Compression
Occupancy map
compression
multiplexer
Compressed
bitstream
Input
point
cloud
frame
Compression of intra point cloud frames
Occupancy
map
Intra compression
Auxiliary patch-info compression
Patchinfo
Figure 1. Overview of the PCC intra frame compression process.

80. HEVC projection + Patches Point Cloud
Compression

81. HEVC projection + Patches Point Cloud
Compression

82. HEVC projection + Octree (4) Point Cloud
Compression
Best results in subjective and objective evaluations
test model for cat2 with significant industry support

83. Mesh based approach cat 2
Point Cloud
Compression
Good results for simple applications

84. MPEG Decision
Point Cloud
Compression
- Test models based on octree for cat1 and cat3 content
- Test model based on HEVC + Patch for cat 2 (dynamic)
- Mesh based could move to a different activity (TBD)

85. Research directions
Point Cloud
Compression
- HEVC mapping related issues: occlusion points, inter prediction
Should be improved, difficult datasets without fixed normals enabling patches
- Attribute and large scale point cloud data (tree coding,
Graph transforms), out of core processing
- Mesh based geometry coding
- At some point the transport and storage formats will need attention
- Requirements of specific applications and use cases, especially
cat 3. mobile mapping should be further improved.

86. Industry directions
Point Cloud
Compression
- Rapid need for PCC in AR/VR frameworks of mobile devices
- Adoption application standards (khronos, gltf, but also ATSC, DVB)
- Mapping geospatial market is interesting for large players, but slightly
out of traditional MPEG domain
- Software and implementation aspects will become key