2. n high level understanding of images
n May involve segmentation
n mid level image understanding
n From low level representations, such as pixels and edges, to provide
representation that is compact and expressive
n Segmentation often involves grouping, perceptual organisation and
fitting.
n Image partition is taken place in the image spatial domain, but grouping,
fitting and so on can be in other domain, e.g. frequency or spatio-
frequency
Why Segmentation
3. Image Segmentation
Original image Bottom up segmentation Object level segmentation
Berkeley Segmentation
Dataset is a good starting
point (evaluation metrics)
4. Image Segmentation
n Segmentation with little constraint
n Thresholding
n Region growing, split and merge
n Watershed
n With weak constraint
n Graph cut
n Deformable models, such as active contour
n Interactive segmentation, such as intelligent scissors, grab
cut
n With strong constraint
n Active shape model, active appearance model based
segmentation
n Atlas model based segmentation
n Registration populating segmentation
n Two key questions: prior generalisation and model
adaptation
Bottomup
Modeldriven
5. Outline
n Deformable Segmentation
n Edge based 2D segmentation
n Extension to 3D
n Direct sequential segmentation of temporal volumetric data
n Incorporating statistical shape prior
n 3D + T (4D) constrained segmentation
n Tracking using implicit representation
n Implicit representation using RBF (region based)
n Hybrid approach
n Level set intrinsic regularisation (initialisation invariance)
n Integrated Reconstruction & Segmentation
n Combinatorial Optimisation
n Minimum path: lymphatic membrane segmentation
n Optimal surface (minimum cut): coronary segmentation
8. Deformable Segmentation
n Design issues
n Representation & numerical method
n Explicit Vs Implicit
n FEM, FDM, Spectral methods
n RBF-Level Set (Xie & Mirmehdi 07, Xie 11)
n Boundary description & stopping function
n Gradient based (Caselles et al. 97, Xie & Mirmehdi 08, Xie 10
& 11, Yeo et al. 11 )
n Region based (Paragios 02, Chan & Vese 01, Xie 09 & 11)
n Hybrid approach (Wang & Vemuri 04, Xie & Mirmehdi 04)
n Initialisation and convergence
n Initialisation independency (Xie 10, Xie 11, Yeo et al. 11)
n Complex topology & shape (Xie & Mirmehdi 07, 08, Paiement
14)
n These issues are often interdependent
9. Deformable model
n Active contour:
n Dynamic curves within image domain to recover object shapes.
n Deformable surface:
n Its extension to 3D.
n Applications:
n Object localisation
n Motion tracking
n Segmentation (e.g. colour/texture)
n Two general types: explicit and implicit models
n Kass et al. 1988, Caselles et al.1993, and many more…
10. Explicit model
n Parametric snake
n Represented explicitly as parameterized curves, spline, polynormial
function
n Example classic parametric snake
n Snake evolves to minimize the internal and external forces (Let C(q)
be a parameterized planar curve);
n Initialisation problem;
n Concavity convergence problem;
( ) ( ) ( ) ( )( ) .
22
∫∫∫ ∇−ʹ′ʹ′+ʹ′= dqqCIdqqCdqqCCE λβα
Internal forces External force
11. Explicit model
n Point based on tracking
n Resolution problem
n Addition and deletion
n And …
n Topological problems
n Non-intrinsic, parameterisation dependent;
n Hard to detect multiple objects simultaneously.
n Example:
?
12. Explicit model
n Advantages
n Explicit control
n Point correspondence
n Probably easier to impose shape regularisation
n Computational efficiency
n Should be considered when
n Known topology
n No (or predicated) topological changes
n Open curves
n …
n Numerical method
n Finite element method (FEM)
n Discretise into sub-domain
n Cohen & Cohen, IEEE T-PAMI, 1993
13. Implicit model
n Popularly based on the Level Set technique
n Implicit snake models
n Introduced by Caselles et al. and Malladi et al. (1993);
n Based on the theory of curve evolution
n Numerically implemented via level set methods;
n Snake evolves to minimize the weighted length in a Riemannian space
with a metric derived from the image content;
n Weighted length minimisation example:
A
B
A
B
14. Curve evolution
n Curvature flow
n k is the curvature, N denotes the inward normal
n The curvature measures how fast each point moves along its normal
direction;
n A simple closed curve will evolve toward a circular shape and
disappear;
n It smoothes the curve.
NCt
!
κ=
15. Curve evolution
n Constant flow
n c is a constant, N denotes the inward normal
n Each point moves at a constant speed along its normal direction;
n It can cause a smooth curve to become a singular one;
n A.k.a. the balloon force.
NcCt
!
=
16. Level set method
n A computational technique for tracking propagating interface
n Embed the curve into a surface, 2D scalar field
n Zero level set corresponds to the embedded curve
n Deforming the surface, instead of explicitly deforming the curve
17. Level set method
n A computational technique for tracking propagating interface
n Embed the curve into a surface, 2D scalar field
n Zero level set corresponds to the embedded curve
n Deforming the surface, instead of explicitly deforming the curve
18. Level set method
n Key ideas introduced by Dervieux & Thomasset
n Lecture Notes in Physics 1980
n Well-known after the seminal work by Osher & Sethian
n Osher & Sethian, J. Computational Physics 1988
n Fluid dynamics, computational geometry, material science, computer vision,
…
n Introduced to snake methods by Casselles et al. & Malladi et al.
n Casselles et al., Nemuer. Math. 1993
n Malladi et al., IEEE T-PAMI 1995
n Advantages:
n Implicit, intrinsic, non-parametric
n Accurate modelling front propagation
n Capable of handling topological changes (almost!)
19. Level set method
n General curve evolution
n is the level set function, F is the speed function
n The curvature flow can be re-formulated as:
n The constant flow can be re-formulated as:
0|| =Φ∇+Φ Ft
Φ
||
||
Φ∇⎟⎟
⎠
⎞
⎜⎜
⎝
⎛
Φ∇
Φ∇
⋅∇=Φt
|| Φ∇=Φ ct
20. Level set method
n Numerical method
n Finite difference method (FDM)
n A local method to estimate the partial derivatives
n Upwind scheme
n Reinitialisation
n Reshaping the level set surface to retain smoothness
n Periodically performed
n Fast marching method, or
n By solving the following PDE:
|)|1)((sign Φ∇−Φ=Φt
21. Level set method
n Challenges:
n Computational complexity
n FDM – a local method to estimate the partial derivatives
n Dense computational grid
n More expensive in 3D
n Fast marching, narrow band, additive operator splitting (AOS)
n More sophisticated topological changes
n Signed distance function
n Reinitialisation is necessary to eliminate accumulated numerical error
n Prevent the level set developing new components
n Do not allow perturbations away from zero level set
n Can not create new contours
n E.g. fail to localise internal object boundaries
22. Level set method
n Challenges:
n Computational complexity
n FDM – a local method to estimate the partial derivatives
n Dense computational grid
n More expensive in 3D
n Fast marching, narrow band, additive operator splitting (AOS)
n More sophisticated topological changes
n Signed distance function
n Reinitialisation is necessary to eliminate accumulated numerical error
n Prevent the level set developing new components
n Do not allow perturbations away from zero level set
n Can not create new contours
n E.g. fail to localise internal object boundaries
23. Numerical solution
n The level set time derivative is approximated by forward
difference:
n Force fields can be classified into three types
n Curvature flow
n Constant flow
n Advection force field
n Each requires different differencing scheme
26. Numerical solution
n Advection flow:
n Let denote the external velocity force field
n Check the sign of each component
n Construct one-sided upwind differences
n E.g. GVF,
28. Motivation
n Convergence study – 4 disc problem
GeodesicDVF GGVF GeoGGVF CVF MAC
DVF: Cohen & Cohen, IEEE T-PAMI, 1993
Geodesic: Caselles et al., IJCV, 1997
GGVF: Xu & Prince, Signal Processing, 1998
GeoGGVF: Paragios et al., IEEE T-PAMI, 2004
CVF: Gil & Radeva, EMMCVPR 2003
CPM: Jalba et al., IEEE T-PAMI 2004
Xie & Mirmehdi, MAC, IEEE Trans. Pattern Analysis & Machine Intelligence 2008.
29. Motivation
n Convergence study – 4 disc problem
GeodesicDVF GGVF GeoGGVF CVF MAC
DVF: Cohen & Cohen, IEEE T-PAMI, 1993
Geodesic: Caselles et al., IJCV, 1997
GGVF: Xu & Prince, Signal Processing, 1998
GeoGGVF: Paragios et al., IEEE T-PAMI, 2004
CVF: Gil & Radeva, EMMCVPR 2003
CPM: Jalba et al., IEEE T-PAMI 2004
Xie & Mirmehdi, MAC, IEEE Trans. Pattern Analysis & Machine Intelligence 2008.
30. Motivation
n Objectives
n Long range force interaction
n Dynamic force field, instead of static
n Bidirectional – allow cross boundary initialisation
n Efficiency
n Region based or Edge based
n Prior knowledge
n Boundary assumptions
n Discontinuity in regional statistics
n Discontinuity in image intensity
n Application dependent
n Goal: improving edge based performance
n Comparable to region based approaches
n Benefit from less prior knowledge, simpler assumption, and efficiency
n There are scenarios boundary description does not need region support
31. MAC model
n Proposed method
n Novel external force field
n Based on hypothesised magnetic interactions between object boundary
and snake
n Significant improvements upon initialisation invariancy &
convergence ability
n Yet, a very simple model
n Magnetostatics
32. MAC model
n Edge orientation
n Analogy to current orientation
n Rotating image gradient vectors
= 1: anti-clockwise rotation; = 2: clockwise rotation.
: normalised image gradient vectors.
n (actually, these are 3D vectors)
n Current orientation on snake
n Similar to edge current orientation estimation
n Rotating level set gradient vectors
33. MAC model
n Magnetic force on snake
n Derive the force on snake exerted from image gradients
: electric current unit vector on snake
: current magnitude on snake, constant
: electric current vector on edges
: current magnitude on edges
: unit vector between two point, x and s
: permeability constant
n Uniqueness
n The force on snake is dynamic
n Relies on both spatial position and evolving contour
n Always perpendicular to the snake
n Global force interaction
34. MAC model
n Snake formulation
: curvature
: snake inward normal
n Level set representation
n Force field extension
n Snake is extended in a 2D scalar function
n Accordingly its forces upon it
n Fast marching
n In this case, simply compute forces for each level set
36. MAC model
n Edge preserving force diffusion
n Minimise noise interference
n Nonlinear diffusion of magnetic flux density
n Similar to GGVF, but…
n Add edge weighting term in diffusion control
n As little diffusion as possible at strong edges
n Homogeneous and noisy area which lack consistent support
from edges will have larger diffusion
37. MAC model
n Edge preserving force diffusion
n Fast implementation
n Decompose the magnetic flux term
n Fast computation in the Fourier domain
47. n Extension to 3D
Yeo, Xie, Sazonov, Nithiarasu, GPF, IEEE Trans. Image Processing 2011.
48. GPF model
n Geometrical Potential Force
n Suitable for 3D data
n Based on hypothesised geometrically induced force field between
deformable model and object boundary
n Generalisation of the MAC model
n Unique bi-directionality
n Dynamic force interaction
n Global view of object boundary representation
Yeo, Xie, Sazonov, Nithiarasu, GPF, IEEE Trans. Image Processing 2011.
49. GPF model
n Interaction force acting on due to is given as
q – corresponding geometrically induced potential created by
69. n Segmentation of LV borders allows quantitative analysis of
perfusion defects and cardiac function.
4D SPECT Segmentation
SPECT slice of the LV A DoughnutCardiac motion (mid-slice)
Yang, Mirmehdi, Xie, Hall, CI2BM, MICCAI workshop 2009. (MIA under-review)
70. Frontal view of
opaque surface
Top view of
opaque surface
Short-axis view
Frontal view
Correspondence between short-axis
slice and 3D frontal view
Frontal view of opaque surface overlaid on
orthogonal slice planes
Frontal view of
transparent surface
4D SPECT Segmentation
72. Training Image and
Shape Sequences
Gaussian
Analysis
PCA
Gaussian
Priors
Spatiotemporal
Priors
Segmentation
Unseen
Sequence
CACE
Evolution
Constraint (based on
Gaussian and
Spatiotemporal Priors)
Update Level Sets
and Spatiotemporal
Parameters
Convergence
End
No
Yes
Training
GtL trans.
4D SPECT Segmentation
73. The image cannot be displayed.
Your computer may not have
enough memory to open the
image, or the image may have
been corrupted. Restart your
computer, and then open the file
again. If the red x still appears, you
may have to delete the image and
then insert it again.
The image cannot be displayed.
Your computer may not have
enough memory to open the
image, or the image may have
been corrupted. Restart your
computer, and then open the file
again. If the red x still appears, you
may have to delete the image and
then insert it again.
The image cannot be displayed.
Your computer may not have
enough memory to open the
image, or the image may have
been corrupted. Restart your
computer, and then open the file
again. If the red x still appears,
you may have to delete the image
and then insert it again.
The image cannot be displayed.
Your computer may not have
enough memory to open the
image, or the image may have
been corrupted. Restart your
computer, and then open the file
again. If the red x still appears, you
may have to delete the image and
then insert it again.
The image cannot be displayed.
Your computer may not have
enough memory to open the
image, or the image may have
been corrupted. Restart your
computer, and then open the file
again. If the red x still appears, you
may have to delete the image and
then insert it again.
The image cannot be displayed.
Your computer may not have
enough memory to open the
image, or the image may have
been corrupted. Restart your
computer, and then open the file
again. If the red x still appears, you
may have to delete the image and
then insert it again.
The image cannot be displayed.
Your computer may not have
enough memory to open the
image, or the image may have
been corrupted. Restart your
computer, and then open the file
again. If the red x still appears, you
may have to delete the image and
then insert it again.
The image cannot be displayed.
Your computer may not have
enough memory to open the
image, or the image may have
been corrupted. Restart your
computer, and then open the file
again. If the red x still appears, you
may have to delete the image and
then insert it again.
The image cannot be
displayed. Your computer
may not have enough
memory to open the image,
or the image may have been
corrupted. Restart your
computer, and then open the
file again. If the red x still
appears, you may have to
delete the image and then
insert it again.
The image cannot be
displayed. Your computer
may not have enough
memory to open the image,
or the image may have been
corrupted. Restart your
computer, and then open the
file again. If the red x still
appears, you may have to
delete the image and then
insert it again.
The image cannot be
displayed. Your computer
may not have enough
memory to open the image,
or the image may have been
corrupted. Restart your
computer, and then open the
file again. If the red x still
appears, you may have to
delete the image and then
insert it again.
The image cannot be
displayed. Your computer
may not have enough
memory to open the image,
or the image may have been
corrupted. Restart your
computer, and then open the
file again. If the red x still
appears, you may have to
delete the image and then
insert it again.
The image cannot be
displayed. Your computer
may not have enough
memory to open the image,
or the image may have been
corrupted. Restart your
computer, and then open the
file again. If the red x still
appears, you may have to
delete the image and then
insert it again.
The image cannot be
displayed. Your computer
may not have enough
memory to open the image,
or the image may have been
corrupted. Restart your
computer, and then open the
file again. If the red x still
appears, you may have to
delete the image and then
insert it again.
The image cannot be
displayed. Your computer
may not have enough
memory to open the image,
or the image may have been
corrupted. Restart your
computer, and then open the
file again. If the red x still
appears, you may have to
delete the image and then
insert it again.
The image cannot be
displayed. Your computer
may not have enough
memory to open the image,
or the image may have been
corrupted. Restart your
computer, and then open the
file again. If the red x still
appears, you may have to
delete the image and then
insert it again.
CCACE: 85.8%
SCMS: 63.1%
Results
on input
image
Results
against
ground
truth
Results
on input
image
Results
against
ground
truth
4D SPECT Segmentation
79. Automatic Bootstrapping & Tracking
Chiverton, Xie & Mirmehdi, IEEE T-IP 2012.
n Online shape learning coupled with automatic bootstrapping
n Finite size shape memory
n Statistical shape modelling
n Level set based tracking – similar to previous approach
80. n RBF-Level Set based Active Contouring
Xie & Mirmehdi, Image and Vision Computing 2011 & BMVC07.
81. Conventional level set technique
n Problems
n Computational complexity
n Dense computation grid, particularly expensive in 3D
n Some solutions: fast marching, narrow band, AoS schemes, …
n Can’t handle more sophisticated topological changes
n Usually requires re-initialisation to maintain a smooth surface to prevent
numerical artefacts contaminating the solution
n Perturbations away from the zero level set are missed
Conventional level set:
The hole is missed!
82. RBF-Level Set
n RBF-Level Set
n Use radial basis function to interpolate level set
n Updating expansion coefficients to deform level set
n Transfer PDE to ODE: efficient
n Much coarser computational grid, even irregular
n More complex topological changes readily achievable
Conventional level set Proposed method
83. RBF-Level Set
n RBF-Level Set
n Use radial basis function to interpolate level set
n Updating expansion coefficients to deform level set
n Transfer PDE to ODE: efficient
n Much coarser computational grid, even irregular
n More complex topological changes readily achievable
Conventional level set Proposed method
84. RBF-Level Set
n RBF interpolation
n Level set function, : a scalar function, usually obtained from the signed
distance transform
n Interpolate using a linear combination of a radial basis function,
where p(x) is a first degree polynomial and are the expansion coefficients
n The interpolation can be expressed as:
where
6
85. RBF-Level Set
n Updating RBF level set
n Original level set evolution:
where F is the speed function in the normal direction
n Transferred evolution:
n The spatial derivatives can be solved analytically
n First order Euler’s method
n Iteratively updating the expansion coefficients to evolve level set
n Benefits:
n Coarse computational grid, could be irregular
n No need for re-initialisation
n More complex topological changes achievable
89. RBF-Level Set
n Active modelling using RBF level set
n A region based approach
n Texem based modelling
n Active contour formulation:
n m is the number of classes
n 1/m is the average expectation of a class
n u is the posterior of the class of interest
n Level set representation:
90. I
RBF-Level Set
n Texems are image representations at various sizes that
retain the texture or visual primitives of a given image.
n A two-layer generative model
n Each texem represented by mean and variance: m={µ,ω}
n A bottom-up learning procedure
learning
{ }…M
Z
Xie-Mirmehdi, IEEE T-PAMI, 29(8), 2007.
91. RBF-Level Set
n Example learnt texems (7x7)
n Multiscale branch based texems
n Texem grouping for multi-modal regions
Xie-Mirmehdi, IEEE T-PAMI, 29(8), 2007.
96. RAGS model
n Region-aided (RAGS) model
n Bridge boundary and region-based techniques
n Fusing global information to local boundary description
n Improvements towards weak edges
n More resilient to noise interference
Geodesic snake Proposed methodGGVF snake
Xie & Mirmehdi, IEEE Trans. Image Processing, 2004.
97. n Integrated Reconstruction, Registration and Segmentation
A. Paiement et al., IEEE Transactions on Image Processing, January 2014.
98. Motivation
n Modelling from 3D/4D imaging data raises two intertwined issues:
n segmentation
n interpolation
n Segmentation
n partition 3D space containing the object and to distinguish data points belonging to the
object from background points
n e.g. 2D slices, ranging from simple stacks of parallel slices to more
complicated spatial configurations
99. Motivation
n Segmentation
n 2D independent segmentation is often not desirable
n all the slices are better segmented simultaneously in 3D/4D
n However, interpolation is thus necessary
n since data often does not span the whole 3D space
n only partial support from data
n Imaging conditions
n some modalities require integration over a thick slice to improve signal to noise ratio
n e.g. 1.5T cine cardiac MRI typical slice thickness 7mm; hence spacing is 7mm or bigger
n large spacing is also desirable in order to reduce acquisition time (patient discomfort,
motion artefact)
n in the 4D case, data must also be interpolated between available time frames
100. Motivation
n Argument
n “the success of one stage (segmentation or interpolation) depends on the accuracy of the
other”
n Approaches
n two sequential approaches: perform these two stages in opposing order
n some first segment slices independently then interpolate from 2D contours
n shape interpolation, notable work: Liu et al., surface reconstruction from Non-parallel
curve network, CGF 27(2) 2008.
n combining registration and segmentation
n segment sparse volumes made up of 2D slices by registering and deforming a model on
images (e.g. ASM): prior is often necessary
n level set based method: foundation (earlier work) for what presented here
n integrate segmentation and interpolation into a new RBF interpolated level set framework
n simplicity and flexibility of level set
n stability of RBF
n inherent interpolation provided by RBF
101. Proposed Method
n Interpolate level set function using Strictly Positive Definitive (SPD) RBF
n combining registration and segmentation
n segment sparse volumes made up of 2D slices by registering and deforming a model
on images (e.g. ASM): prior is often necessary
n level set based method: foundation (earlier work) for what presented here
n integrate segmentation and interpolation into a new RBF interpolated level set framework
n simplicity and flexibility of level set
n stability of RBF
n inherent interpolation provided by RBF
102. Proposed Method
n Instead of evolve phi through expansion coefficients (which involves
inverting a large matrix), evolve alpha by minimising an energy functional E:
n F may be any functional and is defined by the chosen segmentation method.
n Conventional variational level set method:
n using chain rule, a gradient descent method yields:
103. Proposed Method
n Rename as .
n S is the speed of the moving front and is generally defined on the contour C
only.
n The alpha evolution function can thus be simplified as
n is an approximation of the Dirac function
n this imposes a restriction of S to the contour C
n practical choice of regularised Dirac function:
§ epsilon = 1 for sharp RBF; epsilon = 3 for flatter RBF.
104. Proposed Method
n RBF based interpolation methods usually define one control point per data
point
n we define one control point per voxel of a discrete space,
n thus allow rewrite as a
convolution:
n The initial expansion coefficients can be computed in the Fourier domain
114. n Team
n Dr. Feng Zhao, Mr. Ehab Essa, Mr. Jingjing Deng, Mr. Mike Edwards,
Mr. Robert Palmer, Mr. Yaxi Ye, Mr. David James, Mr. Jonathan
Jones
n Alumni
n Dr. Ben Daubney, Dr. Huazizhong Zhang, Dr. Dongbin Chen, Dr. Si
Yong Yeo, Dr. Cyril Charron, Dr. John Chiverton, Dr. Ronghua Yang,
Mr. Liu Ren, Mr. Arron Lacey
n Clinical collaborator
n Swansea Singleton Hospital ABM UHT at Morriston
n Bristol Royal Infirmary
n Cardiff Hospital
n Plymouth Hospital
Acknowledgement
csvision.swan.ac.uk