The document discusses DIEGO, a distributed system used by Cloud Foundry that orchestrates containerized workloads. It describes DIEGO's architecture which includes Cells that host applications, a Brain that monitors health and coordinates scheduling, and a backing service (BBS) like etcd. DIEGO runs Tasks, which are one-off units of work, and Long-Running Processes (LRPs) that have multiple instances for high availability.

2. The Go gopher was designed by Renee French

3. DIEGO
what’s new with
onsi fakhouri

4. last year…

5. last year…
story

6. last year…

7. last year…
HOPE

8. last year…
HOPEA NEW

9. this year….

10. this year….
…it’s complicated

11. this year….
…but there’s a plot twist

12. this year….
…but there’s a plot twist

13. DIEGOstrikes back

14. ? ?

15. ? ?

16. ? a rewrite
RUNTIME

17. ? a rewrite
RUNTIME
DEA HM Warden

18. ? a rewrite
RUNTIME
(in GO)
DEA HM Warden

19. ?
DEAGO

20. ?
DEAGO

21. ?
DIEGO

22. ? DIEGO is
a distributed system that
orchestrates containerized workloads

23. ? DIEGO is
a distributed system that
orchestrates containerized workloads

24. ? DIEGO is
a distributed system that
orchestrates containerized workloads

25. ? DIEGO is
a distributed system that
orchestrates containerized workloads

26. ? DIEGO is
a distributed system that orchestrates
containerized workloads

27. ? DIEGO is
a distributed system that orchestrates
containerized workloads
Cells

28. ? DIEGO is
a distributed system that orchestrates
containerized workloads
Cells
Brain

29. ? DIEGO is
a distributed system that orchestrates
containerized workloads
Cells
Brain
BBS
(currently etcd)

30. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)

31. ? DIEGO
a distributed system that orchestrates containerized
workloads
Cells
Brain
BBS
(currently etcd)
scheduler

32. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
scheduler

33. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
scheduler

34. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
scheduler

35. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
health-monitor

36. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
health-monitor

37. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
health-monitor

38. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
health-monitor

39. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
health-monitor

40. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
health-monitor

41. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
health-monitor

42. ? DIEGO
a distributed system that orchestrates containerized workloads
Cells
Brain
BBS
(currently etcd)
health-monitor

43. ? DIEGO runs
a distributed system that orchestrates containerized workloads

44. ? DIEGO runs
a distributed system that orchestrates containerized workloads
one-off tasks
long running
processes

45. ? DIEGO runs
a distributed system that orchestrates containerized workloads
long running
processes
Task
a unit of work
runs at most once

46. ? DIEGO runs
a distributed system that orchestrates containerized workloads
Task LRP
a unit of work
runs at most once
N long-running instances
distributed across cells for HA
monitored & restarted

47. ? DIEGO runs
a distributed system that orchestrates containerized workloads
Task LRP
generic, platform independent, abstraction

48. ? DIEGO runs
a distributed system that orchestrates containerized workloads
Task LRP
generic, platform independent, abstraction

49. ? DIEGO runs
a distributed system that orchestrates containerized workloads
Task LRP
working today
generic, platform independent, abstraction

50. ? DIEGO runs
a distributed system that orchestrates containerized workloads
Task LRP
successful abstraction
working today

51. …confusion

52. …confusion
=
?

53. …confusion
? ?

54. …confusion
?

55. …confusion

56. ?

57. ? ?

58. ?
isolation
?

59. ? isolation
shared resources
processA
processB
processC
processD
processE
processF
kernel
tenant 1 tenant 2 tenant 3

60. ? isolation
shared resources
kernel
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3

61. ? isolation
CPU
kernel
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3

62. ? isolation
resource isolation
namespace isolation
CPU
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3

63. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
CPU

64. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
CPU

65. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
cgroups
CPU

66. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
cgroups
processD
processE
processF
CPU

67. ? isolation
shared resources
kernel
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3

68. ? isolation
kernel
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
ProcessID

69. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
PID 2 3 4 5 6 7

70. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
PID 2 3 4 5 6 7

71. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
PID 2 3 4 5 6 7

72. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
PID 2 3 4 5 6 7
PID namespace

73. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
PID 2 3 4 5 6 7
PID namespace

74. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
PID 2 3 4 2 2 3
PID namespace

75. ? isolation
resource isolation
namespace isolation
processA
processB
processC
processD
processE
processF
tenant 1 tenant 2 tenant 3
PID
shared resources
kernel
Network
Mount
User
namespaces

76. ?
?

77. ?
? =
isolation
User
Network
cgroups
PID

78. ?
?
? =
isolation
PID
User
Network
cgroups

79. ?
? =
isolation
PID
User
Network
cgroups
+
contents

80. ?
? =
isolation
PID
User
Network
cgroups
+
contents
+
processes

81. ?
? =

82. ?
Tasks
LRPs
in

83. ?
Tasks
LRPs
in Garden

84. ?
Garden
allows Diego to programmatically say
“make me a container” “put this in it” “then run this”
via a platform-agnostic API

85. ?
Garden
allows Diego’s abstractions to be flexible

86. ?
cf push

87. ?
cf push haiku
here is my source code
i do not care how
run it on the cloud for me

88. ?
app
source
code
Task
staging
cf push

89. ? cf push
compiled asset
app + app-specific dependencies
assumes a particular execution context
cflinuxfs2

90. ? cf push
?

91. ? cf push
LRP

92. ? cf push

93. ? cf push
cflinuxfs2
preloaded rootfs

94. ? cf push
cflinuxfs2
preloaded rootfs
download droplet

95. ? cf push
cflinuxfs2
preloaded rootfs
download droplet
start command

96. ? cf push
Droplet LRP
{
memory: 128mb,
rootfs:
“preloaded:cflinuxfs2”,
setup: <download-droplet>,
run: {metadata}.start-command
}

97. ? cf push
Droplet LRP
{
memory: 128mb,
rootfs:
“preloaded:cflinuxfs2”,
setup: <download-droplet>,
run: {metadata}.start-command
}

98. ? cf push
{
memory: 128mb,
rootfs:
“preloaded:cflinuxfs2”,
setup: <download-droplet>,
run: {metadata}.start-command
}
Droplet LRP

99. ? cf push
{
memory: 128mb,
rootfs:
“preloaded:cflinuxfs2”,
setup: <download-droplet>,
run: {metadata}.start-command
}
Droplet LRP

100. ?
cf push

101. ?
cf push-docker

102. ? cf push-docker

103. ? cf push-docker
docker image

104. ? cf push-docker
docker image docker metadata

105. ? cf push-docker
docker image docker metadata
docker registry
}

106. ? cf push-docker
docker image docker metadata
docker registry
}

107. ?
Docker LRP
{
memory:128mb,
rootfs: “docker://docker-image”,
run: {docker metadata}.start-
command
}
cf push-docker

108. ?
Docker LRP
{
memory:128mb,
rootfs: “docker://docker-image”,
run: {docker metadata}.start-
command
}
cf push-docker

109. ?
Docker LRP
{
memory:128mb,
rootfs: “docker://docker-image”,
run: {docker metadata}.start-
command
}
cf push-docker

110. ?
Docker LRP
{
memory:128mb,
rootfs: “docker://docker-image”,
run: {docker metadata}.start-
command
}
cf push-docker

111. ?
?

112. ?
?
(anything)

113. ?
?
(anything)

114. ?
?
(anything) (anything)

115. ?
?
(anything) (anything)

116. ?
?

117. ?
?
appc

118. ?
cf push-docker

119. ?
cf push -stack windows

120. ?
Garden-Windows
resource isolation
kernel job object
disk quotas
namespace isolation
user profiles
Host Web Core
(an isolated IIS instance)
Garden-Linux
resource isolation
cgroups
namespace isolation
PID
Network
User
Mount

121. ?
collaborating with Microsoft
Garden-Windows

122. ?
Garden-Windows
provides a container experience for Windows 2012
that will only get better with Windows 2016
allows us to build a cf push experience

123. ?
Garden-Linux Garden-Windows

124. ?
Garden-Linux Garden-Windows
?
Garden API

125. ?
Garden APIGarden-LinuxGarden-Windows

126. ?
.net LRP
{
memory: 128mb,
rootfs:
“preloaded:windows2012R2”,
setup: <download-application>
run: {metadata}.start-command
}

127. ?
.net LRP
{
memory: 128mb,
rootfs:
“preloaded:windows2012R2”,
setup: <download-application>
run: {metadata}.start-command
}

128. ?
.net LRP
{
memory: 128mb,
rootfs:
“preloaded:windows2012R2”,
setup: <download-application>
run: {metadata}.start-command
}

129. ?
.net LRP
{
memory: 128mb,
rootfs:
“preloaded:windows2012R2”,
setup: <download-application>
run: {metadata}.start-command
}

130. ?
.net LRP
{
memory: 128mb,
rootfs:
“preloaded:windows2012R2”,
setup: <download-application>
run: {metadata}.start-command
}

131. ?
3 different contexts

132. ? 1 cluster

133. ? ?

134. API

135. API

136. architecture code

137. architecture code
tests
TDD

138. architecture code
tests
e.g. performance, simulation

139. architecture code
tests
e.g. performance, simulation

140. architecture code
tests

141. architecture code
tests
(agile architecture)

142. last year…

143. Cells
Brain
BBS

144. Distributed auction
Cells
Brain
BBS

145. Distributed auction
Cells
Brain
BBS

146. architecture

147. architecture code

148. architecture
tests
code

149. architecture
simulation
code

150. architecture
100 cells
code
simulation

151. architecture
100 cells
200 cells
code
simulation

152. architecture code++
100 cells
200 cells
simulation

153. architecture code++
100 cells
200 cells
simulation

154. architecture code++
100 cells
200 cells
400 cells
simulation

155. architecture code++++++
100 cells
200 cells
400 cells
simulation

156. architecture code++++++
100 cells
200 cells
400 cells
simulation

157. architecture code++++++
100 cells
200 cells
400 cells
1000s cells?
simulation

158. architecture code++++++
100 cells
200 cells
400 cells
1000s cells?
simulation

159. architecture code++++++
100 cells
200 cells
400 cells
1000s cells?
simulation

160. Cells
Brain
BBS
Distributed auction

161. Cells
Brain
BBS
Centralized HA
scheduler

162. API
(it’s just simpler this way)

163. API

164. cf push
Cloud
Controller

165. Cloud
Controller
DEA
cf push
stage
DEA
DEA
DEA
run

166. Cloud
Controller
cf push
stage
run

167. Cloud
Controller
cf push
stage
run
app-specific

168. Cloud
Controller
cf push
stage
run
app-specific generic

169. Cloud
Controller
cf push
stage
run
CC
Bridge
app-specific generic
task
LRP

170. Cloud
Controller
cf push
stage
run
CC
Bridge
task
LRP
Cells
BrainBBS
app-specific generic

171. Cloud
Controller
cf push
stage
run
CC
Bridge
task
LRP
Cells
BrainBBS
app-specific generic

172. Cloud
Controller
cf push
stage
run
CC
Bridge
task
LRP
Cells
BrainBBS
app-specific generic

173. (DB != API)

174. Cloud
Controller
cf push
stage
run
CC
Bridge
task
Cells
BrainBBS
LRP

175. Cloud
Controller
cf push
stage
run
CC
Bridge
Cells
BrainBBS
task
LRP

176. Cloud
Controller
cf push
stage
run
CC
Bridge
Cells
BrainBBS

177. Cloud
Controller
cf push
stage
run
CC
Bridge
Cells
BrainBBS
ReceptorAPI

178. Cloud
Controller
cf push
stage
run
CC
Bridge
Cells
BrainBBS
ReceptorAPI

179. Cloud
Controller
CC
Bridge
Cells
BrainBBS
ReceptorAPI
generic consumer

180. Cloud
Controller
CC
Bridge
Cells
BrainBBS
ReceptorAPI
generic consumer

181. Cloud
Controller
CC
Bridge
Cells
BrainBBS
ReceptorAPI
generic consumer
other consumers?

182. ? ?

183. Cells
BrainBBS
ReceptorAPI
Task or LRP

184. Cells
BrainBBS
ReceptorAPI
Task or LRP
meh

185. Cells
BrainBBS
ReceptorAPI
Task or LRP
gorouter
http traffic

186. Cells
BrainBBS
ReceptorAPI
Task or LRP
gorouter
http traffic
loggregator
logs

189. vagrant up

190. vagrant up
terraform apply

191. vagrant up
terraform apply
ltc create <app>

192. lattice.cf

193. lattice.cf
Local VM

194. lattice.cf
Local VM
AWS
Digital Ocean
Google Cloud Platform
OpenStack (thanks!)

195. ?
?Why

196. ?
CC
UAA
Diego
Loggregator
Gorouter
Buildpacks
Services
BOSH

197. ?
CC
UAA
Diego
Loggregator
Gorouter
Buildpacks
Services
BOSH

198. ?
CC
UAA
Diego
Loggregator
Gorouter
Buildpacks
Services
BOSH
Diego
Loggregator
Gorouter

199. ?
CC
UAA
Diego
Loggregator
Gorouter
Buildpacks
Services
BOSH
single-tenant

200. ?
CC
UAA
Diego
Loggregator
Gorouter
Buildpacks*
Services
BOSH
docker
single-tenant

201. ?
CC
UAA
Diego
Loggregator
Gorouter
Buildpacks*
Services
BOSH
BYOS
docker
single-tenant

202. ?
CC
UAA
Diego
Loggregator
Gorouter
Buildpacks*
Services
BOSH
no rolling upgrades
BYOS
docker
single-tenant

203. ?
cluster root
(now go play)

204. ?
?Why

205. ?
…is a useful low-barrier solution to
real-world problems
…makes exploring Diego easy
…is a softer onramp to the CF tech
stack
…allows us to efficiently prototype
new ideas for Diego’s future
Lattice…

206. ? ?

207. WHEN?

208. “rewrite the DEA”
Diego’s scope is much more than

209. WHEN?

210. Diego is running in production on PWS
Managing ~5% of the load
Running Pivotal’s internal applications

211. WHEN?

212. Diego is in beta while we
validate performance at O(~100s) of cells
secure Diego’s internal components

213. WHEN?

214. Start using it alongside the DEAs now
and give us feedback

215. WHEN?

216. Diego should be out of beta within Q3
(probably)

217. Then what?

218. Placement Constraints
top of backlog post-beta

219. cf ssh <app/index>
working now, CLI support on the way
shell access, port forwarding, scp

220. TCP Routing
kicking off with GE
Cloud Foundry and IOT protocol support
Atul Kshirsagar - Monday 1:30

221. Private Docker Registry
in collaboration with SAP
Let Diego Manage your Docker Application
Georgi Saber - Tuesday 2:10

222. Support for persistence
(a long term goal)
The Road to Persistence on Cloud Foundry Diego
Caleb Miles & Ted Young - Tuesday 11:20

223. Container-Container networking
(a long term goal)

224. Condenser
lightweight buildpacks for Lattice

225. ? ?

226. github.com/cloudfoundry-incubator/diego
lattice.cf
xray.cf
open-house: monday@1:30