Slides I presented at GlueCon 2018

1. Google Cloud Platform
How Does Kubernetes
Build OpenAPI Specifications?
Gluecon
2018-05-16
Daniel Smith <dbsmith@google.com>
Staff Software Engineer
originalavalamp@ (twitter)
(c) Google LLC

2. Kubernetes Resource Model in 60 seconds
● Allow humans and automated systems to work together
● Standardize all the things!
○ metadata
○ verbs
● Little control loops instead of big state machines
● JSON and proto transport mechanisms
● Clients generated from OpenAPI specs!
● And OpenAPI specs are generated from...

3. Our IDL. ish.
// Deployment enables declarative updates for Pods and ReplicaSets.
type Deployment struct {
metav1.TypeMeta `json:",inline"`
// Standard object metadata.
// +optional
metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
// Specification of the desired behavior of the Deployment.
// +optional
Spec DeploymentSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
// Most recently observed status of the Deployment.
// +optional
Status DeploymentStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

4. Our IDL. ish.
// Deployment enables declarative updates for Pods and ReplicaSets.
type Deployment struct {
metav1.TypeMeta `json:",inline"`
// Standard object metadata.
// +optional
metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
// Specification of the desired behavior of the Deployment.
// +optional
Spec DeploymentSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
// Most recently observed status of the Deployment.
// +optional
Status DeploymentStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}
User-visible documentation
Instructions for OpenAPI generator
Compilable go code

5. Our IDL - Continued
// DeploymentSpec is the specification of the desired behavior of the Deployment.
type DeploymentSpec struct {
// Number of desired pods. This is a pointer to distinguish between explicit
// zero and not specified. Defaults to 1.
// +optional
Replicas *int32 `json:"replicas,omitempty" protobuf:"varint,1,opt,name=replicas"`
// Template describes the pods that will be created.
Template v1.PodTemplateSpec `json:"template" protobuf:"bytes,3,opt,name=template"`
// Indicates that the deployment is paused.
// +optional
Paused bool `json:"paused,omitempty" protobuf:"varint,7,opt,name=paused"`
...
// The maximum time in seconds for a deployment to make progress before it
// is considered to be failed. The deployment controller will continue to
// process failed deployments and a condition with a ProgressDeadlineExceeded
// reason will be surfaced in the deployment status. Note that progress will
// not be estimated during the time a deployment is paused. Defaults to 600s.
ProgressDeadlineSeconds *int32 `json:"progressDeadlineSeconds,omitempty" protobuf:"varint,9,opt,name=progressDeadlineSeconds"`
}

6. ...shouldn’t that be a proto file?

7. ...shouldn’t that be a proto file?
● Maybe

8. (Phase 1) Compile step
● Repo: kubernetes/kube-openapi
● Uses a go parser / code generator library (“gengo”)
● Defines some extension tags...
// This is the comment tag that carries parameters for open API generation.
const tagName = "k8s:openapi-gen"
const tagOptional = "optional"
// Known values for the tag.
const (
tagExtensionPrefix = "x-kubernetes-"
tagPatchStrategy = "patchStrategy"
tagPatchMergeKey = "patchMergeKey"
patchStrategyExtensionName = "patch-strategy"
patchMergeKeyExtensionName = "patch-merge-key"
)

9. Output artifact: another go file
func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
...
"k8s.io/api/apps/v1.Deployment": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "Deployment enables declarative updates for Pods and ReplicaSets.",
Properties: map[string]spec.Schema{
"kind": {
SchemaProps: spec.SchemaProps{
Description: "Kind is a string value ...",
Type: []string{"string"},
Format: "",
},
},
"apiVersion": {
SchemaProps: spec.SchemaProps{
Description: "APIVersion defines the versioned schema ...",
Type: []string{"string"},
Format: "",
},
},
"metadata": {
SchemaProps: spec.SchemaProps{
Description: "Standard object metadata.",

10. Output artifact: another go file
func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
...
"k8s.io/api/apps/v1.Deployment": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "Deployment enables declarative updates for Pods and ReplicaSets.",
Properties: map[string]spec.Schema{
"kind": {
SchemaProps: spec.SchemaProps{
Description: "Kind is a string value ...",
Type: []string{"string"},
Format: "",
},
},
"apiVersion": {
SchemaProps: spec.SchemaProps{
Description: "APIVersion defines the versioned schema ...",
Type: []string{"string"},
Format: "",
},
},
"metadata": {
SchemaProps: spec.SchemaProps{
Description: "Standard object metadata.",
from: https://github.com/go-openapi/spec
// OpenAPIDefinition describes single type. Normally
these definitions are auto-generated using gen-openapi.
type OpenAPIDefinition struct {
Schema
spec.Schema
Dependencies []string
}

11. Output artifact: another go file
...
"k8s.io/apimachinery/pkg/util/intstr.IntOrString": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Type: intstr.IntOrString{}.OpenAPISchemaType(),
Format: intstr.IntOrString{}.OpenAPISchemaFormat(),
},
},
},
Define your own!
// OpenAPISchemaType is used by the kube-openapi generator when constructing
// the OpenAPI spec of this type.
//
// See: https://github.com/kubernetes/kube-openapi/tree/master/pkg/generators
func (_ IntOrString) OpenAPISchemaType() []string { return []string{"string"} }
// OpenAPISchemaFormat is used by the kube-openapi generator when constructing
// the OpenAPI spec of this type.
func (_ IntOrString) OpenAPISchemaFormat() string { return "int-or-string" }

12. Phase 2 (runtime): Time to add the verbs...
From here...
case "PUT": // Update a resource.
doc := "replace the specified " + kind
if isSubresource {
doc = "replace " + subresource + " of the specified " + kind
}
handler := metrics.InstrumentRouteFunc(action.Verb, resource, subresource, requestScope,
restfulUpdateResource(updater, reqScope, admit))
route := ws.PUT(action.Path).To(handler).
Doc(doc).
Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
Operation("replace"+namespaced+kind+strings.Title(subresource)+operationSuffix).
Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
Returns(http.StatusOK, "OK", producedObject).
// TODO: in some cases, the API may return a v1.Status instead of the versioned object
// but currently go-restful can't handle multiple different objects being returned.
Returns(http.StatusCreated, "Created", producedObject).
Reads(defaultVersionedObject).
Writes(producedObject)
addParams(route, action.Params)
routes = append(routes, route)

13. Phase 2 (runtime): Construct the spec
// BuildAndRegisterOpenAPIVersionedService builds the spec and registers a handler to provides access to it.
// Use this method if your OpenAPI spec is static. If you want to update the spec, use BuildOpenAPISpec then
RegisterOpenAPIVersionedService.
func BuildAndRegisterOpenAPIVersionedService(servePath string, webServices []*restful.WebService, config *common.Config, handler
common.PathHandler) (*OpenAPIService, error) { ... }
// BuildOpenAPISpec builds OpenAPI spec given a list of webservices (containing routes) and common.Config to customize it.
func BuildOpenAPISpec(webServices []*restful.WebService, config *common.Config) (*spec.Swagger, error) {
o := openAPI{
config: config,
swagger: &spec.Swagger{
SwaggerProps: spec.SwaggerProps{
Swagger: OpenAPIVersion,
Definitions: spec.Definitions{},
Paths: &spec.Paths{Paths: map[string]spec.PathItem{}},
Info: config.Info,
},
},
}
err := o.init(webServices)
if err != nil {
return nil, err
}
return o.swagger, nil
}

14. Phase 2.5: Aggregation
// MergeSpecs copies paths and definitions from source to dest, rename definitions if needed.
// dest will be mutated, and source will not be changed. It will fail on path conflicts.
func MergeSpecs(dest, source *spec.Swagger) error {
return mergeSpecs(dest, source, true, false)
}
// MergeSpecsIgnorePathConflict is the same as MergeSpecs except it will ignore any path
// conflicts by keeping the paths of destination. It will rename definition conflicts.
func MergeSpecsIgnorePathConflict(dest, source *spec.Swagger) error {
return mergeSpecs(dest, source, true, true)
}
// FilterSpecByPaths removes unnecessary paths and definitions used by those paths.
// i.e. if a Path removed by this function, all definition used by it and not used
// anywhere else will also be removed.
func FilterSpecByPaths(sp *spec.Swagger, keepPathPrefixes []string) {
...
}

15. Phase 3: Client usage
● The spec can change on the fly, so use ETAGs
● kubectl (our CLI) caches discovery information
● It is big, so compress
● It is slow to unmarshal JSON, so use proto
○ We reused the proto format from gnostic
● Many versions
○ OpenAPI version
○ proto encoding version
○ Our API version

16. Future work ideas
● Declare as much as possible in our IDL
○ Validation
○ Defaults
○ Const/immutability marker
○ Which of the standard verbs we support?
○ Subresources?
● OpenAPI -> protobuf definition?
○ `x-proto-tag` extension? (From openapi2proto)
● Custom extension tags
○ We’re moving `kubectl apply` (schema-aware smart update feature) to the control plane
○ (this could be a large rabbit hole)

17. Google Cloud Platform
Thank you!
Sound interesting? We’re hiring...
(c) Google LLC