Gluecon 2017: Metadata is the Glue
•Télécharger en tant que PPTX, PDF•
2 j'aime•2,401 vues
The document discusses how metadata is crucial for connecting different software systems and applications within companies. It describes how companies are becoming networks of interconnected software applications for various business functions like manufacturing, HR, marketing, etc. It argues that for this software integration to work effectively, the different systems need standardized and well-defined metadata descriptions in formats like API Modeling Framework (AMF) and OpenAPI Spec. This metadata provides information about how the APIs of different systems connect and interface with each other.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Gluecon 2017: Metadata is the Glue
Similaire à Gluecon 2017: Metadata is the Glue (20)
Plus de MuleSoft
Plus de MuleSoft (20)
Dernier
Dernier (20)
Gluecon 2017: Metadata is the Glue
- 1. Uri Sarid, CTO, MuleSoft Gluecon 2017 Metadata is the Glue @usarid
- 2. All contents © MuleSoft Inc. @usarid Companies are becoming software companies… 2@usarid
- 3. All contents © MuleSoft Inc. @usarid Companies are becoming software companies… 3 MANUFACTURING HR FINANCE MARKETING SALES SERVICES R&D @usarid
- 4. All contents © MuleSoft Inc. @usarid But what kind of software? 4 Isn’t it already digitized? Don’t we already have software for that? The odds are good but the goods are odd @usarid
- 5. All contents © MuleSoft Inc. @usarid But what kind of software? 5 and they’re all connected in all sorts of ways! @usarid
- 6. All contents © MuleSoft Inc. @usarid Where is this going? 6@usarid
- 7. All contents © MuleSoft Inc. @usarid Where is this going? 7@usarid
- 8. All contents © MuleSoft Inc. @usarid Software Creation Coherent Software Assembly 8 Order Management
- 9. All contents © MuleSoft Inc. @usarid Software Creation Coherent Software Assembly 9 Order Management Inventory Shipping & Fulfillment
- 10. All contents © MuleSoft Inc. @usarid Software Creation Coherent Software Assembly 10 Order Management Inventory Shipping & Fulfillment Customer Experience application network
- 11. All contents © MuleSoft Inc. @usarid The key is good metadata 11
- 12. All contents © MuleSoft Inc. @usarid The key is good metadata 12 API API API API API API
- 13. All contents © MuleSoft Inc. @usarid The key is good metadata 13
- 14. All contents © MuleSoft Inc. @usarid The key is good metadata 14
- 15. All contents © MuleSoft Inc. @usarid The API spec 15 designemit domain model docs live console scripting – API Notebook mocking SDK manage
- 16. All contents © MuleSoft Inc. @usarid The functional spec 16 model your API product
- 17. All contents © MuleSoft Inc. @usarid The functional spec 17 see the result as consumers would
- 18. All contents © MuleSoft Inc. @usarid The functional spec 18 flesh out the request, the response…
- 19. All contents © MuleSoft Inc. @usarid Use common behaviors 19 reusable library of behavior traits common paging behavior common caching behavior apply the traits as needed end up with a correct, consistent API
- 20. All contents © MuleSoft Inc. @usarid Use common structures 20 a collection resource type a member resource type apply the resource types as needed end up with a correct, consistent API
- 21. All contents © MuleSoft Inc. @usarid Model underlying data types 21 traits resource types end up with a correct, consistent API apply data types as needed
- 22. All contents © MuleSoft Inc. @usarid Policies are extensions 22 traits resource types data types version your functional spec like you version your code this is your source code policies are operational extensions they shouldn’t affect the functional spec reusable security scheme clients see an effective API spec with security built in
- 23. All contents © MuleSoft Inc. @usarid Overlay usecase: implementation 23 functional API spec general, reusable annotations for AWS API Gateway configuration specific config for this API’s implementation
- 24. All contents © MuleSoft Inc. @usarid API metadata is layered and reusable 24
- 25. All contents © MuleSoft Inc. @usarid What format? 25 API Modeling Framework RAML separation of concerns reuse - consistency modeling tools platforms layered modeling OpenAPI Spec common description common description
- 26. All contents © MuleSoft Inc. @usarid What format? Yes! 26 API Modeling Framework RAML separation of concerns reuse - consistency modeling tools platforms layered modeling OpenAPI Spec common description common description
- 27. All contents © MuleSoft Inc. @usarid AMF: Framework for API metadata 27 SPECIFICATION OpenAPI Spec parser/emitterRAML parser/emitter great tooling… higher-level concepts… AMF Common document modelAPI Common API-domain modelAPI
Notes de l'éditeur
- When the Glue Conferences started in 2008, "Glue" was roughly about "web oriented architecture" and "Interop for the Web". This has stood the test of time, as has the quest for systems coupling that's as loose as possible and no looser; for approaches and tools that empower developers to do more and have less friction. Nine years later, we're weaving a truly huge network of applications and APIs, and perhaps the most important element of the “glue” that will enable this growth to continue and compound will be metadata.
- Let’s start with where things stand today, in the enterprise world. The adage of ”software is eating the world” is largely already true in companies, certainly major enterprises.
- Pretty much wherever you look, in every sector of an enterprise, from front office to back office, from manufacturing to marketing, from internal corporate functions to external customer-facing ones, the business is run using software – in fact, lots of software, lots of systems.
- So if some executive were to ask about any part of the business whether it’s already “digital”, whether it’s a good bet that if he or she were to look into how that part of the business runs, would he find that it’s run on software? The answer, as my son’s friends like to say when they consider whether they’re likely to meet people at a party, is that the odds are good, but the goods are odd. There is a high likelihood of digital systems being in place, but they are of all shapes and sizes, more often than not kludged together, some new and some semi-abandoned, some fresh and exciting but many humming away under a desk or on a rack or on someone’s personal cloud account – and that someone may no longer even be with the company. Some is 1980’s tech that came with that acquisition from 8 years ago, some was the personal project of the chief architect that left in a hurry after the delayed release, and much seemed such a good idea at that time. To be sure, much of it works, and keeps the lights on. But is it ready for today’s challenges? And can you build agility on it?
- The question isn’t restricted to just the software systems themselves. Because for most of them, their value derives when they’re connected to, and integrated with, other systems. So over the years and decades, thin and thick pipes have been built across these systems, spreading their complexity, leading to collective behavior that may not be fully understood by any one person in the company today. The result is delays in implementing any new or changed capabilities, fragility, and often – paralysis. The opposite of agility.
- So where is this going? The answer is: it’s just going to get amplified. Because functions and people within the enterprise are, if anything, becoming hyper-specialized, and so are the software systems they build or buy, so there are many more of them – and many more needed to tie together to achieve larger business goals. Developments like massive deployments of IoT devices, or massive investments in microservices, greatly increase the fragmentation of the software estate.
- And any illusions that this is remotely controllable by “central IT” are shattered by the dominant themes of software infrastructure moving to the cloud and the consumers of software moving to mobile. Add to that the growth of shadow IT across all functions, and the compelling value propositions of SaaS offerings, and it’s clear the genie will never go back in the bottle. Fragmentation is here to stay, so we’d better get good at dealing with it.
- Specifically, I argue that while virtually all companies need to become software-powered companies, most companies should not become great software-creating companies; rather, they should become great software-assembling companies. The fundamental capabilities they need, whether it’s managing orders, managing inventory, fulfilling orders, managing relations with customers, etc. etc. – all these exist, and most companies are much better off figuring out how to leverage those capabilities much better, and set themselves up for leveraging new ones as they become available. Agility comes when you can quickly compose and recompose yourself to adjust to, or get ahead of, the needs of the day and the opportunities that come up, rather than getting bogged down by having to build things from scratch every time, or having to untangle how everything was previously connected. You need a way to leverage your existing investments even if you need to recompose them, you need to create more leverage with every new project, and you need to make full use of the talent of your organization. How? In every project, if you implement it by creating small services with productized APIs, they will not only be easier to deal with for this project – they’ll form building blocks for future projects.
- When the next project or three show up, developers will reuse services produced before, though perhaps in new ways; they’ll innovate through new composition, but they’ll also likely to create a few new building blocks.
- And those in turn will be useful for the next project. It won’t be long before the reinvention of wheels goes down dramatically, and a network effect sets in: every new project adds value to the network of applications already in place. Note that those applications consist of the applications the developers were working to compose (indicated in black), as well as the composite applications they created to do so (shown in blue). We call the result an “application network”, and it is analogous to computer networks in that it’s an inherently distributed and bottoms-up concept, one that’s resilient in the face of change, one that offers plug&play advantages to its users, and one whose benefits amplify as it gains scale.
- For today, I want to focus on a key enabler of such a network: I argue that the key is the metadata, specifically the value you get out of having good metadata about the building blocks of the application network.
- To get even more specific, I’ll focus on a particularly critical part of the application network: the APIs that expose any service’s capabilities to the rest of the network of services. I include evented as well as web APIs in my definition, but we’ll focus on web APIs, specifically RESTful (or at least resource-oriented) APIs, for today.
- Why metadata? Because metadata lets developers be intentional, and because it allows computers and automation to help developers focus on the job at hand and do that job better. An analogy is to be found in surgery: when the surgeon can literally see what he’s doing, enhanced and augmented by cutting edge technology, he ca n make intentional decisions on what to remove and what to keep, where to cut and where to suture.
- And she can focus on applying her smarts and her training and her intuition and creativity to the task at hand, while letting the machines overcome the drudgery of the procedure, the miniature physical manipulation often needed, and the remoteness of the patient.
- The metadata that describes the API is the API specification: a machine-readable document that’s also, ideally highly readable and writeable by humans (or at least by developers). How do you get such metadata? Ideally it’s intentionally designed, like any good product, with the consumers in mind. But perhaps it’s derived from the implementation of the service. Though rare today, it might also be the result of higher-level business modeling. However it’s arrived at, it ca be put to use to generate developer-friendly documentation; a live console to greatly ease exploring and testing the service via its API; it cam be simulated – mocked – before it’s implemented, so consumers can verify it meets their needs before an investment is made in building it; it can be used to generate SDK that faithfully reflect its capabilities and connect to the service in the language of choice; it can be used to quickly script scenarios of consumption, including mashing together multiple APIs, by using the API Notebook, proving out the value of the API (even if it’s just mocked) and building up documentation in the process; and it can be used to manage the service, e.g. by applying policies to appropriate parts of the API.
- Let’s look at that functional spec. I find it a great way to model the domain of the service I’m creating, similar to how you’d model a UI with a tool like Balsamiq, or how we all used to model with entity-relation diagrams and database tools. You lay out your domain objects as resource paths, indicate objects that “live” within other objects, and put in the operations you’d expose on them.
- If you do this in an API designer, you should be seeing the user’s view of the docs forming right beside the spec, giving you more visual confirmation of what you’re building from a consuming developer’s perspective.
- Then you usually want to dive into a particular resource or method and flesh out any data that should be sent in the request and what the responses would look like.
- Very quickly you would realize that you end up repeating yourself, at least if you’re designing a consistent API that does paging similarly across its methods, that does caching similarly, etc. So you can create (or reuse) separate traits for each of these common behaviors, and refer to them back in your API spec. In fact you can include all these traits into libraries and just pull them in when you need them.
- Similarly, you’ll want to have common structures for your resources too. The most common ones are resources that are collections, and resources that are members of collections. Again you can model them as reusable patterns, resource types, and even give those types the appropriate behaviors, using traits for example. Then refer to them in your API spec. You’ll notice that your spec becomes very succinct, even as it’s more expressive and consistent – note all the detail reflected in the generated documentation, for example. That detail will of course be useful in all the ways we mentioned before that leverage the spec.
- One very important area to model is the data moving through your API. Here’s an example that has some simple base types, then bigger types that compose them and primitive types together, and finally some types that inherit from those composed types and extend them. It’s super easy to model data this way, and then again put the models in libraries so they can be reused.
- Now, you can add security to your API spec as well, because clients for example will need to be aware of it in order to access your API. But in terms of SDLC, security policies are often applied later in the pipeline, and often by different people than the developers who built the APIs. For example, ops teams (or automated ops systems) might apply rate limiting policies only in staging and production, and the headers that result could be useful to clients of the API, but of course they were not there in the functional spec the developers created and maintain. So a best practice is to version the functional spec like you version your product – in fact, it is the functional customer-facing aspect of your product. Then policies, at least those that modify the interface, can be expressed as extensions to the API spec in separate extension documents, and these as always can refer to reusable components, such as a common security scheme describing Oauth 2. From an SDLC perspective, the developers develop and publish the functional spec and iterate it when the functionality changes, while operationally the appropriate policies are applied to it per environment; when a policy is applied, it’s expressed as an extension to that API spec; and clients or tools who parse that extension automatically get the extended “effective” API spec they use to consume that API.
- That kind of separation of concerns is really important when you have lots of services in play, so you want your API lifecycle to fit into a sensible SDLC that you can use consistently. Here’s another example, this time in the context of Amazon’s AWS API Gateway and Lambda. The developer who created the service maintains the functional spec of their service, but also needs to configure the API Gateway in front of his service to authenticate traffic and to work properly with his implementation in AWL Lambda. The AWS API Gateway requires a combined configuration file that has both the implementation-specific config values as well as the functional spec. To achieve all of this in a sensible way, it’s best to leave just the functional description in the functional API spec, and separately employ a single reusable library that describes the configuration data needed by any AWS API Gateway. The values of that configuration, of course, are specific for each implemented service, so an overlay file brings them together: it references the library, it points to the functional API spec, and it overlays on that spec the configuration values for the specific implementation of that service, expressed as typed annotation values – with the types being prescribed in that library. That ensures the configuration values adhere structurally to the AWS requirements, it provides the mashed-up configuration file (functional spec + overlaid config data) needed by the API Gateway, and it keeps the functional spec pristine – in fact, by using an overlay rather than an extension, you’re *guaranteed* that the functional aspects of the API (everything describing the HTTP interactions) isn’t altered by overlaying it with the config data.
- And so I think it’s very effective to model API metadata as layers: the functional description is the foundation, and it reuses common behaviors (traits), common structures (resource types). It employs data representations that themselves are modeled as a set of data types. And then it’s extended to express operational policies, or overlaid with more metadata for configuring implementations, for binding to test frameworks, for automatic UI generation, and so on. There are so many things you can bind with this metadata glue…
- The question that often comes up is: what format should I use to express my API spec? Many of us have chosen to use RAML, the RESTful API Modeling Language, which was built specifically to enable the kind of layered modeling I showed here. It’s an open specification itself, and it’s got a healthy, broad ecosystem of tools and vendors behind it, as well as broad adoption, whether it’s large enterprises such as banks or whether it’s cutting-edge consumer companies such as Spotify. Another format is the Open API Initiative’s OpenAPI Spec (OAS), which provides a more flat description format that’s even more broadly adopted and supported. As you may have heard, MuleSoft recently joined the Open API Initiative, and along with other companies is committed to supporting OAS as a common description format everywhere. And to make RAML and OAS truly interoperable, we created the API Modeling Framework, and open sourced it as well. This has lots of benefits, among them bringing the value of API modeling via RAML to the entire Open API ecosystem: we use RAML as the source code description of our APIs (the API specs, that is); we can then leverage any OAS-based tool via AMF. And we can consume any OAS-described API by using AMF’s representation of APIs as a format-agnostic model.
- So the answer to whether you should use RAML or OAS is: yes! Each has their benefit, and now that they’ll be interoperable, you can use either or both, depending on where you’re using the API spec.
- There are lots of other benefits to AMF, which Antonio Garrote described in his talk. You can find more information at https://github.com/raml-org/api-modeling-framework. It’s useful to understand its structure, so you can explore what it can do on your own too. It’s built on an extensible basis of parsers and emitters – these are what map RAML, OAS, and any other formats in the future to and from the common document model within AMF. That layered document model is exposed via an API itself, so you can programmatically retrieve and manipulate the DOM in a format-agnostic way. And that should be the basis of excellent tooling – tooling that can support all these formats automatically, and provide common capabilities such as design, validation, documentation, etc. The document model is then expressed as a logical API-domain model: these are the resources, the methods, the responses, etc. This too is exposed for programmatic access and manipulation as an API, so tools that deal with the service model, for example mocking the service, connecting to the service, validating or governing what it exposes, etc. can be built conveniently at this layer. And finally, extending the API specification to higher levels of abstraction can be done on top of AMF – but I’ll leave that to a future talk. For now, let me note that AMF is not only open source and supports multiple open specs, it’s also built on standard open technologies from the W3C, such as RDF and SHACL, and it is being implemented in multiple languages. So let me close with an invitation to check out AMF as well as RAML and OAS, and let’s collaborate on building out this next level of glue.