I put this presentation together as a result of work with SunGard ATS in building mashups between public and private cloud systems. Presented at CloudExpo East 2010.

2. Cloud Mashups
Michael Heydt – Senior Technology Manager, SCS ATG
Michael.heydt@sungard.com, mike@heydt.org
42spikes.com, @mikeheydt
Building cloud-cloud and cloud-premises mashups
Defining a new application paradigm
Via lessons learned with cloud systems

3. Agenda
History
Future
Idea
Demo
How-To
Closing

4. HISTORY
Mashups up to now

5. Legacy Mashups
Consumer
Platform
Enterprise

6. Consumer Mashups
Mashup software is public via a web site,
All services are “external”,
Likely not to use custom services,
And consumed through service portal or Iframe / portlets
Yahoo Pipes

7. Platform Mashups
Provide a single provider face to multiple cloud services
Facebook + EC2 storage / Salesforce

8. Enterprise Mashups
Mashing of services within an enterprise,
Likely with some web data and custom services,
A single identity system,
Exposure of data through corporate portals
JackBe Presto

9. Guess What?
These are so 2009
They are not “cloud mashups”, and…
I’m not talking about any of these!

10. FUTURE
C2C and C2P Mashups,
A Brave New World

11. New Architectures
Cloud-2-Cloud
Cloud-2-Premises
Mash across cloud providers and your private systems
Enabled with new technologies

12. A New Paradigm
Quickly building applications by
Integrating services from multiple clouds
And data from on-premises systems
Into a new service
Hosted in the cloud and
Used in your application
And scalable beyond anything before
Requires new skills and technologies
This is what I’ll be talking about!

13. ETRO
CLOUD-2-CLOUD MASHUPS
ARE SERVICES AND DATA
ORCHESTRATED ACROSS
MULTIPLE CLOUDS
Services and data orchestrated between multiple clouds
Orchestration services in the cloud
Multiple identities
Data and transforms in the cloud
The reality: There will be multiple cloud platforms

14. ETRO
CLOUD-2-PREMISES MASHUPS
EXTEND CLOUD MASHUPS INTO
YOUR PRIVATE SERVICES AND
DATA
Communications from the cloud to on-premises systems
Be they private cloud systems or non-cloud systems
Your stuff is safe at home but still can be used by the cloud

15. ETRO
TOGETHER
THEY REPRESENT THE NEXT
EVOLUTION OF SOA, PLATFORM
AND APPLICATIONS
Business and consumer collaboration exemplified

16. IDEA
What can I do with this?

17. CONSUMER / PROVIDER ENERGY
COLLABORATION SYSTEMS
Collect price data from grid operators,
Expose uniform data feed via cloud standards (oData, …),
Providers visualize prices with graphs and charts in real-time,
Collect meter data from smart grids and dynamically analyze,
Process contracts from on-premises systems,
Consumers can view usage, pricing and billing info, and
Homes integrate with the smart grid
An ecosystem mashed from multiple subsystems in different
clouds and private systems

18. Concepts
Small independent services wired together on a bus
Multiple cloud providers
Consume public data into cloud data
Scheduled execution of tasks in the cloud
On-premises data access from the cloud
Federated access control
Large scale event processing
Scalable cloud services
Integration of cloud and on-premises data
Data as a Service
This is the stuff you have to know

19. Subsystems
Price collection
Price visualization
Meter event processing
Consumer Portal
Home and smart grid integration
Contracts processing
These are the primary parts of the system

20. Price Collection
Scrape data from energy regulators
Store in cloud storage
Provide access via oData, REST, SOAP
Get the data from elsewhere and make it DaaS

21. Price Visualization
Access price data from rich clients of various forms
Standard data formats allow any clients to consume
Standard “mashlets” allow the same view across platforms
Consume DaaS with mashups and RIA components

22. Meter Data Event Processing
Thousands if not millions of meters sending data,
Every 15 minutes,
Across the smart grid,
Massive flows of real-time events,
How to know what’s going on?
How to handle the volume?
How to store it?
Flex the ISB and CEP

23. Consumer Portal
Scalable web application in the cloud,
Supporting an ever growing user base,
Mashing on-premises customer data and cloud pricing data
Access control from on-premises custom STS
Scalable cloud web app with data tunnels and federated ID

24. Home / Smart Grid Collaboration
Consumers use portal to establish rules for their home
Home collaborates with the grid via cloud agents
Grid delivers commands based upon events
Having the grid charge your car when prices are lowest
Control your home remotely

25. Contracts Processing
Daily and monthly analytics for trading
Usually requiring big iron on site
Many hours or days to complete
Data needs to stay out of the cloud
Save money on the big iron and feel safe about the data

26. DEMO
Show some stuff already will you!

27. Architecture

28. HOW-TO
The Building Blocks

29. Toolbox
AppFabric Service Bus
Microsoft Azure
Amazon EC2
Access Control Service
Port Bridge / On-premises AFSB services

30. Microsoft Azure
.NET Platform as a Service (PaaS)
Scalable Web and Worker Roles
Runs .NET applications in the cloud
But also provides capabilities for PHP, Java, and more
Provides Internet Service Bus and Access Control Service
These are unique to Azure and Critical for C2C/P

31. Amazon EC2
Infrastructure as a Service (IaaS)
Essentially virtual machine hosting in the cloud
Many pre-built virtual machines to pick from (AMIs), or
Roll your own
Scalable, Durable, Totally Customizable
You are likely using it but don’t even know it

32. AppFabric Service Bus
Internet scale messaging bus
Enables inter/intra cloud, cloud-premises
Named endpoints for services to rendezvous
Negotiates end-to-end network traversals
Optimizes network traversal
Tightly integrated with the Access Control Service
Synchronous, fire and forget, and pub/sub multicast
Access via WCF or REST
This is the backbone of C2C/P systems

33. AFSB: NetEventRelay

34. AFSB: NetTcpRelay

35. Access Control Service
Abstracts orchestration of auth[*]
Claims based model
Controls access to the ISB
Performs claims mapping between providers
Your apps and services can integrate
Can integrate with your auth services
You can integrated custom auth services via WIF
Ensures all services on the ISB are only used by who you allow

36. Access Control Service

37. Premises #1 AFSB Service
A WCF service that connects to the AFSB
API exposed to cloud applications
Requests securely routed to on-premises service
Service can interoperate with on-premises system directly
Makes premises systems accessible to your cloud app

38. Premises #2
Port Bridge
Tunneling of level 3 traffic
To/from cloud to non-cloud / non-service systems
Port Agent in cloud, Port Bridge on premises
Agent and Bridge use AFSB for connectivity and tunneling
Because it uses ISB, it also uses ACS
Cloud enables apps that don’t talk cloud
Makes premises systems accessible to your cloud app

39. Deep-dive
If we have time

40. Closing
Wrap up and where is this going?

41. Some Comments
Plumbing is in place for c2c/p mashups
Build small services
Glue them together with AFSB
Let ACS handle the security
Focus on dynamic code allocation
Expose data through oData services
Better tools are still needed

42. Future Tools
Declarative and graphical designers for applications
Dynamic code storage and allocation to cloud services
Automatic deployment of services and wiring
Evolutionary upgrade of services
Semantics data repositories
Adaptive systems that auto-wire based on usage patterns
Intentional systems that auto build based upon needs
This is going to be great!

43. REFERENCES
WWW.MICROSOFT.COM/WINDOWSAZURE
AWS.AMAZON.COM
ODATA.ORG
42SPIKES.COM (my blog)

44. PRINCIPLES
Architecture for C2C/P mashups

45. The Principles
Reuse
Functional Services
Data As A Service
Identity
Security
Interconnection
Discoverability
Wiring
Mashup
Mashlet
Transformation
Semantics

46. Reuse
Fundamentally all mashups are reused
Either in user interfaces,
Other services, or
In other mashups
Nothing lives alone

47. Functional Services
Provide an operation,
An interaction with another application,
A retrieval of data from another system, or
A change of state
SOA Classic

48. Data as a Service (DaaS)
Mashups need data
UI’s need data
Services can expose data
Data itself is a service
No need to code!

49. Identity
Multiple entities each requiring different credentials
Need Identify Federation
Providing Identity Management and mapping
Used to implement security
You or your system will have many

50. Security
Use of Identity
To authenticate a user or service
Against another service
To control access to services, and
What can be done with the service
Information wants to be free, but you may not want it to

51. Interconnection
Services need to talk to each other
Protocols need to be bridged
Tunnels need to be created
Building the nervous system of the mashup
This is the Internet Service Bus

52. Wiring
The definition of the flow of the data through the mashup
From one service to another
Utilizing an interconnection
These are the rules of how the data flows, not how it is moved

53. Transformation
Services often disagree on the structure of data
This is a service that changes the format of data
Using specific rules
The physical transformation of data according to semantics

54. Semantics
A description of the meaning of data
Not the content of data
Often used to understand data in a Mashlet, or
Change data for another service
The rules of the vocabularies of data between the services

55. PATTERNS
Things you will do all the time

56. The Patterns
Classic Mashup
Internet Service Bus
Dynamic Service
Deploy Service
Identity Federation
Premises Tunnel
Wiring
Mashup
Mashlet
Data as a Service
Hub
Plug Into Hub

57. Classic Mashup
Ingest data from multiple feeds
Merge data
Transform data
Filter data
Expose results
Assumes all web services

58. Internet Service Bus
Backbone for cloud-cloud and cloud-premises composition,
Providing discovery through standard naming techniques, and
Interoperation through dynamic routing / binding of services,
Provides protocol bridging across heterogeneous clouds, and
Ability for two services to attach without knowing their actual
locations or protocols
The nervous system

59. Dynamic Service
A service that is provisioned
On a schedule, or
On demand, and
Is discoverable and connected automatically to mashups,
To provide scalability and availability, which
May be autonomous, or
Accessed through it’s own web service, or
Via an endpoint on the ISB
Your mercenaries

60. Deploy Service
Dynamic spin up of a service in a cloud provider,
Connecting the service to and endpoint on the ISB, and
Configuration of the ACS and IF
Run it only when you need it

61. Identity Federation
A service that provides unified mapping of credentials from
multiple services and clouds,
To provide seamless access controls across all services
Presenting yourself in the light of another

62. Premises Tunnel
Establishes protocol level tunnel for native protocols,
From a cloud service,
To an on-premises service,
Via the Internet Service Bus,
Forming a secure means on integration cloud and premises
Commonly using the port bridge
Safely expose private data, systems and services to the cloud

63. Mashup
An ordered sequence of wires
That pass data into services
Using the interconnects
Retrieving results from the services
Transforms data according to semantics
Flows the results into other services
And exposing the result as a new service
Secretly (or no so) a business process

64. Mashlet
Reads the results of a mashup via its address
Applying a visualization
May be exposed itself as a web construct, or
A platform specific user control
And knows how to display one or more types of data
Somewhere, sometimes, you need to see the results

65. Data As A Service
Data being directly exposed as a service
Without the ceremony (code) of normal services
Information wants to be free (within your permission)

66. Hub
Service integration through Internet Service Bus, with
Automatic data transformation,
Automatic wiring and interconnection,
Dynamic deployment of services,
Abstraction of cloud and premises systems,
Automated orchestration of services,
Identify federation, and
Access control
No one or thing lives alone anymore

67. Plug Into Hub
Configuration of a deployed service,
To communicated with a Hub,
Attaching the service to the hub’s orchestration,
Providing data transforms and
Access control and identify mapping to the hub
I want to play with others like me

68. Transform!
Change data from one form to another
Using a specific transform, or
Semantic information
The Optimus Prime