APIs have finally crossed over to the world of enterprise software, data analytics and application integration. Spearheaded by Amazon, propagated by internet startups and now adopted by the largest of businesses including Wall Street top firm Goldman Sachs - the APIs are here to stay. In this presentation we are linking all the facts and examine the opportunities stemming from Resource Oriented Architecture - a holistic approach to API implementation in large organizations.
2. Who is this guy?
Software developer
First lines of code around 1987 in Atari Basic
Professionally since 1997 (Java, Python, SQL)
Telecom and finance
Founder and CEO
vt.enterprise – software consulting business
SlashDB – automatic API gateway to data
Speaker at technology conferences
www.SlashDB.com 2
3. SlashDB’s story: built on experience
Conceived out of 20 years of experience
Citi, Merrill Lynch, Commerzbank, hedge funds
Market data distribution, master data management
Trading and analytical systems development
Industry agnostic API gateway to data
Delivers databases to web and mobile applications
Provides Enterprise and Web interoperability
Designed to anticipate changes
Low risk, incremental and easy API build-out
Highly Scalable
www.SlashDB.com 3
4. Agenda
From Systems of Record to Systems of Engagement
Money talks – the business case for APIs in
enterprise systems
What is Resource Oriented Architecture (“ROA”)
How it compares to ETL and Data Warehousing
How it compares to Service Oriented
ROA application to data science
Searchable hypermedia API
SlashDB – API automation for people with deadlines
www.SlashDB.com 4
6. 6
Why is it that we know more about what our
high school girlfriend had for dinner than
what is going on within our organization?
– Tony Zingale, CEO, Jive Software
(The image is for illustration purposes only. Jessica was not my high school girlfriend.)
www.SlashDB.com
7. Emergence of Systems of
Engagement
Phrase coined in 2011 by Geoffrey Moore
Managing Director, TCG Advisors
Systems of Engagement and the Future of Enterprise IT
http://www.aiim.org/~/media/Files/AIIM%20White%20Papers/Systems-of-Engagement-Future-of-
Enterprise-IT.ashx
Peer to peer communication and collaboration
Mobile Apps, IM, Twitter, Facebook, etc.
Compelling user interfaces, quick and intuitive
But, not very useful for business unless they can
leverage data from Systems of Record
ERP, CRM, HR, Orders, Research,
Accounting, Master Data
www.SlashDB.com 7
8. Transition to Systems of
Engagement
Cloud technologies are key enabler
Virtualization
Progress in telecommunication
HTTP as a standard transport protocol
Location-aware applications
Agile software development methodology
www.SlashDB.com 8
9. Business Applications
9
Global Workforce Mobility
Sales, service
Digital Marketing
Data driven documents and interactive presentations
Enterprise Resource Planning
Data APIs for partners and clients
HR and Administration
Time card, office resource optimization
Controlling and Risk Management
Transaction surveillance, reporting, compliance
R&D
Data science, quantitative analysis
Financial and Accounting?
Early cash flow prediction from CRM data
www.SlashDB.com
10. How to bridge the divide?
10www.SlashDB.com
Systems of Engagement
REST/HTTP API
Enterprise Systems of Record
SQL, Client/Server
13. 2002 at Amazon
• All teams will henceforth expose their data and functionality
through service interfaces.
• Teams must communicate with each other through these
interfaces.
• There will be no other form of inter-process communication
allowed: no direct linking, no direct reads of another team’s data
store, no shared-memory model, no back-doors whatsoever. The
only communication allowed is via service interface calls over the
network.
• It doesn’t matter what technology they use.
• All service interfaces, without exception, must be designed from
the ground up to be externalizable. That is to say, the team must
plan and design to be able to expose the interface to developers
in the outside world. No exceptions.
• Thank you; have a nice day!
• Anyone who doesn’t do this will be fired.
13
“
”- Jeff Bezos
www.SlashDB.com
19. Store it all in one place?
• It was hard with just
on-premises
systems
• Illusory idea with
today’s Cloud apps
• Try it with your
contact list for
starters…
19www.SlashDB.com
20. What is Resource Oriented
Architecture (“ROA”)
• “Style of software architecture and
programming paradigm for designing and
developing software in the form of
resources with RESTful interfaces.”
– Wikipedia
• Uniform data access layer to all data
assets in their unobstructed form for
reading and writing in various
representations. – my take
20www.SlashDB.com
21. What year was this wireframe
drawn?
21
"The programmer
used GET
commands to
navigate between
related records."
www.SlashDB.com
22. 22
Charles Bachman
– Designed and developed first database
management system
– Graph rather than hierarchy
– 1973 honored with ACM’s Turing Award
– 2014 Nat’l Medal of Technology and Innovation
1962
https://www.slashdb.com/2015/10/09/flashback-friday-charles-bachman/
23. • Single access point, but without copying data
• Self-service reporting, data feeds or to integrate systems
API gateway to data
23www.SlashDB.com
24. Database content as
HTTP resources
24
http://demo.slashdb.com/db/Chinook/Customer/CustomerId/1.html
Service location
• On the intranet, or
• In the cloud
Database
name.
Supported
RDBMS:
• MS-SQL,
• Oracle
• MySQL
• PostgreSQL,
and more
Table to query Field to filter and
value to lookup:
• Text
• Number
• Date
Data format
• JSON
• XML
• CSV
• HTML
Combine
several
/db automatically makes hyperlinks directly to data
Related records are hyperlinked thus search engine ready
Filtering, drill-down, slices feel natural, URLs look nice
Custom queries also possible (SQL Pass-thru)
www.SlashDB.com
25. ETL/Data Warehousing
25
Analytical Systems
• Data duplication
• Stale data
• Brittle overnight
feeds
• Central bottleneck
• Does not scale out
• Not easily
accessible nor
searchable
26. Well implemented ROA beats
ETL/DW handily
www.SlashDB.com
On-demand access
Adapts to changes
Works with everything,
accessible by non-IT
Search engine ready
Encourages data sharing and
collaboration
Instantly web and mobile
compatible
Scales out and up
Easily cacheable
Overnight copying
Manual redesign or code
Specialized interfaces,
programmer’s help needed
No support for search
Reinforces data siloes, causes
data duplication
Requires development of a web
service
Data warehouses only scale up
Harder to cache
Resource Oriented Architecture Traditional ETL and Data Warehouse
26
27. Service Oriented vs.
Resource Oriented
Service Oriented
• Endpoint Represents
Action
• Transaction, Unit of Work
• Message
• API controlled by
functional design
• Harder to adapt and scale
beyond “enterprise”
• Harder to deprecate
functionality
Resource Oriented
• Endpoint Represents
State
• Addressable Resource
• Update to Resource
• API automatically evolves
with data
• Harder to model into
complex transactions
• Clients must be resilient
to change
27www.SlashDB.com
28. Best practices for ROA
Don’t forget about “R” in
REST
JSON isn’t the only data format
URL should be easy to
understand
Avoid inventing a mini query
language
Preserve and enhance
underlying data source
authorization mechanisms
Resources should be easy to
discover
Hypermedia API
Ideally every resource
address should allow
reading and writing
Avoid URL query string to
address data
28www.SlashDB.com
29. Easily accessible, but
protected
29
Authentication
Basic HTTP Authentication (binhexed username/password)
API key (appended to URL or in HTTP headers)
Possible integrations with Single Sign-on and API management systems
Authorization
Declaratively state authorizations to databases and queries
Extends internal database mechanisms (GRANT statement)
Admin functions permissioning (i.e. add users, databases)
Data encryption and obfuscation
Use SSL to encrypt all traffic
Use database views to hide or shuffle personal data
www.SlashDB.com
31. dbSlashDB Web Service
automatic data API & cache,
search engine friendly
Application Integration
and Data Science
use any programming language
Reports, Dashboards
and Mobile Apps
deliver now, anticipate future
Unobstructed Data Sharing
standard formats, HTTP delivery
Databases
stores of record
for mission critical data
www.SlashDB.com
Custom Storage
32. Data science is a process
32
Data acquisition, storage, discovery and
mining, statistical learning, machine learning,
predictive analytics, quantitative modeling,
visualization and reporting, decision making
Competency gaps at every step
Varying data formats
Data integrity and lineage often lost
www.SlashDB.com
33. 2015, Global Bank
Upwards 50% of
my time goes into
data reconciliation
efforts.
“ The biggest pain is
sharing data
between Python,
R, etc.
The problem is -
there should be
one specified entry
point for data.
Consistency of
column names and
possible values
between different
versions of the
data.
There are a lot of
holes in the data
process. I think the
#1 priority would
be creating a good
schema.
”
Finding what you
need in this zoo.
(…) Currently this
is done by talking
to people!
33www.SlashDB.com
34. Searchable API
34
Users know what they need, but may not
know where to find it
True hypermedia API should contain hyperlinks
to related resources
Search engine crawl/index is trivial when all
resources are hyperlinked
Try it yourself at:
http://demo.slashdb.com/search.html
(i.e. search for: “customers from Brazil”)
www.SlashDB.com
35. Resource Oriented API
solves many of the issues
Single access point that’s easy to work with
Combines the best features of plain files
(simplicity) and databases (data integrity)
Has authentication, authorization and
encryption
Pragmatic data access for people and
programs
Search engine ready
35www.SlashDB.com
37. Resource Oriented API is a
sensible investment
Multiply the return on your data assets
(the other ROA)
Avoid pitfalls of file-based data sharing
Avoid dangers of direct database access
Avoid opaqueness of ESB, RPC, RMI, SOAP,
CORBA, etc., etc.
Attract top young developers
(hint: they want to work on cool stuff, and they don’t know databases)
37www.SlashDB.com
38. Name at least one of the
two modes for accessing
data in SlashDB?
38www.SlashDB.com
39. Powerful features suit every purpose
Data Discovery
Intuitive URL scheme
No SQL coding
Preserves complex relations
Configurable traversal depth
SQL Pass-Thru
Supports any valid SQL or stored
procedure
Maps query params to URL
Convenient data formats
XML, CSV, JSON, HTML
39
End user friendly
No code to use in Excel
Easy to use with Matlab,
Mathematica, R, Python, curl
Scalability
Stateless design
Caching
Available as Virtual Appliance
Supports leading RDBMS
www.SlashDB.com
40. Summary
Businesses, which use APIs tend to outperform
Wrapping Systems of Record in APIs enables
transition to Systems of Engagement
Resource Oriented Architecture
Complementary to SOA
Better than ETL/DW
Great way to roll out APIs holistically
Try SlashDB for your API and ROA needs
www.SlashDB.com 40
42. Credits & References
• S&P Churn 2002-2012
“Creative Destruction Whips through Corporate America”
by Richard Foster, Innosight
http://www.innosight.com/innovation-resources/strategy-innovation/upload/creative-destruction-whips-through-corporate-america_final2015.pdf
• 2002 at Amazon
“The Secret to Amazon’s Success Internal APIs”
by Kin Lane, API Evangelist
http://apievangelist.com/2012/01/12/the-secret-to-amazons-success-internal-apis/
• Flattening the Competition
Google Finance, chart prepared by V. Olex
https://www.google.com/finance?q=amzn
• 2015 Global IT Spending
“Want money for that new project? Then it's time to go on a moose hunt”
by Steve Ranger, ZDNet
http://www.zdnet.com/article/want-money-for-that-new-project-then-its-time-to-go-on-a-moose-hunt/
• SaaS Revenue Projections
“Enterprise software spend to reach $620 billion in 2015: Forrester”
by Natalie Gagliordi, ZDNet
http://www.zdnet.com/article/enterprise-software-spend-to-reach-620-billion-in-2015-forrester/
• APIs at Goldman Sachs
Business Insider http://www.businessinsider.com/goldman-sachs-wants-to-become-the-google-of-wall-street-2017-4
CIO Magazine http://www.cio.com/article/3138459/apis/at-goldman-sachs-apis-point-the-way-toward-a-platform-future.html
• What is Resource Oriented Architecture
Wikipedia
http://en.wikipedia.org/wiki/Resource_oriented_architecture
• Data & Analytics: Benefits & Challenges
“5 Insights & Predictions On Disruptive Tech From KPMG's 2015 Global Innovation Survey”
by Louis Columbus
http://www.forbes.com/sites/louiscolumbus/2015/11/08/5-insights-predictions-on-disruptive-tech-from-kpmgs-2015-global-innovation-survey/
• Other graphics
Still frame from the movie “Back to the Future”
http://2.bp.blogspot.com/-AnHNMztbj8o/TpXyXL2CmaI/AAAAAAAADGg/QZzORg_4l9o/s1600/outatime.jpg
• Photographs of D. Trump, Flicker and public domain sources
• Logos and other trademarks are the property of their respective owners; used here for illustration purposes only, no association or endorsement implied.
42www.SlashDB.com
Notes de l'éditeur
The term boils down to businesses embarking on a transition from enterprise systems designed around centralized records to systems, which are more decentralized and encourage peer interactions. Mobile apps, Facebook, Twitter, instant messaging.
Geoffrey Moore coined the term is 2011 paper published by Association for Information and Image Management.
Trouble is though, no matter how compelling and easy to use those apps are they are of little value to enterprise employees unless they can hook up to the old Systems of Record.
Enterprise evolution to systems of engagement will only succeed if it can leverage and add value to existing systems of record.
As if it weren't hard enough find your way around data within an organization (whoever spend two weeks chasing after a DBA knows what I am talking about)
Another difficult aspect of those systems of engagement is that they often are outside of the corporate firewall, on employee's mobile devices or third party websites. How do we link the two worlds?
VPN can be brittle especially on mobile
Periodic syncing, that’s like going back PDA
Web services are expensive to develop
Creative Destruction Whips Through Corporate America. Lifespan of company in S&P 500
1958 – 61 years, now 18 years.
No company is entitled to its business model.
I order to achieve this level of interaction we are proposing a new way to think about data integration.
Beside (or instead of) overnight data feeds into data warehouses we need a light, on-demand facade, which abstracts databases, tables and records into online resources.
Those resources have to accessible to both software engineers and domain knowledge workers (data scientists, business intelligence, quantitative analysts, salespeople). We call this a Resource Oriented Architecture.
Our solution has been to automatically hyperlink all the data in order to abstract it as online resources, similar to how web pages are built except built out of systems of record.
What you are seeing is an actual URL from /db. It is easy to understand what it represents. The host name could be local to your intranet or remote in the cloud. What follows is /db followed by a database name. After that we see a table name followed by a pair of field and value, which constitute a filter on the table. You can have more than one of these. Lastly there is desired a data format. It is also worth pointing out that related records are linked and therefore can be crawled by a search engine. For brevity's sake we cannot show all URL options on this slide but I will show more in the demo. Where automatic URL are not sufficient there also is an option to use custom SQL queries mapped to a URL.
Traditional data warehousing and ETL cannot really cope with the issue because ultimately they just create copies of data. Stores of record change over time, feeds need to be regularly maintained, monitored.
The innovation that has taken place in databases however has actually gone back to the old albeit improved idea of key-value store (think dbm developed in 1979 by AT&T) and gave us the NoSQL movement. Reportedly infinitely scalable but even harder to weave into the information flow in enterprise. Innovation tends to go for the bigger and faster, which is great but not always the most pragmatic direction. Meanwhile, the powerful sharing (engaging) nature of the web has been facilitated by a simple idea of abstracting information resources with URLs transmitted over relatively low-performance hypertext protocol.