Talk presented by Aarón Fas & Andrés Viedma at the JBcnConf 2015. 'Microservices' is one of the most popular buzzwords in the industry now, but are they really a step forward? Or they might be more a problem than a solution? When are they really helpful? How should they be addressed? What challenges will we face if we decide to implement a microservices based architecture? One year ago, Tuenti moved from a monolithic PHP backend to a Java + PHP microservices architecture. In this talk, we'll share our experiences so far: how we addressed the change, how we implemented it, why we think it's been valuable for us (and how is that related to the company culture), why it might not be a good idea for your company / application and, mostly, what lessons we have learned from this experience.

1. .Experiences with
Microservices at
Aarón Fas
Andrés Viedma

2. Microservices?
I know what you’re
probably thinking...

6. Who did you say these guys are?
Andrés Viedma
@andres_viedma
Aarón Fas
@aaronfc
Java
dinosaur
Useless gadgets
buyer

7. About

8. From Social Network...

9. From Social Network...

10. To Mobile Operator
(full MVNO)

11. The PHP
Monolith
One single source
repository
PHP???

12. Do you need a
release?
Take a ticket
and wait...

13. .Microservices

14. Microservices… again… (and take a shot)
❖ Distributed, independently deployable
components
❖ Well defined interfaces
❖ Simple communication interface (HTTP?)
❖ Each service has its own DB
❖ Each service has its own source repository

15. Microservices… again… (and take a shot)
❖ Distributed, independently deployable
components
❖ Well defined interfaces
❖ Simple communication interface (HTTP?)
❖ Each service has its own DB
❖ Each service has its own source repository
THAT IS
SOA !!!

16. Microservices… again… (and take a shot)
❖ Distributed, independently deployable
components
❖ Well defined interfaces
❖ Simple communication interface (HTTP?)
❖ Each service has its own DB
❖ Each service has its own source repository
Is that important enough to
deserve a new name???

17. Mixing technologies
❖ Allows using different languages
❖ Different platform versions
❖ Incremental technology changes / evolution

18. Separation of responsibilities
❖ Forces separation of responsibilities
➢ Subsystems with well defined facades
➢ Different source repositories

19. Separation of responsibilities
❖ Forces separation of responsibilities
➢ Subsystems with well defined facades
➢ Different source repositories
YOU DON’T NEED
MICROSERVICES!
USE
JARS !!!

20. Continuous deployment
«Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.»
«The best architectures, requirements, and designs
emerge from self-organizing teams.»
-- Principles of the Agile Manifesto

21. Continuous deployment
«Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.»
«The best architectures, requirements, and designs
emerge from self-organizing teams.»
-- Principles of the Agile Manifesto1 Service => 1 Team?
Better than Continuous delivery!:
Continuous deployment
Team responsible of the
deployments?

22. Beware! High costs
❖ No transactions!
➢ Distributed tx?
❖ Requires a much more complex infrastructure
❖ Difficult integration testing

23. For us: Seemed like a good idea
❖ We have small self-organized teams =>
Continuous deployment is a reality
❖ We wanted Java, we had PHP
❖ Strong SRE / DevOps team
❖ Our software was intended mainly to access
3rd parties => transactions not possible anyway

24. .Communications
protocol

25. Existing libraries
❖ No PHP implementation
➢ Avro, Etch, Netflix stack
❖ Only serialization
➢ Protocol buffers
❖ Didn’t exist or were too new
➢ Cap’n Proto, gRPC
❖ Thrift?
➢ Good option, but a lot of PHP boilerplate

26. TService
❖ Own abstraction layer - RPC based
❖ Basic implementation: JSON-RPC
❖ Interface Definition Language (IDL)
❖ Generates Java / PHP / Erlang:
➢ Interchange objects
➢ Client
➢ Server stub

27. TService IDL
/**
* Manages the transfer of balance between subscriptions.
* @version 1
*/
interface BalanceTransferService {
/** Transfer money from one subscription to another one. */
String transfer(Donation donation) throws NoSuchSubscriptionException;
(...)
}
/** Donation between two subscriptions. */
class Donation {
/** Id of the donor */
long from;
/** Amount of money to transfer */
int amount;
(...)
}
class NoSuchSubscriptionException extends Exception {
int code = 100;
}
Java???

28. TService Versioning
Interface v1
Service
Client 1
Client 2
(compatible changes)
● New methods
● New fields in objects
● New parameters in
methods
● Delete methods /
parameters / fields

29. TService Versioning
Interface v1
Service
Interface v2
Client 1
Client 2
(compatible changes)

30. TService Versioning
Interface v1
Service
Interface v2
Client 1
Client 2
(compatible changes)

31. .Java Platform

32. Technology stack

33. XConfig
❖ Own configuration system
❖ YAML files based
❖ Git repository
❖ Overriding system: by env, common / service
❖ Hot reloading
➢ Everything adjusts to changes: even DB pools!
➢ No restart required

34. Async jobs
TService request
processing Enqueue
job
Queued jobs
Executor thread pool

35. Async jobs
TService request
processing Enqueue
job
Queued jobs
Executor thread pool
Cron jobs
Cron jobs programming in
config

36. Feature disabling
❖ Activation / deactivation of features by config
➢ Is the new development risky?
➢ Is the rest of services / environment ready for the
change?
❖ Partial activation of a feature for a % of users
➢ Incremental activation of an optional risky change
➢ A / B tests

37. Integration tests
❖ Custom JUnit runner
➢ Bootstraps the platform
➢ Cleans / restarts the local database
➢ Allows the use of @Inject in tests
➢ Allows overriding in dependency injection => inject
mocks of the other services
❖ Uses special, “development” XConfig repo

38. .Monitoring

39. Monitoring, a priority
❖ What is happening or has happened?
➢ Logs
➢ Metrics
➢ Alarms
❖ Distributed architectures are much more
difficult to track

40. And basically because...

41. .Let’s talk about logs

42. Logging
❖ Logging library in Java?
➢ Log4j
❖ We needed full details
➢ Filters to expand/simplify information logged
➢ Multiple appenders logged into distinct storages

43. ❖ Overview of appenders
Logging
log.info(...); Logger
MySQL Appender
LogStash Appender
Hadoop Appender

44. ❖ Following call’s path (TService calls logging)
Logging
ServiceA ServiceB ServiceC
GlobalID = 100
RequestID = 1
GlobalID = 100
RequestID = 2
GlobalID = 100
RequestID = 3
Benefits
● Locate in/out for calls
● Get all interactions

45. Logging
❖ Kibana dashboard
What does it look like?

46. Change query

47. Customize filters

48. Log types by color

49. Full log details

50. .Let’s talk about metrics

51. Metrics
❖ We graphs
➢ As easy as possible to track new metrics
❖ Do not reinvent the wheel
➢ Already using StatsD/Graphite on PHP side
❖ What are we tracking?
➢ Basic monitoring metrics added by the platform
➢ Metrics from Tomcat JMX
➢ Metrics related to business

52. Metrics
❖ Multiple graphs dashboards tested
➢ Default graphite one
➢ Grafana

53. Graphite’s is a little ugly...

54. Grafana is prettier

55. Layout customized

56. Much better UI to create graphs

57. .Let’s talk about alarms

58. Alarms
❖ Graphs are ok, but we don’t have people 24x7
staring at them.
➢ We need notifications
❖ Different things to monitor
➢ SQL queries
➢ Graphite metrics
➢ HTTP requests
➢ ...

59. Alarms
❖ Created our own alarms system
➢ Multiple data sources and easily extensible
➢ Quick edition of conditions
➢ Observers for alarms
❖ We ended up using mainly
➢ MySQL and Graphite data sources
➢ Java Expression Language on config checkers
➢ Email notifications

60. … and then, we found Cabot
Separated by service

61. Cabot overview
Multiple integrations

62. Cabot overview
Service status overview

63. Cabot overview
Graphite checks

64. Cabot overview (Creating new check)
Set graphite metric

65. Cabot overview (Creating new check)
Check data

66. Cabot overview (Creating new check)
Set check type/value

67. Cabot overview (Creating new check)
Set importance

68. Cabot
❖ Benefits of using Cabot
➢ Friendlier UI than config files
➢ No dependency on the service monitored
➢ Opensource and many integrations

69. Alarms
❖ Where are we heading now?
➢ Moving now most Graphite alarms to Cabot
➢ Replacing thresholds with dynamic expectations (Holt
Winters)
❖ It is still the main alarms platform being used

70. .That’s all about monitoring

71. .Some Lessons
learned

73. GO
ASYNC!!!

74. Don’t get blocked for too long
❖ Concurrent requests: don’t wait for free threads
➢ Own Rate limit mechanism
➢ Tune container thread pool size
➢ Tune database pool (and other possible blocking pools)
❖ Tune clients timeout
➢ It may depend on called service / operation
➢ It may depend on the caller

75. Asynchronous logging
log.info(...)
Appender
MySQL
Appender
Logstash
Appender
Hadoop
Logger

76. Asynchronous logging
log.info(...)
When the ring buffer is full…
WAIT!
Appender
MySQL
Appender
Logstash
Appender
Hadoop
Logger
Ring buffer
Async Logger
Not configurable!

77. Asynchronous logging
log.info(...)
When the ring buffer is full…
WAIT!
Appender
MySQL
Appender
Logstash
Appender
Hadoop
Logger
Ring buffer
Async Logger
Async
Appender
Async
Appender
Async
Appender
Not configurable!
When async appender full,
messages are discarded

78. Asynchronous operations
❖ Getters
➢ Make them fast (sacrifice consistency)
➢ Cache
➢ Use default values
❖ Setters
➢ No operation result
➢ Wait for a notification of operation finished
➢ Query the status of the change

79. Message queues
❖ Operation queues
➢ Retry system
➢ Persistent queues
❖ Publish / subscribe model (pub/sub)
➢ Event driven
➢ Reactive programming

80. Circuit breaker
❖ From the client, consider the status of the
service
➢ Previous calls
➢ Health checks
❖ If it’s degraded, don’t call it (close the circuit)
➢ Return a default response
➢ Enqueue the operation for later retry
➢ Throw an error

81. .Do It Yourself

82. Many implementations available
❖ Communication layer
➢ gRPC, Cap’n proto, Thrift…
➢ REST, JSON…
❖ Services platform
➢ Spring boot, Dropwizard, Spark, Ninja, Jodd…
❖ Netflix stack
➢ Hystrix, Ribbon…

83. Make your own combination!
(it can’t be so
difficult…)

84. Aarón Fas
@aaronfc
Andrés Viedma
@andres_viedma
.Thanks!
Questions?