Discover how Mydbops achieved an impressive 80% cost savings and ensured uninterrupted availability through a transformative MySQL database case study. Join Vinoth Kanna RS, Co-Founder of Mydbops, as he shares insights into optimizing infrastructure, enhancing observability, and navigating critical technology decisions. Learn from real-world challenges, innovative solutions, and valuable takeaways for your own database management endeavors.
MySQL Transformation Case Study: 80% Cost Savings & Uninterrupted Availability - Mydbops Meetup 14
1. 07 October 2023
Mydbops Open Source Database Meetup 14
Vinoth Kanna RS
Co-Founder, Mydbops
MySQL Database Transformation Case Study
2. Interested in Open Source technologies
▪
MySQL, MongoDB Certified Professional
▪
Active MySQL Community Contributor
▪
Tech Speaker/Blogger
▪
Co-Founded, Mydbops IT Solutions, 2016
▪
Vinoth Kanna RS
About Me
6. Case Brief
Audit DB infrastructure
▪
Reduce TCO without impacting Performance, Availability
▪
Plan for version upgrade
▪
Decouple application and database
▪
Propose and Implement best practices (HA, Backups, etc)
▪
7. Case Brief - MySQL DB Infra
DC 1
PDC Source PDC R 01 PDC R 02 PDC R 03 PDC R 700
DC 2
SDC Relay SDC R1 SDC R2 SDC R3 SDC R 700
DC 7
DC7 Relay DC7 R1 DC7 R2 DC7 R3 DC7 R 700
PDC - Primary DC SDC - Secondary DC R - Replica App + DB Hosted On All Servers
MySQL Version - 5.6
8. Observability
Generic Database Monitoring
Provide database stats
▪
Informative
▪
Organised based on component
or technology
▪
Difficulties in navigation
▪
Battlecat
Built to suit
▪
Decisive
▪
Translate Industry expertise to a
product that can be used to
manage databases at scale
▪
OS Exporters + Custom Exporters
▪
10. Observability
Average of 150 connections established from each application
▪
1,00,000 - 1,50,000 connections / DC
▪
Critical queries, Approx 10% returned data around 64MB of data / Query
▪
Avg. Active threads and QPS metrics were collected too.
▪
More you consolidate, More the concurrency, Lesser the time to react during issue
▪
Multiple applications and Additional databases on replica
▪
17. Choice Of Technology - MySQL HA Solutions
RTO (Recovery Time Objective)
RPO (Recovery Point Objective)
Failure Types:
How long does it take to recover from a single failure
▪
How much data can be lost when a failure occurs
▪
High Availability: Single Server Failure, Network Partition
▪
Disaster Recovery: Full Region/Network Failure
▪
Human Errors
▪
18. Choice Of Technology - Group Replication Cluster
MySQL's Native HA Solution
Synchronous Replication
Group replication has turned more stable, improved well with 8.0 release
Tuneable Consistency
Scale out reads effectively
Configurable Writer Modes
Built on robust replication framework uses binlogs with GTID for replication
19. Choice Of Technology - MySQL Compatible Load Balancers
MySQL Router
▪
Maxscale
▪
ProxySQL
▪
HA Proxy
▪
TCP / DNS based load balancing
▪
20. Choice Of Technology - Database Aware Load Balancers
Schema Based Routing
SQL Regex Based Routing
User Based Routing
Load Balancing
Replication, Group Replication,
Galera Replication Awareness
Connection Pooling
Audit Logging
Native Read Write Split
Query Caching
Query Mirroring
21. Choice Of Technology - ProxySQL
Ability to scale better in multi core systems
Clustering, Config management can be simplified
Measurable performance (qps or connections) / vpcu
Audit Logging, JSON Format, Introduced On v 2.0.6 Synchronous Replication
Dynamic / Runtime configuration changes, Provides granular control
Routing and Query statistics exposed are extensive and useful
25. Callenges - MySQL 5.6 with GTID OFF
Source
MySQL 8.0.34
MySQL 5.6
GTID: OFF
MySQL 5.7
GTID: OFF GTID: ON
assign_gtids_to_anonymous
_transactions=LOCAL;
Node 1
Node 2 Node 3
MySQL
8.0.34
GR Cluster
Actual Implementation
26. Callenges - Auto Local GTID Creation
Master MySQL 8.0.34
GTID: OFF GTID: ON
Log Slave updates: ON
change replication source to
. . . . . . . . . . . . . . . . . . . . .
assign_gtids_to_anonymous_
transactions=LOCAL;
#210212 23:00:42 server id 320
end_log_pos 399 . . . . . Xid = 1245
use `test`/*!*/;
truncate table event_logs
/*!*/;
# at 399
SET @@SESSION.GTID_NEXT= 'bb71f03a-
1ab9-42ba-b3a2-1803de73944f:1224'/*!*/;
#210212 23:00:42 server id 320 Xid = 1245
use `test`/*!*/;
truncate table event_logs
/*!*/;
# at 399
Replicate To
GR Cluster
27. Implementation - PDC and SDC
Node 1
Node 2 Node 3
MySQL
8.0.34
GR Cluster
R1 R2 R3
R4 R5
ProxySQL
Cluster
From DNS LB
Weighted,
User Based
Routing
Writes
Consistent
Reads
App 1 Read
App 2 Read
App 3 Read
28. Implementation - Infra Savings
Before
~ 3500 servers Across 7 DC's
▪
EOL MySQL Version
▪
Unorganised data platform
▪
No Defined HA
▪
Operational complexity
▪
After
10 - 15 Servers / DC
▪
Est. Saving of 1 Milliion $
▪
Simplified Data Platform
▪
MySQL 8 Upgrade
▪
Measurable RTO, RPO
▪
29. Key Takeaways
Audit regularly
Scale with technology upgrades
Ensure DB Observability
Test Failover Strategies, Backups
System failure is inevitable even in database managed services
Define RPO, RTO
MySQL 5.7, End Of Life - 31st Oct 2023