Slides of Colin Charles and me talking at the MySQL Conference 2009: http://www.mysqlconf.com/mysql2009/public/schedule/detail/5664

1. MySQL Server Backup, Restoration,
and Disaster Recovery Planning
Colin Charles <byte@sun.com>
Lenz Grimmer <lenz@sun.com>
MySQL Conference 2009, Santa Clara, CA
2009-04-23
Sun Microsystems
1

2. Disclaimer
• Covering Linux / Unix only
• MySQL Cluster (NDB) has its own backup
method
2

3. Backing up MySQL data
• When do you need backups?
• What needs to be backed up?
• When should backups be performed?
• Where will the backups be stored?
• How can backups be performed?
3

4. When Do You Need Backups?
• Hardware failure
> A system crash may cause some of the data in
the databases to be lost
> A hard­disk failure will most certainly lead to lost
data
• User/Application failure
> Accidental DROP TABLE or malformed DELETE
FROM statements
> Editing the table files with text editors, usually
leading to corrupt tables
4

5. What needs to be backed up?
• Database content
> for full backups
> logical or physical backup
• Log files
> for incremental backups
> point­in­time recovery
• Configuration information
> /etc/my.cnf
> Cron jobs
• Consider using an SCM (bzr, git, hg) for
config files
5

6. When should backups be performed?
• On a regular basis
• Not during high usage peaks (off hours)
• Static data can be backed up less
frequently
6

7. Where to store backups?
• On the database server
> At least on a separate file system/volume or hard disk
drive
• Copied to another server
> On or off site
> Cloud storage (using encryption)
• Backed up to tape/disk
> Stored on or off site
• Choose multiple locations
7

8. The Data Directory
• Databases and most log and status files are
stored in the data directory by default
• Default directory compiled into the server
> /usr/local/mysql/data/ (tarball installation)
> /var/lib/mysql (RPM packages)
• Data directory location can be specified during
server startup with
­­datadir=/path/to/datadir/
• Find out the location by asking the server
mysql> SHOW VARIABLES like 'data%';
8

9. The Binary Log
• Contains all SQL commands that change data
(statement based) or the actual data that was modified
(row­based)
• Also contains additional information about each query
(e.g. query execution time)
• Binary log is stored in an efficient binary format
• Use mysqlbinlog to decipher the log contents
• Log turned on with ­­log­bin[=file_name]
• Update logs are created in sequence
e.g. file_name­bin.001, file_name­bin.002, etc.
• Binary log is transaction­compatible
• mysqld creates binary log index file which contains
names of the binary log files used
9

10. Managing The Binary Log
 Purpose of the Binary Log:
 Enable replication
 Ease crash recovery
 SHOW MASTER LOGS shows all binary log files
residing on the server
 FLUSH LOGS or restarting the server creates a
new file
 RESET MASTER deletes all binary log files
 PURGE MASTER deletes all binary log files up to
a certain point
 Don't delete logs that slaves still need
10

11. mysqldump
 Dumps table structure and data into SQL statements
$ mysqldump mydb > mydb.20090413.sql
 Dumps individual tables or whole databases
 Default output from mysqldump consists of SQL
statements:
– CREATE TABLE statements for table structure
– INSERT statements for the data
 Can also be used directly as input into another mysqld
server (without creating any files)
 $ mysqldump ­­opt world | mysql ­
hwork.mysql.com world
11

12. mysqldump hints
• Use ­­single­transaction when
backing up InnoDB tables
• ­­lock­all­tables is useful for
performing consistent MyISAM backups
> But locks all DML statements until backup is
done
• ­­flush­logs flushes the binary log file
(checkpointing)
12

13. Recovering from Backups
• Restoring tables to the state before a crash requires
both the backup files and the binary log
> Restore the tables to the state they were at the time
of the backup from the backup files
> Extract the queries issued between the backup and
now from synchronised binary logs
• If you are recovering data lost due to unwise
queries, remember not to issue them again
DB Recovery = Last full backup & binlogs
13

14. Example SQL level restore
• Restore the last full backup
mysql < backup.sql
• Apply all incremental changes done after
the last full backup
mysqlbinlog hostname­bin.000001 | mysql
14

15. MySQL table files backup
• Also called “physical” backup
• MyISAM Database files can simply be
copied after issuing FLUSH TABLES
WITH READ LOCK;
• The mysqlhotcopy Perl script automates
this process
• Locking all tables for consistency can be
expensive, if the file backup operation
takes a long time
15

16. OSS backup tools
• The usual suspects: cp, tar, cpio, gzip,
zip called in a shell script via a cron job
• rsync or unison for bandwidth­friendly,
remote backups
• Don't use these on live tables!
(Remember ma.gnolia.com?)
• Complete network­based backup solutions
like afbackup, Amanda or Bacula provide
more sophisticated features (e.g. catalogs)
16

17. XtraBackup / Maatkit
• http://percona.com/percona­lab.html
• Online backup for InnoDB (and XtraDB)
• Does not work with the InnoDB plugin
• Add this to my.cnf:
> [xtrabackup]
target_dir = /home/backups
• To backup:
> xtrabackup –backup
• http://maatkit.org/
• mk­parallel­dump / mk­parallel­restore
> Multithreaded Perl wrapper scripts
17

18. Linux backup support
• File system snapshots
> LVM
> Zumastor
> btrfs
> R1Soft Linux Hot Copy
• DRBD (“RAID1 over the network”)
• Distributed file systems
> OpenAFS
> GFS
> Lustre
> Novell iFolder
18

19. Backup using file system snapshots
• File system snapshots provide a very
convenient and fast backup solution for
backing up entire databases without
disruption
• Snapshot volume size does not need to be
very large (10­15% are sufficient in a
typical scenario)
• Backup of files from a snapshot volume
can be performed with any tool
19

20. Linux LVM snapshot creation
Basic principle:
mysql> FLUSH TABLES WITH READ LOCK
$ lvcreate ­s –­size=<size> ­­name=backup
<LV>
mysql> UNLOCK TABLES
$ mount /dev/<VG>/backup /mnt
$ tar czvf backup.tar.gz /mnt/*
$ umount /mnt
$ lvremove /dev/<VG>/backup
20

21. Benefits of MySQL Snapshot Backups
• “Almost hot” (no downtime)
• Supports all storage engines
• Fast, low overhead
• Easy integration
• Can be combined with log recovery
• Fast recovery
• (Usually) Free
21

22. Snapshot Backup Caveats
• Not incremental
• InnoDB ignores FLUSH TABLES WITH
READ LOCK
• FLUSH TABLES performance impact
• Possible I/O performance impact while
snapshot is active (Linux LVM)
• Handling data spread on multiple volumes
(DB logs on separate LV or DBs spread
across multiple LVs)
22

23. Linux LVM Snapshots
• Atomic, instant & exact copy of another LV
• Low disk space requirements (COW)
• LVM2 provides read & write access on
snapshots
> Useful for testing purposes (e.g. software
updates)
> Or cloning Xen DomU instances
> Or starting another MySQL instance
23

24. Linux LVM Overview
24

25. ZFS
• 128bit File System
• Solaris/OpenSolaris, FreeBSD, Linux (zfs­
fuse), Mac OS X
• Simple administration
• Pooled storage (no partitions/volumes)
• Copy­on­write transactions
25

26. ZFS (2)
• Checksums, self­healing (no silent data
corruption)
• Striping / mirroring / RAID / Compression
• ZFS Volumes (iSCSI)
• CIFS / NFS
26

27. ZFS Snapshots
• Read­only, point­in­time copy of the
filesystem
• Instantaneous creation
• (Virtually) unlimited number of snapshots
• Initially, no additional space used
• Writable copies (Clones)
• Incremental replication (zfs send/receive)
• Snapshots are simple & cheap to create!
zfs snapshot fsname@snapname
27

28. The mylvmbackup script
• A Perl script for quickly creating MySQL backups
using LVM snapshots
• Snapshots are mounted to a temporary directory
and all data is backed up using tar,rsync or
rsnap
• Timestamped archive names allow running
mylvmbackup many times without risking to
overwrite old archives
• Can perform InnoDB log recovery on the
snapshot prior to backup (LVM2)
• Requires Perl, DBI and DBD::mysql
• http://www.lenzg.net/mylvmbackup/
28

29. MySQL replication
• Backing up a replication slave is less time­critical
(Master is not blocked for updates)
• A slave can use different storage engines
• One Master can replicate to many slaves
• Keep the limitations of MySQL replication in
mind
• Make sure to back up the master.info and
relay-log.info files as well as any
SQL_LOAD-* files (if LOAD DATA INFILE is
replicated)
29

30. Commercial backup solutions
• Acronis True Image
• ARCServe
• Arkeia
• InnoDB HotBackup
• SEP sesam
• Veritas vxfs snapshots
• R1Soft CDP
• Zmanda Recovery Manager (ZRM)
30

31. Backup Method Comparison
• Output from mysqldump is portable to any other
DBMS (without the ­­opt option) whereas
copied files only work with MySQL
• Full backups are expensive
• Restoring from logs can be tricky
• The file copying methods are much faster than
mysqldump
• So it comes down to your preferences:
– Which tool do you prefer to use
– Speed vs. portability
31

32. Backup Principles
• Perform backups regularly
• Turn on the binary update log
> Update logs are needed to restore the database without
losing any data
• Synchronise update logs with the backup files
> Use FLUSH LOGS
• Name your backups consistently and understandably
> Include the date in the file name mydb.20090414.sql
• Store your backups on a different file system than where
your databases are
32

33. General backup notes
• Putting the binary logs on a different file system
(or even a different drive) than the data directory
is recommended (increases performance and
avoids data loss)
• Verify the backup is consistent and complete!
• Define backup schedules and policies as well
as recovery procedures
• Test that these actually work!
33

34. The MySQL Online Backup API
• An API to perform a streaming MySQL online
backup, independent of the Storage Engine
• Transactional tables will contain data only from
committed transactions
• Non­transactional tables will contain data only
from completed statements
• Referential integrity will be maintained between
all tables backed up with a specific backup
command
• Now available on MySQL Forge:
http://forge.mysql.com/wiki/OnlineBackup
34

35. Online Backup example
• Future (MySQL 6)
• Commands
> BACKUP DATABASE sakila TO 'sakila­
backup.sql';
> Metadata: SELECT * FROM
mysql.online_backup WHERE
backup_id = 1 G
> RESTORE FROM 'sakila­backup.sql';
> Metadata: SELECT * FROM
mysql.online_backup WHERE
backup_id = 2 G
35

36. Disaster Recovery
• Business continuity planning
> 1. Minimize financial loss
> 2. Reduce time to restore operations
> 3. Increase sense of security
• Emergency planning
> People involved
> Steps to perform
> Location of offsite backups?
36

37. Discussion
Thank you!
Lenz Grimmer <lenz@sun.com>
http://lenzg.net/
Colin Charles <byte@sun.com>
http://bytebot.net/
37

38. The Error Log
• When started with mysqld_safe, all error
messages are directed to the error log
• The log contains info on when mysqld was started
and stopped as well as errors found when running
$ cat /var/log/mysql.err
000929 15:29:45 mysqld started
/usr/sbin/mysqld: ready for connections
000929 15:31:15 Aborted connection 1 to db: 'unconnected'
user: 'root' host: `localhost' (Got an error writing communication
packets)
000929 15:31:15 /usr/local/mysql/bin/mysqld: Normal shutdown
000929 15:31:15 /usr/local/mysql/bin/mysqld: Shutdown Complete
000929 15:31:54 mysqld started
/usr/sbin/mysqld: ready for connections
38

39. Backing Up InnoDB Databases
• Use mysqldump ­­single transaction to make an
on­line backup
• InnoDB Hot Backup (commercial)
• To take a ’binary’ backup, do the following:
1. Shutdown the MySQL server
2. Copy your data files, InnoDB log files, .frm files and
my.cnf file(s) to a safe location
3. Restart the server
• It is a good idea to backup with mysqldump also, since
an error might occur in a binary file without you noticing
it
39