SlideShare a Scribd company logo

Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks

•
•
Jaime Crespo
Jaime Crespo

Tutorial delivered at Percona Live London 2014, where we explore new features and techniques for faster queries with MySQL 5.6 and 5.7 and MariaDB 10, including the newest options in MySQL 5.7.5 and MariaDB 10.1. Download here the virtual machine with the example database: http://dbahire.com/pluk14 Update: WordPress has a workaround for STRICT mode: https://core.trac.wordpress.org/ticket/26847

Technology
1 of 111
Download to read offline
DBAHire.com Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Jaime Crespo Percona Live London 2014 -3 Nov 2014- dbahire.com/pluk14 1
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Agenda - 3 hours 2 1. Introduction 8. Subquery Optimization 2. Break (VM installation) 9. Fulltext Search 3. General Optimizer Improvements 10. GIS Improvements 4. Computed/Virtual Columns 11. Break 5. Query Rewrite Plugins 12. Query Profiling 6. SQL Mode Changes 13. Results and Conclusions 7. JOIN Optimization 14. Q&A
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks INTRODUCTION © 2014 DBAHIRE.COM 3
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks This is Me Fighting Bad Query Performance © 2014 DBAHIRE.COM • MySQL Consultant at DBAHire.com • Used to work for Oracle (MySQL), Percona • Loves MySQL query optimization and HA 4
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL Versions • Oracle MySQL 5.6 (GA) almost 2 years old • MariaDB 10 over 7 months old • Several lab/alpha releases of MySQL 5.7 and MariaDB 10.1 with great new features – We will focus only on SQL-related ones – We will not perform formal benchmarks or comparisons 5
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks New Query Optimization Features • Index Condition Pushdown • Subquery Optimizations (materialization and semijoin) • IN-to-EXISTS • JOIN-to-WHERE • Multi-Range Read • Batched Key Access • Persistent InnoDB Statistics • EXPLAIN FORMAT=JSON • EXPLAIN INSERT/UPDATE/DELETE • Hash Join • New cost-based optimizer • Optimizer Trace • Filesort with a short LIMIT © 2014 DBAHIRE.COM • EXISTS-to-IN • EXPLAIN FOR CONNECTION • Faster eq_ref • Extended secondary keys • Engine-Independent statistics • Improved Spatial Indexes (GIS) • Histograms • Virtual columns • PERFORMANCE_SCHEMA improvements • Forced strict SQL mode; functional dependencies in GROUP BY • Query rewrite plugin • Configurable optimizer_search_depth 6
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks I Already Mentioned Some of Those Topics © 2014 DBAHIRE.COM One Year Ago • Check my presentation here: http://www.slideshare.net/jynus/query-optimization56 7
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Example Application • OSM Database is licensed under the Open DataBase License and is Copyright OpenStreetMap Contributors 8
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GENERAL OPTIMIZER IMPROVEMENTS © 2014 DBAHIRE.COM 9
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM EXPLAIN FOR CONNECTION (SHOW EXPLAIN FOR) mysql-5.7.5 (osm)> SHOW PROCESSLIST; +----+------+-----------+------+---------+------+-----------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------+------+---------+------+-----------+------------------+ | 18 | root | localhost | osm | Query | 1867 | updating | UPDATE nodes n | | | | | | | | reference | JOIN users u | | | | | | | | tables | ON n.user=u.name | | | | | | | | | SET n.user=u.id | | 19 | root | localhost | NULL | Query | 0 | starting | SHOW PROCESSLIST | +----+------+-----------+------+---------+------+-----------+------------------+ 2 rows in set (0.00 sec) mysql-5.7.5 (osm)> EXPLAIN FOR CONNECTION 18; +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ | id | select | table | parti- | type | possible | key | key | ref | rows | filte- | Extra | | | _type | | tions | | _keys | | _len | | | red | | +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ | 1 | SIMPLE | u | NULL | index | name | name | 258 | NULL | 11271 | 100.00 | Using where; | | | | | | | | | | | | | Using index | | 1 | UPDATE | n | NULL | ref | user | user | 258 | osm.u. | 280 | 100.00 | NULL | | | | | | | | | name | | | | | +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ 2 rows in set (0.00 sec) 10 Use “SHOW EXPLAIN FOR #” with MariaDB 10
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks MySQL 5.7.5 Introduced an New Cost- © 2014 DBAHIRE.COM Based Model for Its Optimizer mysql-5.5.40 (osm) > SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G 59 rows in set (1.37 sec) 11 mysql-5.5.40 (osm) > SELECT STRAIGHT_JOIN * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G 59 rows in set (0.86 sec) This condition is very selective, but there is no index available
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Why does it take the wrong table order? mysql-5.5.40 (osm) > EXPLAIN EXTENDED SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** © 2014 DBAHIRE.COM id: 1 select_type: SIMPLE table: node_tags type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 839031 filtered: 100.00 Extra: *************************** 2. row **************************** id: 1 select_type: SIMPLE table: nodes type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.node_tags.node_id rows: 1 filtered: 100.00 Extra: Using where 2 rows in set, 1 warning (0.00 sec) 12
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Unindexed Columns Are not Accounted mysql-5.5.40 (osm) > EXPLAIN EXTENDED SELECT STRAIGHT_JOIN * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** © 2014 DBAHIRE.COM id: 1 select_type: SIMPLE table: nodes type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2865521 filtered: 100.00 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 1 filtered: 100.00 Extra: 2 rows in set, 1 warning (0.00 sec) 13 The optimizer assumes that all rows will be returned
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM What’s new in 5.7? mysql-5.7.5 (osm) > EXPLAIN SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2773853 filtered: 11.11 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 3 filtered: 100.00 Extra: NULL 2 rows in set, 1 warning (0.00 sec) 14 It is actually 0.002%, but it is good enough
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM If things go wrong -our Friend optimizer_switch mysql-5.7.5 (osm) > SHOW VARIABLES like 'optimizer_switch'G ******************* 1. row ******************* Variable_name: optimizer_switch Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on, index_merge_intersection=on,engine_condition_pushdown=on,index _condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_l oop=on,batched_key_access=off,materialization=on,semijoin=on,l oosescan=on,firstmatch=on,subquery_materialization_cost_based= on,use_index_extensions=on,condition_fanout_filter=on 1 row in set (0.00 sec) 15
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Improved EXPLAIN FORMAT=JSON "nested_loop": [ © 2014 DBAHIRE.COM { "table": { "table_name": "nodes", "access_type": "ALL", "possible_keys": [ "PRIMARY" ], "rows_examined_per_scan": 2773853, "rows_produced_per_join": 308175, "filtered": 11.11, "cost_info": { "read_cost": "512783.58", "eval_cost": "61635.01", "prefix_cost": "574418.60", "data_read_per_join": "21M" }, 16 Cost information is now included Engine statistics are now floats for improved precision
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Optimizer Cost Tuning 17 This point depends on the hardware and buffer status
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Configurable Costs mysql-5.7.5 ((none))> SELECT * FROM mysql.server_cost; +------------------------------+------------+---------------------+---------+ | cost_name | cost_value | last_update | comment | +------------------------------+------------+---------------------+---------+ | disk_temptable_create_cost | NULL | 2014-10-21 23:03:30 | NULL | | disk_temptable_row_cost | NULL | 2014-10-21 23:03:30 | NULL | | key_compare_cost | NULL | 2014-10-21 23:03:30 | NULL | | memory_temptable_create_cost | NULL | 2014-10-21 23:03:30 | NULL | | memory_temptable_row_cost | NULL | 2014-10-21 23:03:30 | NULL | | row_evaluate_cost | NULL | 2014-10-21 23:03:30 | NULL | +------------------------------+------------+---------------------+---------+ 6 rows in set (0.00 sec) mysql-5.7.5 ((none))> SELECT * FROM mysql.engine_cost; +-------------+-------------+--------------------+------------+---------------------+---------+ | engine_name | device_type | cost_name | cost_value | last_update | comment | +-------------+-------------+--------------------+------------+---------------------+---------+ | default | 0 | io_block_read_cost | NULL | 2014-10-21 23:03:30 | NULL | +-------------+-------------+--------------------+------------+---------------------+---------+ 1 row in set (0.00 sec) 18 The unit is “read of a random data page”
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Changing Costs Example mysql-5.7.5 (osm) > EXPLAIN FORMAT=JSON SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'G "cost_info": { "query_cost": "22567.80" 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > UPDATE mysql.server_cost SET cost_value = 10; mysql-5.7.5 (osm) > FLUSH OPTIMIZER_COSTS; <session restart> mysql-5.7.5 (osm) > EXPLAIN ...G "cost_info": { "query_cost": "661371.00" 1 row in set, 1 warning (0.00 sec) 19 More info on usage in the manual: http://dev.mysql.com/doc/refman/5.7/en/cost-model.html
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Proposed Features • Technology-dependent costs (HD, SSD, memory) – Some of them could be auto-detected • Buffer “hotness”-aware statistics • Statistics for non-indexed columns/histograms 20 For more on cost optimizer changes and future, read Olav Sandstå’s slides: http://www.slideshare.net/olavsa/mysql-optimizer- cost-model
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Engine-Independent Statistics/Histograms © 2014 DBAHIRE.COM in MariaDB 10 mariadb-10.0.14 (osm) > SET histogram_size = 255; Query OK, 0 rows affected (0.00 sec) mariadb-10.0.14 (osm) > SET use_stat_tables = 2; Query OK, 0 rows affected (0.01 sec) mariadb-10.0.14 (osm) > ANALYZE TABLE node_tags; +---------------+---------+----------+-----------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------+---------+----------+-----------------------------------------+ | osm.node_tags | analyze | status | Engine-independent statistics collected | | osm.node_tags | analyze | status | OK | +---------------+---------+----------+-----------------------------------------+ 2 rows in set (3.01 sec) mariadb-10.0.14 (osm) > ANALYZE TABLE nodes; +-----------+---------+----------+-----------------------------------------+ | Table | Op | Msg_type | Msg_text | +-----------+---------+----------+-----------------------------------------+ | osm.nodes | analyze | status | Engine-independent statistics collected | | osm.nodes | analyze | status | OK | +-----------+---------+----------+-----------------------------------------+ 2 rows in set (32.52 sec) mariadb-10.0.14 (osm) > SET optimizer_use_condition_selectivity = 3; (or 4 for histograms) Query OK, 0 rows affected (0.00 sec) 21
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Stats! mariadb-10.0.14 (osm) > EXPLAIN EXTENDED SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: nodes type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2865312 filtered: 0.39 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 3 filtered: 100.00 Extra: 2 rows in set, 1 warning (0.01 sec) 22 Much better estimate
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Other Changes/Bugs Fixed • UNION ALL does not create temporary tables, returns tables faster • (a, b) IN ((1, 2), (2, 3)) can use index ranges • EXPLAIN EXTENDED is now the default behavior • I.C. Pushdown support for partitioned tables • IGNORE clause meaning has been standardized between sentence types • Increased default for optimizer_search_depth 23
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks COMPUTED/VIRTUAL COLUMNS © 2014 DBAHIRE.COM 24
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Syntax ALTER TABLE nodes ADD COLUMN lon DECIMAL (10, 7) as (longitude/10000000) VIRTUAL, ADD COLUMN lat DECIMAL (9, 7) as (latitude/10000000) VIRTUAL; • They can be used to simplify SELECTS, calculating values on the fly •Non accessed rows are not calculated 25 VIRTUAL is optional (default)
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Virtual Columns Performance mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE lat > 51.5000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (1.10 sec) mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE latitude > 515000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (0.32 sec) mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE latitude/10000000 > 51.5000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (0.79 sec) © 2014 DBAHIRE.COM 26
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Explanation mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE lat > 51.5000000; +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | 1 | SIMPLE | nodes | NULL | ALL | NULL | NULL | NULL | NULL | 2602012 | 33.33 | Using where | +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE latitude > 515000000; +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | nodes | NULL | range | latitude | latitude | 4 | NULL | 1301006 | 100.00 | Using where; Using index | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE latitude/10000000 > 51.5000000; +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | nodes | NULL | index | NULL | latitude | 8 | NULL | 2602012 | 100.00 | Using where; Using index | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ mysql-5.7.5 (osm) > ALTER TABLE nodes add index(lon); ERROR 1951 (HY000): Key/Index cannot be defined on a non-stored virtual column. 27
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Stored Columns mysql-5.7.5 (osm) > ALTER TABLE nodes CHANGE lat lat DECIMAL (9, 7) as (latitude/10000000) STORED; ERROR 1954 (HY000): 'Changing the STORED status' is not yet supported for virtual columns. mysql-5.7.5 (osm) > ALTER TABLE nodes DROP COLUMN lat, ADD COLUMN lat DECIMAL (9, 7) as (latitude/10000000) STORED; Query OK, 2865312 rows affected (4 min 51.05 sec) Records: 2865312 Duplicates: 0 Warnings: 0 28 MariaDB uses the 'PERMANENT' keyword
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks A Query That Cannot Be Indexed mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE MONTH(timestamp) = 1; +----------+ | count(*) | +----------+ | 373557 | +----------+ 1 row in set (0.57 sec) © 2014 DBAHIRE.COM mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE MONTH(timestamp) = 1G ************* 1. row ************* id: 1 select_type: SIMPLE table: nodes partitions: NULL type: index possible_keys: NULL key: nodes_timestamp_idx key_len: 5 ref: NULL rows: 2576970 filtered: 100.00 Extra: Using where; Using index 1 row in set, 1 warning (0.00 sec) 29
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks We Add an Indexed Stored Computed © 2014 DBAHIRE.COM Column mysql-5.7.5 (osm) > ALTER TABLE nodes ADD COLUMN `month` tinyint UNSIGNED AS (MONTH(timestamp)) STORED, ADD INDEX(`month`); Query OK, 2865312 rows affected (7 min 9.12 sec) Records: 2865312 Duplicates: 0 Warnings: 0 30
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Functional Index Implementation mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE month = 1; +----------+ | count(*) | +----------+ | 373557 | +----------+ 1 row in set (0.05 sec) © 2014 DBAHIRE.COM mysql-5.7.5 (osm) > EXPLAIN SELECT month, count(*) FROM nodes WHERE month = 1G ************* 1. row ************* id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ref possible_keys: month key: month key_len: 2 ref: const rows: 775360 filtered: 100.00 Extra: Using index 1 row in set, 1 warning (0.00 sec) 31
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Why Not Just Use Triggers? CREATE TABLE `nodes_trigger` ( `node_id` bigint(64) NOT NULL, `latitude` int(11) NOT NULL, `longitude` int(11) NOT NULL, `changeset_id` bigint(20) NOT NULL, `visible` tinyint(1) NOT NULL, `timestamp` datetime NOT NULL, `tile` int(10) unsigned DEFAULT NULL, `version` bigint(20) NOT NULL, `month` tinyint(3) unsigned DEFAULT NULL, PRIMARY KEY (`node_id`,`version`), KEY `nodes_timestamp_idx` (`timestamp`), KEY `nodes_tile_idx` (`tile`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TRIGGER nodes_trigger_BI BEFORE INSERT ON nodes_trigger FOR EACH ROW SET NEW.month = MONTH(NEW.timestamp); CREATE TRIGGER nodes_trigger_BU BEFORE UPDATE ON nodes_trigger FOR EACH ROW SET NEW.month = MONTH(NEW.timestamp); CREATE TABLE `nodes_computed` ( `node_id` bigint(64) NOT NULL, `latitude` int(11) NOT NULL, `longitude` int(11) NOT NULL, `changeset_id` bigint(20) NOT NULL, `visible` tinyint(1) NOT NULL, `timestamp` datetime NOT NULL, `tile` int(10) unsigned DEFAULT NULL, `version` bigint(20) NOT NULL, `month` tinyint(3) unsigned AS (MONTH(timestamp)) STORED, PRIMARY KEY (`node_id`,`version`), KEY `nodes_timestamp_idx` (`timestamp`), KEY `nodes_tile_idx` (`tile`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; 32
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Performance? $ ./sandboxes/msb_5_7_5/my sqldump osm nodes > nodes.sql $ time sed -e 's/nodes/nodes_trigger/' < nodes.sql | ./sandboxes/msb_5_7_5/use osm real 2m21.528s user 0m3.073s sys 0m0.463s $ time sed -e 's/nodes/nodes_computed/' < nodes.sql | ./sandboxes/msb_5_7_5/use osm real 2m24.458s user 0m3.079s sys 0m0.475s mysql-5.7.5 (osm) > INSERT INTO nodes_trigger(node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version) SELECT node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version FROM nodes; Query OK, 2865312 rows affected (1 min 6.84 sec) Records: 2865312 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > INSERT INTO nodes_computed(node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version) SELECT node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version FROM nodes; Query OK, 2865312 rows affected (54.75 sec) Records: 2865312 Duplicates: 0 Warnings: 0 33 No significative improvements seen within a session
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Consistency? mysql-5.7.5 (osm) > UPDATE nodes_trigger SET month = 13 WHERE rand() < 0.1; Query OK, 0 rows affected (1.71 sec) Rows matched: 286169 Changed: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT count(*) FROM nodes_trigger WHERE month=13; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.45 sec) mysql-5.7.5 (osm) > UPDATE nodes_computed SET month = 13 WHERE rand() < 0.1; Query OK, 0 rows affected, 65535 warnings (1.31 sec) Rows matched: 286177 Changed: 0 Warnings: 286177 mysql-5.7.5 (osm) > SHOW WARNINGS; +---------+------+-----------------------------------------------------------------------------------+ | Level | Code | Message | +---------+------+-----------------------------------------------------------------------------------+ | Warning | 1953 | The value specified for virtual column 'month' in table 'nodes_computed' ignored. | mysql-5.7.5 (osm) > SELECT count(*) FROM nodes_computed WHERE month=13; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.57 sec) 34 Maybe an error should be produced in strict mode, as MariaDB does?
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Features and Limitations • Virtual “non-stored” columns cannot be indexed • “Stored columns” can be PRIMARY, UNIQUE, FOREIGN and MULTI keys • It cannot contain subqueries or other tables or rows references • It cannot contain used-defined stored functions • It can contain server and user-defined variables • It can be computed based on other virtual columns 35
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks QUERY REWRITE PLUGINS © 2014 DBAHIRE.COM 36
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks New APIs for Query Rewriting • One for pre-parsing rewriting • Another for post-parsing rewriting © 2014 DBAHIRE.COM 37
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Post- Plugin Installation $ ./sandboxes/msb_5_7_5/use -u root < 5.7.5/lib/plugin/ install_rewriter_plugin.sql – It is mostly broken in the lab build (for my tests) – Uninstall it before restart, or MySQL will segfault © 2014 DBAHIRE.COM 38 More on this: http://mysqlserverteam.com/the-query-rewrite-plugins/
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Query Rewriting Setup mysql-5.7.5 ((none)) > INSERT INTO query_rewrite.rewrite_rules (pattern, replacement) VALUES ('SELECT ?', 'SELECT ? + 1'); Query OK, 1 row affected (0.01 sec) mysql-5.7.5 ((none)) > CALL query_rewrite.flush_rewrite_rules(); Query OK, 1 row affected (0.00 sec) mysql-5.7.5 ((none)) > SELECT 1; +-------+ | 1 + 1 | +-------+ | 2 | +-------+ 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 ((none)) > SHOW WARNINGS; +-------+------+-------------------------------------------------------------------+ | Level | Code | Message | +-------+------+-------------------------------------------------------------------+ | Note | 1105 | Query 'SELECT 1' rewritten to 'SELECT 1 + 1' by plugin: Rewriter. | +-------+------+-------------------------------------------------------------------+ 1 row in set (0.00 sec) 39
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Considerations • It cannot correct malformed queries- pattern and replacement must be syntactically correct • Useful for query optimizations for 3rd party applications • User is notified that a rewrite happened with a note-level message • Low overhead (5%); specially for untouched queries • You can do stupid things like: mysql-5.7.5 ((none)) > INSERT INTO query_rewrite.rewrite_rules( pattern, replacement ) VALUES ( 'SELECT 1', 'DROP TABLE test.test' ); Query OK, 1 row affected (0.01 sec) 40
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks SQL MODE CHANGES © 2014 DBAHIRE.COM 41
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Default SQL Mode Changes • MySQL 5.5 and earlier – '' • MySQL 5.6 (from 5.6.6): – 'NO_ENGINE_SUBSTITUTION' is the default – NO_ENGINE_SUBSTITUTION and STRICT_TRANS_TABLES were suggested in upstream default my.cnf • MySQL 5.7 (from 5.7.5): – 'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_EN GINE_SUBSTITUTION' is the new default 42 More on this: http://www.tocker.ca/2014/01/14/making-strict-sql_mode-the-default.html
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Stricter Defaults mysql> CREATE TABLE `test` ( `id` int(11) NOT NULL, `type` enum('movie','album','videogame') NOT NULL, `date` datetime NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB; mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); 43 What happens?
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks MySQL 5.5/5.6 with default settings mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); Query OK, 1 row affected, 3 warnings (0.00 sec) mysql> SHOW WARNINGS; +---------+------+-----------------------------------------------+ | Level | Code | Message | +---------+------+-----------------------------------------------+ | Warning | 1364 | Field 'id' doesn't have a default value | | Warning | 1265 | Data truncated for column 'type' at row 1 | | Warning | 1264 | Out of range value for column 'date' at row 1 | +---------+------+-----------------------------------------------+ 3 rows in set (0.00 sec) mysql> SELECT * FROM test; +----+------+---------------------+ | id | type | date | +----+------+---------------------+ | 0 | | 0000-00-00 00:00:00 | +----+------+---------------------+ 1 row in set (0.00 sec) © 2014 DBAHIRE.COM 44
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL 5.7 mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); ERROR 1265 (01000): Data truncated for column 'type' at row 1 mysql> INSERT INTO test (type, date) VALUES ('videogame', -1); ERROR 1292 (22007): Incorrect datetime value: '-1' for column 'date' at row 1 mysql> INSERT INTO test (type, date) VALUES ('movie', now()); ERROR 1364 (HY000): Field 'id' doesn't have a default value mysql> INSERT INTO test (id, type, date) VALUES (1, 'videogame', now()); Query OK, 1 row affected (0.01 sec) 45
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GROUP BY Behavior Changes mysql-5.6.21 (osm) > SELECT way_id, count(*), node_id © 2014 DBAHIRE.COM FROM way_nodes GROUP BY way_id ORDER BY count(*) DESC LIMIT 10; +-----------+----------+------------+ | way_id | count(*) | node_id | +-----------+----------+------------+ | 155339744 | 1187 | 1558095871 | | 243986064 | 1156 | 2604713337 | | 87136668 | 1128 | 1013304944 | | 148812873 | 852 | 1618837453 | | 149200774 | 835 | 34921158 | | 183618216 | 826 | 1940223096 | | 273858696 | 824 | 1267549776 | | 261584374 | 770 | 2669122104 | | 227880171 | 704 | 2240011804 | | 193564006 | 684 | 1808872763 | +-----------+----------+------------+ 10 rows in set (1.24 sec) 46 Undeterministic behavior
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks With ONLY_FULL_GROUP_BY (default in 5.7) mysql-5.7.5 (osm) > SELECT way_id, count(*), node_id © 2014 DBAHIRE.COM FROM way_nodes GROUP BY way_id ORDER BY count(*) DESC LIMIT 10; ERROR 1055 (42000): Expression #3 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'osm.way_nodes.node_id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by 47
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Problem with ONLY_FULL_GROUP_BY in Functional dependency © 2014 DBAHIRE.COM MySQL <= 5.6 mysql-5.6.21 (osm) > SET SQL_mode='ONLY_FULL_GROUP_BY'; Query OK, 0 rows affected (0.00 sec) mysql-5.6.21 (osm) > SELECT u.id as `user id`, u.display_name as `user name`, count(*) as `# changesets` FROM users u JOIN changesets c ON u.id = c.user_id GROUP BY u.id ORDER BY count(*) DESC LIMIT 10; ERROR 1055 (42000): 'osm.u.display_name' isn't in GROUP BY 48 More on this: http://rpbouman.blogspot.com/2014/09/mysql-575-group-by-respects-functional.html
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks 5.7 Aims for SQL99 Compliance mysql-5.7.5 (osm) > SELECT u.id as `user id`, u.display_name as `user name`, count(*) as `# changesets` FROM users u JOIN changesets c ON u.id = c.user_id GROUP BY u.id ORDER BY count(*) DESC LIMIT 10; +---------+---------------+--------------+ | user id | user name | # changesets | +---------+---------------+--------------+ | 31257 | Ed Avis | 4333 | | 508 | Welshie | 2696 | | 1016290 | Amaroussi | 2351 | | 352985 | ecatmur | 1450 | | 142807 | SDavies | 1342 | | 736 | Steve Chilton | 1182 | | 346 | Tom Chance | 1175 | | 38784 | Tom Morris | 1165 | | 88718 | UrbanRambler | 1151 | | 1611 | Harry Wood | 1091 | +---------+---------------+--------------+ 10 rows in set (0.09 sec) © 2014 DBAHIRE.COM 49
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Backward Compatibility • Some ORMs and frameworks change the default SQL Mode: – Ruby on Rails 4+ sets STRICT_ALL_TABLES – Drupal 7+ sets TRADITIONAL • Other applications do not work in standard-compliance modes: – Wordpress does not work in strict mode: http:// www.xaprb.com/blog/2013/03/15/wordpress-and-mysqls-strict-mode/ – Cacti does not work with ONLY_FULL_GROUP_BY,NO_ZERO_DATE 50
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Deprecated Modes • ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, and NO_ZERO_IN_DATE are deprecated and do nothing – Use STRICT_TRANS_TABLES or STRICT_ALL_TABLES, which include those modes and produce an error 51
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks JOIN OPTIMIZATIONS © 2014 DBAHIRE.COM 52
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM (Block) Nested Loop Join • Until MySQL 5.5, there was only one algorithm to execute a JOIN: 53 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 name London Eye
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM “New” JOIN Methods • MySQL 5.6 added: – Batch Key Access* • MariaDB has since 5.3: – Batch Key Access* – Hash Joins – Slightly modified versions of the above (“incremental” buffers for joining 3 or more tables) 54 *I already mentioned BKA last year
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Batch Key Access 55 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 amenity cafe Buffer sorted in physical/ PK order
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Hash Joins • Only work for equi-joins 56 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 amenity cafe Hash table for faster access to rows
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL Configuration • BKA requires changes of default optimizer configuration: mysql-5.7.5 (osm) > SET optimizer_switch= 'mrr=on'; mysql-5.7.5 (osm) > SET optimizer_switch= 'mrr_cost_based=off'; mysql-5.7.5 (osm) > SET optimizer_switch= 'batch_key_access=on'; 57
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MariaDB configuration mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_incremental=on'; mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_hashed=on'; mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_bka=on'; - Enabled by default mariadb-10.0.14 (osm) > SET join_cache_level = 3 (for hash joins) mariadb-10.0.14 (osm) > SET join_cache_level = 5 (for BKA) - Also, configure join_buffer_size appropriately. - Hash joins, like BKA, are highly dependent on disk-bound DBs to be effective due to the extra overhead 58
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Nested Loop Join (cold buffers, buffer_pool=100MB) mariadb-10.0.14 (osm) > EXPLAIN SELECT changeset_id, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visibleG ******************** 1. row ******************** id: 1 select_type: SIMPLE table: changesets type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 69115 Extra: Using index; Using temporary; Using filesort ******************** 2. row ******************** id: 1 select_type: SIMPLE table: nodes type: ref possible_keys: changeset_id key: changeset_id key_len: 8 ref: osm.changesets.id rows: 19 Extra: 2 rows in set (0.00 sec) mariadb-10.0.14 (osm) > SELECT visible, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visible; +---------+----------+ | visible | count(*) | +---------+----------+ | 1 | 2865312 | +---------+----------+ 1 row in set (32.86 sec) 59
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Hash Join (cold buffers, buffer_pool=100M, © 2014 DBAHIRE.COM join_buffer=4M) mariadb-10.0.14 (osm) > EXPLAIN SELECT changeset_id, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visibleG ******************** 1. row ******************** id: 1 select_type: SIMPLE table: changesets type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 69115 Extra: Using index; Using temporary; Using filesort ******************** 2. row ******************** id: 1 select_type: SIMPLE table: nodes type: hash_ALL possible_keys: changeset_id key: #hash#changeset_id key_len: 8 ref: osm.changesets.id rows: 2781732 Extra: Using join buffer (flat, BNLH join) 2 rows in set (0.00 sec) mariadb-10.0.14 (osm) > SELECT visible, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visible; +---------+----------+ | visible | count(*) | +---------+----------+ | 1 | 2865312 | +---------+----------+ 1 row in set (6.66 sec) 60
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks SUBQUERY OPTIMIZATIONS © 2014 DBAHIRE.COM 61
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Delayed Materialization of Derived Tables • EXPLAIN performance improved (it does not execute subqueries) • Derived tables can be indexed for improved performance © 2014 DBAHIRE.COM 62
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example with a Derived Table mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON n.changeset_id = changesets.id; +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2865312 | | | 1 | PRIMARY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | n.changeset_id | 1 | Using index | | 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where | +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ 3 rows in set (1.42 sec) mysql-5.6.21 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON n.changeset_id = changesets.id; +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ | 1 | PRIMARY | changesets | index | PRIMARY | PRIMARY | 8 | NULL | 70917 | Using index | | 1 | PRIMARY | <derived2> | ref | <auto_key0> | <auto_key0> | 8 | osm.changesets.id | 40 | NULL | | 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2853846 | Using where | +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ 3 rows in set (0.00 sec) © 2014 DBAHIRE.COM 63 No subquery execution Auto-generated index
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks A Common 5.5 Performance Problem mysql-5.5.40 (osm) > EXPLAIN SELECT * FROM nodes © 2014 DBAHIRE.COM WHERE nodes.changeset_id IN ( SELECT changesets.id FROM changesets JOIN users ON changesets.user_id = users.id and users.display_name = 'Steve'); +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ | 1 | PRIMARY | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where | | 2 | DEPENDENT SUBQUERY | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index | | 2 | DEPENDENT SUBQUERY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | func | 1 | Using where | +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ 3 rows in set (0.00 sec) mysql-5.5.40 (osm) > SELECT ...; +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 | | 109174 | 515364532 | -1457329 | 1875 | 1 | 2006-01-20 00:01:27 | 2062268519 | 1 | | 276538 | 515324296 | -2094688 | 810 | 1 | 2005-11-25 21:42:53 | 2062267078 | 1 | | 442987 | 515449207 | -1275650 | 1941 | 1 | 2006-01-22 23:50:28 | 2062268628 | 1 | | 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 | | 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 | | 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ 7 rows in set (2.60 sec) 64 This means that the subquery is executed almost 3 million times
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Semijoin Optimization • The only way to execute certain IN subqueries was to execute them with poor strategy – This forced to rewrite certain queries into JOINS or scalar subqueries, when possible • There are now several additional strategies/ optimizations: – Convert to a JOIN – Materialization (including index creation) – FirstMatch – LooseScan – Duplicate Weedout 65
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks The Previous Query is Not a Problem in © 2014 DBAHIRE.COM 5.6/5.7/MariaDB 5.3+ mysql-5.6.21 (osm) > EXPLAIN SELECT * FROM nodes WHERE nodes.changeset_id IN ( SELECT changesets.id FROM change sets JOIN users ON changesets.user_id = users.id and users.display_name = 'Steve'); +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ | 1 | SIMPLE | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index | | 1 | SIMPLE | changesets | ALL | PRIMARY | NULL | NULL | NULL | 70917 | Using where | | 1 | SIMPLE | nodes | ref | changeset_id | changeset_id | 8 | osm.changesets.id | 21 | NULL | +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ 3 rows in set (0.00 sec) mysql-5.6.21 (osm) > SELECT ...; +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 | | 276538 | 515324296 | -2094688 | 810 | 1 | 2005-11-25 21:42:53 | 2062267078 | 1 | | 109174 | 515364532 | -1457329 | 1875 | 1 | 2006-01-20 00:01:27 | 2062268519 | 1 | | 442987 | 515449207 | -1275650 | 1941 | 1 | 2006-01-22 23:50:28 | 2062268628 | 1 | | 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 | | 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 | | 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ 7 rows in set (0.02 sec) 66 Executed as a regular JOIN
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM First Match Strategy mysql-5.7.5 (osm) > EXPLAIN SELECT * FROM changesets WHERE id IN (SELECT changeset_id FROM nodes)G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: changesets partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 70917 filtered: 100.00 Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ref possible_keys: changeset_id key: changeset_id key_len: 8 ref: osm.changesets.id rows: 33 filtered: 100.00 Extra: Using index; FirstMatch(changesets) 67 It is converting the ref into an eq_ref
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Enabling and disabling materialization/ © 2014 DBAHIRE.COM semijoin mysql-5.7.5 (osm) > SHOW VARIABLES like 'optimizer_switch'G ******************* 1. row ******************* Variable_name: optimizer_switch Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on, index_merge_intersection=on,engine_condition_pushdown=on,index _condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_l oop=on,batched_key_access=off,materialization=on,semijoin=on,l oosescan=on,firstmatch=on,subquery_materialization_cost_based= on,use_index_extensions=on,condition_fanout_filter=on 1 row in set (0.00 sec) 68
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks FULLTEXT SEARCH © 2014 DBAHIRE.COM 69
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Fuzzy Search of “gloucester/Gloucester’s/ © 2014 DBAHIRE.COM etc” • “Typical” way to solve this: mysql-5.7.5 (osm) > SELECT way_id as id, v FROM way_tags WHERE v like '%gloucester%'; 425 rows in set (0.46 sec) 70 Too slow
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Let’s Add an Index mysql-5.7.5 (osm) > ALTER TABLE way_tags ADD INDEX(v); Query OK, 0 rows affected (6.44 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT ...; 425 rows in set (0.38 sec) • Still slow, why? mysql-5.7.5 (osm) > EXPLAIN SELECT 'way' as type, way_id as id, v FROM way_tags WHERE v like '%gloucester%'; +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | way_tags | NULL | index | NULL | v_2 | 767 | NULL | 1333338 | 11.11 | Using where; Using index | +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ 1 row in set, 1 warning (0.01 sec) 71
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Fulltext Index mysql-5.7.5 (osm) > ALTER TABLE way_tags add FULLTEXT index(v); Query OK, 0 rows affected (3.20 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT ...; 425 rows in set (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT 'way' as type, way_id as id, v FROM way_tags WHERE MATCH(v) AGAINST ('+gloucester*' IN BOOLEAN MODE); +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ | 1 | SIMPLE | way_tags | NULL | fulltext | v | v | 0 | const | 1 | 100.00 | Using where; Ft_hints: no_ranking | +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ 1 row in set, 1 warning (0.00 sec) 72
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Newer Fulltext Optimizations mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM way_tags_myisam WHERE MATCH(v) AGAINST('gloucester'); +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | way_tags_myisam | fulltext | v | v | 0 | | 1 | Using where | +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec) mysql-5.5.40 (osm) > SHOW STATUS like 'Hand%'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 0 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 0 | | Handler_read_last | 0 | | Handler_read_next | 425 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 16 rows in set (0.00 sec) © 2014 DBAHIRE.COM 73
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Newer Fulltext Optimizations (cont.) mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM way_tags WHERE MATCH(v) AGAINST('gloucester'); +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ | 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away | +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > SHOW STATUS like 'Hand%'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 2 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 0 | | Handler_read_last | 0 | | Handler_read_next | 0 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 18 rows in set (0.00 sec) © 2014 DBAHIRE.COM 74 It’s counting directly from the FULLTEXT index
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Open Issues and Limitations • No postfix support (wildcards) • Simple Ranking (and different from MyISAM) • No stemming support • Some multi-language limitations © 2014 DBAHIRE.COM 75 More on FULLTEXT InnoDB support: http://www.drdobbs.com/database/full-text- search-with-innodb/231902587
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GIS IMPROVEMENTS © 2014 DBAHIRE.COM 76
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Find the Closest Starbucks mysql-5.7.5 (osm) > SET @lat:=51.49353; SET @lon:=-0.18340; mysql-5.7.5 (osm) > SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/10000000 as latitude, sqrt(pow((latitude/10000000 - @lat) * 111257.67, 2) + pow((longitude/10000000 - @lon) * 69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' ORDER BY `distance in metres` ASC LIMIT 1; +-----------+-----------+----------+--------------------+ | node_id | longitude | latitude | distance in metres | +-----------+-----------+----------+--------------------+ | 699693936 | -0.1823 | 51.4945 | 130.9792513096838 | +-----------+-----------+----------+--------------------+ 1 row in set (0.20 sec) 77 You are here
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM This Query is Slow mysql-5.7.5 (osm) > EXPLAIN SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/ 10000000 as latitude, sqrt(pow((latitude/10000000- @lat)*111257.67, 2) + pow((longitude/10000000- @lon)*69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' ORDER BY `distance in metres` ASC LIMIT 1G ********************* 1. row ********************* id: 1 select_type: SIMPLE table: n_t1 partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 832040 filtered: 0.00 Extra: Using where; Using temporary; Using file sort ********************* 2. row ********************* id: 1 select_type: SIMPLE table: n_t2 partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.n_t1.node_id rows: 3 filtered: 1.41 Extra: Using where ********************* 3. row ********************* id: 1 select_type: SIMPLE table: n partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.n_t1.node_id rows: 1 filtered: 100.00 Extra: NULL 3 rows in set, 1 warning (0.00 sec) 78
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Can We Optimize it? • We could add a bounding box: mysql-5.7.5 (osm) > ALTER TABLE nodes add index(latitude, longitude); mysql-5.7.5 (osm) > SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/10000000 as latitude, sqrt(pow((latitude/10000000 - @lat) * 111257.67, 2) + pow((longitude/10000000 - @lon) * 69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' and n.latitude BETWEEN ((@lat - 1000/111257.67) * 10000000) AND ((@lat + 1000/111257.67) * 10000000) and n.longitude BETWEEN ((@lon - 1000/69450.32) * 10000000) AND ((@lon + 1000/69450.32) * 10000000) ORDER BY `distance in metres` ASC LIMIT 1; 79 This is not a square, only an approximation
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks We Create an Index… and Force It mysql-5.7.5 (osm) > EXPLAIN SELECT ...; +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ | id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra | | | | | | | _keys | | len | | | | | +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ | 1 | SIMPLE | n_t1 | NULL | ALL | PRIMARY | NULL | NULL | NULL | 832040 | 0.00 | Using where; | | | | | | | | | | | | | Using temporary; | | | | | | | | | | | | | Using filesort | | 1 | SIMPLE | n | NULL | ref | PRIMARY, | PRIMARY | 8 | osm.n_t1.node_id | 1 | 5.00 | Using where | | | | | | | latitude | | | | | | | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n_t1.node_id | 3 | 1.41 | Using where | +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ 3 rows in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT ... FROM nodes n FORCE INDEX(latitude) ...; +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ | id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra | | | | | | | _keys | | len | | | | | +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ | 1 | SIMPLE | n | NULL | range | latitude | latitude | 8 | NULL | 493666 | 11.11 | Using where; | | | | | | | | | | | | | Using index; | | | | | | | | | | | | | Using filesort | | 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 3 | 1.41 | Using where | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 3 | 1.41 | Using where | +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ 3 rows in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > SELECT ... FROM nodes n FORCE INDEX(latitude) ...; +-----------+-----------+----------+--------------------+ | node_id | longitude | latitude | distance in metres | +-----------+-----------+----------+--------------------+ | 699693936 | -0.1823 | 51.4945 | 130.9792513096838 | +-----------+-----------+----------+--------------------+ 1 row in set (0.09 sec) © 2014 DBAHIRE.COM 80 Still many rows are examined Most of the gain comes from the covering index, not the filtering MySQL ignores the newly created index, why? Performance improvement is not great
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Multiple Range Scans Cannot Be Optimized with BTREE Indexes • We need quadtrees or R-TREE Indexes for indexing in multiple dimensions – The later are implemented in MySQL with the name “SPATIAL indexes”, as they only apply to GIS types • Spatial indexing is available for the first time for InnoDB tables on MySQL 5.7.5 81
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Creating a Spatial Index mysql-5.7.5 (osm) > ALTER TABLE nodes ADD COLUMN coord GEOMETRY NOT NULL; Query OK, 0 rows affected (21.80 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > UPDATE nodes SET coord = point(longitude/1000000, latitude/10000000); Query OK, 2865312 rows affected (34.66 sec) Rows matched: 2865312 Changed: 2865312 Warnings: 0 mysql-5.7.5 (osm) > ALTER TABLE nodes add SPATIAL index(coord); Query OK, 0 rows affected (1 min 50.00 sec) Records: 0 Duplicates: 0 Warnings: 0 82 This is new in 5.7
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM New Query mysql> SET @area := envelope(linestring(POINT(@lon - 500/69450.32, @lat - 500/111257.67), POINT(@lon + 500/69450.32, @lat + 500/111257.67))); mysql> SELECT n.node_id, x(n.coord) as longitude, y(n.coord) as latitude, st_distance(POINT(@lon, @lat), coord) as distance FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' and st_within(coord, @area) ORDER BY st_distance(POINT(@lon, @lat), coord) ASC LIMIT 1; 83 We can use any shape we want thanks to 5.6 improvements
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Performance mysql-5.7.5 (osm) > SELECT ...; +-----------+------------+------------+-----------------------+ | node_id | longitude | latitude | distance | +-----------+------------+------------+-----------------------+ | 699693936 | -0.1822879 | 51.4944808 | 0.0014631428672541478 | +-----------+------------+------------+-----------------------+ 1 row in set (0.02 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT ...; +----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+ | id | select | table | parti | type | possible | key | key | ref | rows | filtered | Extra | | | _type | | tions | | _keys | | _len | | | | | +----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+ | 1 | SIMPLE | n | NULL | range | PRIMARY | coord | 34 | NULL | 2 | 100.00 | Using where; | | | | | | | ,coord | | | | | | Using filesort | | 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where | | | | | | | | | | node_id | | | | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where | | | | | | | | | | node_id | | | | +----+--------+-------+-------+-------+----------+---------+------+---------+------+---------------------------+ 3 rows in set, 1 warning (0.00 sec) 84 This field is almost useless
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Filtering Not using the index: +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 1 | | Handler_read_key | 1914 | | Handler_read_last | 0 | | Handler_read_next | 1954 | | Handler_read_prev | 0 | | Handler_read_rnd | 1 | | Handler_read_rnd_next | 833426 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 1 | +----------------------------+--------+ 18 rows in set (0.00 sec) Using the BTREE index: +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 274 | | Handler_read_last | 0 | | Handler_read_next | 246540 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+--------+ 18 rows in set (0.00 sec) Using the SPATIAL index: +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 522 | | Handler_read_last | 0 | | Handler_read_next | 5254 | | Handler_read_prev | 0 | | Handler_read_rnd | 259 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 18 rows in set (0.00 sec) 85 mysql-5.7.5 (osm) > SHOW STATUS LIKE 'Hand%';
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Geohash Functions • mysql-5.7.5 (osm) > SELECT ST_GeoHash(@lon, @lat, 10); +----------------------------+ | ST_GeoHash(@lon, @lat, 10) | +----------------------------+ | gcpugy47w3 | +----------------------------+ 1 row in set (0.00 sec) • Useful to index coordinates with a BTREE – It could be specially useful combined with indexed STORED columns (emulating quadtrees) 86 More on Geohashing: http://mysqlserverteam.com/geohash-functions/
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM GeoJSON Functions mysql-5.7.5 (osm) > SELECT nm.v, ST_AsGeoJson(n.coord) FROM node_tags n_t JOIN nodes n USING (node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' AND n_t.v='attraction' AND nm.k='name'; +-----------------------------+--------------------------------------------------------+ | v | ST_AsGeoJson(n.coord) | +-----------------------------+--------------------------------------------------------+ | BedZED | {"type":"Point","coordinates":[-0.1560632,51.3821745]} | | Blewcoat School | {"type":"Point","coordinates":[-0.1360484,51.4983179]} | | Camden / Buck Street Market | {"type":"Point","coordinates":[-0.143193,51.5400398]} | | Camden Lock Village | {"type":"Point","coordinates":[-0.1447181,51.5416552]} | | Wimbledon Windmill | {"type":"Point","coordinates":[-0.2315468,51.4376583]} | . . . 87
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM GeoJSON Functions (cont.) $ ./sandboxes/msb_5_7_5/use osm -B -e "SET SESSION group_concat_max_len = 10000; SELECT CONCAT('{"type":"FeatureCollection", "features":[ ', GROUP_CONCAT(CONCAT('{"type":"Feature", "geometry":', ST_AsGeoJson(n.coord), ',"properties":{"name":"',nm.v,'"}}')), ' ]}') FROM node_tags n_t JOIN nodes n USING (node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' and n_t.v='attraction' AND nm.k='name'" http://geojsonlint.com/ 88
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Open Issues • The SRID can be set and retrieved, but all operations are done in squared euclidean coordinates: 89 mysql-5.7.5 (osm) > SET @p1 := GeomFromText('POINT(-1 51)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SET @p2 := GeomFromText('POINT(0 51)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SET @p3 := GeomFromText('POINT(-1 52)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SELECT srid(@p1); +-----------+ | srid(@p1) | +-----------+ | 4326 | +-----------+ 1 row in set (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p2); +-----------------------+ | st_distance(@p1, @p2) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p3); +-----------------------+ | st_distance(@p1, @p3) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.00 sec)
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks QUERY PROFILING © 2014 DBAHIRE.COM 90
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Which Queries Should I Optimize First? • My two favorite methods: – pt-query-digest – PERFORMANCE_SCHEMA • I prefer pt-query-digest for long-term reports, PERFORMANCE_SCHEMA for more real-time evaluation and fine-tuning – Also, PERFORMANCE_SCHEMA was not “ready” until © 2014 DBAHIRE.COM MySQL 5.6 91
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM pt-query-digest • It is a 3rd party tool written in Perl, originally created by Baron Schwartz • It requires activation of the slow log: – SET GLOBAL slow_query_log = 1; – SET long_query_time = 0; • In Percona Server and MariaDB it provides extra information: – SHOW GLOBAL VARIABLES like 'log_slow_verbosity'; 92 Be careful with extra IO and latency!
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM pt-query-digest Execution $ pt-query-digest ~/sandboxes/msb_5_7_5/data/localhost-slow.log # 190ms user time, 30ms system time, 25.45M rss, 219.82M vsz # Current date: Sat Nov 1 17:26:58 2014 # Hostname: localhost.localdomain # Files: /home/dbahire/sandboxes/msb_5_7_5/data/localhost-slow.log # Overall: 21 total, 7 unique, 0 QPS, 0x concurrency _____________________ # Attribute total min max avg 95% stddev median # ============ ======= ======= ======= ======= ======= ======= ======= # Exec time 7s 6us 482ms 324ms 455ms 201ms 433ms # Lock time 18ms 0 11ms 840us 224us 2ms 113us # Rows sent 7.08k 0 1.22k 345.43 420.77 300.48 420.77 # Rows examine 19.22M 0 1.28M 937.14k 1.26M 581.94k 1.26M # Query size 1.10k 11 67 53.81 65.89 18.23 65.89 # Profile # Rank Query ID Response time Calls R/Call V/M Item # ==== ================== ============= ===== ====== ===== =============== # 1 0x5060CBA490BE371E 6.4236 94.4% 14 0.4588 0.00 SELECT way_tags 93
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks pt-query-digest Execution (cont.) # Query 1: 0 QPS, 0x concurrency, ID 0x5060CBA490BE371E at byte 8795 _____ # This item is included in the report because it matches --limit. # Scores: V/M = 0.00 # Attribute pct total min max avg 95% stddev median # ============ === ======= ======= ======= ======= ======= ======= ======= # Count 66 14 # Exec time 94 6s 435ms 482ms 459ms 477ms 15ms 455ms # Lock time 10 2ms 0 227us 137us 204us 54us 113us # Rows sent 87 6.21k 0 1.22k 453.86 420.77 242.38 420.77 # Rows examine 93 17.94M 1.28M 1.28M 1.28M 1.28M 0 1.28M # Query size 80 912 59 67 65.14 65.89 2.70 65.89 # String: # Databases osm # Hosts localhost # Time 2014-11-01... (1/7%), 2014-11-01... (1/7%)... 12 more # Users msandbox # Query_time distribution # 1us # 10us # 100us # 1ms # 10ms # 100ms ################################################################ # 1s # 10s+ # Tables # SHOW TABLE STATUS FROM `osm` LIKE 'way_tags'G # SHOW CREATE TABLE `osm`.`way_tags`G # EXPLAIN /*!50100 PARTITIONS*/ SELECT way_id as id, v FROM way_tags WHERE v like '%gloucester%'G © 2014 DBAHIRE.COM 94
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM PERFORMANCE_SCHEMA • Monitoring schema enabled by default since MySQL 5.6 – performance_schema = 1 (it is not dynamic) • Deprecates the old query profiling • It is way more user-friendly when combined with the SYS schema/ps_helper (a set of views and stored procedures created by Mark Leith) 95
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Installation of the SYS Schema $ git clone https://github.com/MarkLeith/mysql-sys.git Cloning into 'mysql-sys'... remote: Counting objects: 926, done. remote: Compressing objects: 100% (73/73), done. remote: Total 926 (delta 35), reused 6 (delta 2) Receiving objects: 100% (926/926), 452.19 KiB | 225.00 KiB/s, done. Resolving deltas: 100% (584/584), done. $ cd mysql-sys/ $ ~/sandboxes/msb_5_7_5/use < sys_57.sql © 2014 DBAHIRE.COM 96
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Discovering Unused © 2014 DBAHIRE.COM Indexes mysql-5.7.5 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5; +---------------+-------------------+---------------------------------+ | object_schema | object_name | index_name | +---------------+-------------------+---------------------------------+ | osm | acls | acls_k_idx | | osm | changeset_tags | changeset_tags_id_idx | | osm | current_nodes | current_nodes_timestamp_idx | | osm | current_nodes | current_nodes_tile_idx | | osm | current_relations | current_relations_timestamp_idx | +---------------+-------------------+---------------------------------+ 5 rows in set (0.04 sec) mysql-5.7.5 (osm) > SELECT * FROM current_nodes WHERE tile = 100; ... mysql-5.7.5 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5; +---------------+-------------------+---------------------------------+ | object_schema | object_name | index_name | +---------------+-------------------+---------------------------------+ | osm | acls | acls_k_idx | | osm | changeset_tags | changeset_tags_id_idx | | osm | current_nodes | current_nodes_timestamp_idx | | osm | current_relations | current_relations_timestamp_idx | | osm | current_relations | changeset_id | +---------------+-------------------+---------------------------------+ 5 rows in set (0.03 sec) 97 With enough activity, it can help us clean up our schema
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Slow Queries (ordered by © 2014 DBAHIRE.COM server time) ********* 1. row ********* query: SELECT `way_id` AS `id` , `v` FROM `way_tags` WHERE `v` LIKE ? db: osm full_scan: * exec_count: 15 err_count: 0 warn_count: 0 total_latency: 7.83 s max_latency: 1.33 s avg_latency: 521.84 ms lock_latency: 17.94 ms rows_sent: 6779 rows_sent_avg: 452 rows_examined: 20152155 rows_examined_avg: 1343477 rows_affected: 0 rows_affected_avg: 0 tmp_tables: 0 tmp_disk_tables: 0 rows_sorted: 0 sort_merge_passes: 0 digest: 21f90695b1ebf20a5f4d4c1e5e860f58 first_seen: 2014-11-01 17:04:51 last_seen: 2014-11-01 17:05:22 98 mysql-5.7.5 (osm) > SELECT * FROM sys.statement_analysis LIMIT 10G
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables mysql-5.7.5 (osm) > SELECT * FROM sys.statements_with_temp_tables WHERE db = 'osm' LIMIT 10G *************************** 1. row *************************** query: SELECT ? AS TYPE , `node_id` A ... gs` WHERE `k` = ? AND `v` = ? db: osm exec_count: 11 total_latency: 7.57 s memory_tmp_tables: 11 disk_tmp_tables: 0 avg_tmp_tables_per_query: 1 tmp_tables_to_disk_pct: 0 first_seen: 2014-11-01 17:33:55 last_seen: 2014-11-01 17:34:45 digest: 5e6e82799b7c7c0e5c57cfe63eb98d5d 99
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables (cont.) mysql-5.7.5 (osm) > SELECT DIGEST_TEXT FROM performance_schema.events_statements_summary_by_digest WHERE digest = '5e6e82799b7c7c0e5c57cfe63eb98d5d'G *************************** 1. row *************************** DIGEST_TEXT: SELECT ? AS TYPE , `node_id` AS `id` FROM `node_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS TYPE , `way_id` AS `id` FROM `way_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS TYPE , `relation_id` AS `id` FROM `relation_tags` WHERE `k` = ? AND `v` = ? 1 row in set (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'; +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ | 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where | | 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where | | 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where | | NULL | UNION RESULT | <union1,2,3> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary | +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ 4 rows in set, 1 warning (0.01 sec) 100
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables (cont.) mysql-5.7.5 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION ALL SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION ALL SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'; +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where | | 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where | | 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where | +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ 3 rows in set, 1 warning (0.04 sec) 101 No Temporary tables now in 5.7
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks CONCLUSIONS © 2014 DBAHIRE.COM 102
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM 5.7/10.1 Still in Alpha • New features usually involve new bugs – I only suffered a single crashing bug – Some of the bleeding edge features are not yet optimized for performance and usability • The greatest problem is finding documentation, as some features are only documented on the developer’s blogs – It will probably catch up slowly • It is great to have the lab builds, though, for testing 103
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL 5.7 New Features • MySQL 5.6 seemed Percona Server-inspired • MySQL 5.7 seems MariaDB-inspired – Competition is always good for consumer 104
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Many Optimizer Advantages Have to Be © 2014 DBAHIRE.COM Manually Enabled • Modifying on a per-query basis: SET optimizer_switch='batched_key_access=on'; SET join_cache_level=8; # for MariaDB in order to take advantage of them make the features useless unless you are fine-tuning – I expect that to change in the future 105
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks I Herby Declare MyISAM as Dead • All major MyISAM-only features are now on MySQL 5.7 – FULLTEXT – GIS – Transportable tables • There are still reasons to use MyISAM – MyISAM is still required for the mysql © 2014 DBAHIRE.COM schema and non-durable temporary tables (WIP) 106
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Benchmarks • Do not trust first party benchmarks – In fact, do not trust 3rd party benchmarks either • Only care about the performance of your application running on your hardware 107
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Q&A 108
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Knowing More about Query Optimization • Tomorrow: Bill Karwin’s session “Query Patterns, Optimized” • “Optimization, Administration and High Availability with MySQL 5.6” course by DBAHire.com on 15th December (Spain) – More dates coming soon © 2014 DBAHIRE.COM 109
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Talk to me During the Percona Live • Talk to me about: – MySQL/MariaDB/Percona – OpenStreetMap – Game of Thrones and I will offer you 1 hour of free MySQL © 2014 DBAHIRE.COM online consulting by myself* 110 *Subjected to availability
Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Thank You for Attending! • Special thanks to -in ORDER BY rand(): Bill Karwin, Morgan Tocker, Baron Schwartz, Giuseppe Maxia, Ewen Fortune, Henrik Ingo, Tim “do-not- confuse-with-Mark” Callaghan, David Hildebrandt, Cédric Peintre, Kortney Runyan, Miguel Ángel Nieto, Valerii Kravchuk, Aleksandr Kuzminsky and everyone at the Percona and Oracle-MySQL teams and the Percona Live London organization, committee and sponsors • Do not forget, after the session finishes, to please login with your Percona Live account and go to http:// www.percona.com/live/london-2014/sessions/query-optimization- mysql-57-and-mariadb-10-even-newer-tricks to “Rate This Session” for a chance to win great prizes! 111

Recommended

Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
 
ProxySQL High Avalability and Configuration Management Overview
ProxySQL High Avalability and Configuration Management OverviewProxySQL High Avalability and Configuration Management Overview
ProxySQL High Avalability and Configuration Management OverviewRené Cannaò
 
Maxscale 소개 1.1.1
Maxscale 소개 1.1.1Maxscale 소개 1.1.1
Maxscale 소개 1.1.1NeoClova
 
Galera cluster for high availability
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability Mydbops
 
Getting started with postgresql
Getting started with postgresqlGetting started with postgresql
Getting started with postgresqlbotsplash.com
 
MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZE
MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZEMySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZE
MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZENorvald Ryeng
 
Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsQuery Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsJaime Crespo
 
MySQL_MariaDB-성능개선-202201.pptx
MySQL_MariaDB-성능개선-202201.pptxMySQL_MariaDB-성능개선-202201.pptx
MySQL_MariaDB-성능개선-202201.pptxNeoClova
 

Recommended

Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
 
ProxySQL High Avalability and Configuration Management Overview
ProxySQL High Avalability and Configuration Management OverviewProxySQL High Avalability and Configuration Management Overview
ProxySQL High Avalability and Configuration Management OverviewRené Cannaò
 
Maxscale 소개 1.1.1
Maxscale 소개 1.1.1Maxscale 소개 1.1.1
Maxscale 소개 1.1.1NeoClova
 
Galera cluster for high availability
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability Mydbops
 
Getting started with postgresql
Getting started with postgresqlGetting started with postgresql
Getting started with postgresqlbotsplash.com
 
MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZE
MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZEMySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZE
MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZENorvald Ryeng
 
Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsQuery Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsJaime Crespo
 
MySQL_MariaDB-성능개선-202201.pptx
MySQL_MariaDB-성능개선-202201.pptxMySQL_MariaDB-성능개선-202201.pptx
MySQL_MariaDB-성능개선-202201.pptxNeoClova
 
Mongo DB 완벽가이드 - 4장 쿼리하기
Mongo DB 완벽가이드 - 4장 쿼리하기Mongo DB 완벽가이드 - 4장 쿼리하기
Mongo DB 완벽가이드 - 4장 쿼리하기JangHyuk You
 
ProxySQL for MySQL
ProxySQL for MySQLProxySQL for MySQL
ProxySQL for MySQLMydbops
 
MariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB plc
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performanceoysteing
 
MySQL Database Architectures - InnoDB ReplicaSet & Cluster
MySQL Database Architectures - InnoDB ReplicaSet & ClusterMySQL Database Architectures - InnoDB ReplicaSet & Cluster
MySQL Database Architectures - InnoDB ReplicaSet & ClusterKenny Gryp
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
 
Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Mydbops
 
Proxysql sharding
Proxysql shardingProxysql sharding
Proxysql shardingMarco Tusa
 
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL ShellMySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL ShellMiguel Araújo
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresKarwin Software Solutions LLC
 
Advanced Postgres Monitoring
Advanced Postgres MonitoringAdvanced Postgres Monitoring
Advanced Postgres MonitoringDenish Patel
 
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11Kenny Gryp
 
Evolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesEvolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesMydbops
 
binary log と 2PC と Group Commit
binary log と 2PC と Group Commitbinary log と 2PC と Group Commit
binary log と 2PC と Group CommitTakanori Sejima
 
Advanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suiteAdvanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp
 
MySQL Timeout Variables Explained
MySQL Timeout Variables Explained MySQL Timeout Variables Explained
MySQL Timeout Variables Explained Mydbops
 
Innodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライドInnodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライドYasufumi Kinoshita
 
MySQL Query And Index Tuning
MySQL Query And Index TuningMySQL Query And Index Tuning
MySQL Query And Index TuningManikanda kumar
 
Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Mydbops
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Mydbops
 
MySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD TourMySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD TourRonald Bradford
 
MySQL Server Defaults
MySQL Server DefaultsMySQL Server Defaults
MySQL Server DefaultsMorgan Tocker
 

More Related Content

What's hot

Mongo DB 완벽가이드 - 4장 쿼리하기
Mongo DB 완벽가이드 - 4장 쿼리하기Mongo DB 완벽가이드 - 4장 쿼리하기
Mongo DB 완벽가이드 - 4장 쿼리하기JangHyuk You
 
ProxySQL for MySQL
ProxySQL for MySQLProxySQL for MySQL
ProxySQL for MySQLMydbops
 
MariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB plc
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performanceoysteing
 
MySQL Database Architectures - InnoDB ReplicaSet & Cluster
MySQL Database Architectures - InnoDB ReplicaSet & ClusterMySQL Database Architectures - InnoDB ReplicaSet & Cluster
MySQL Database Architectures - InnoDB ReplicaSet & ClusterKenny Gryp
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
 
Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Mydbops
 
Proxysql sharding
Proxysql shardingProxysql sharding
Proxysql shardingMarco Tusa
 
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL ShellMySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL ShellMiguel Araújo
 
Advanced Postgres Monitoring
Advanced Postgres MonitoringAdvanced Postgres Monitoring
Advanced Postgres MonitoringDenish Patel
 
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11Kenny Gryp
 
Evolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesEvolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesMydbops
 
binary log と 2PC と Group Commit
binary log と 2PC と Group Commitbinary log と 2PC と Group Commit
binary log と 2PC と Group CommitTakanori Sejima
 
Advanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suiteAdvanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp
 
MySQL Timeout Variables Explained
MySQL Timeout Variables Explained MySQL Timeout Variables Explained
MySQL Timeout Variables Explained Mydbops
 
Innodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライドInnodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライドYasufumi Kinoshita
 
MySQL Query And Index Tuning
MySQL Query And Index TuningMySQL Query And Index Tuning
MySQL Query And Index TuningManikanda kumar
 
Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Mydbops
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Mydbops
 

What's hot (20)

Mongo DB 완벽가이드 - 4장 쿼리하기
Mongo DB 완벽가이드 - 4장 쿼리하기Mongo DB 완벽가이드 - 4장 쿼리하기
Mongo DB 완벽가이드 - 4장 쿼리하기
 
ProxySQL for MySQL
ProxySQL for MySQLProxySQL for MySQL
ProxySQL for MySQL
 
MariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & OptimizationMariaDB Server Performance Tuning & Optimization
MariaDB Server Performance Tuning & Optimization
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
MySQL Database Architectures - InnoDB ReplicaSet & Cluster
MySQL Database Architectures - InnoDB ReplicaSet & ClusterMySQL Database Architectures - InnoDB ReplicaSet & Cluster
MySQL Database Architectures - InnoDB ReplicaSet & Cluster
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
 
Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )Percona XtraDB Cluster ( Ensure high Availability )
Percona XtraDB Cluster ( Ensure high Availability )
 
Proxysql sharding
Proxysql shardingProxysql sharding
Proxysql sharding
 
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL ShellMySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick Figures
 
Advanced Postgres Monitoring
Advanced Postgres MonitoringAdvanced Postgres Monitoring
Advanced Postgres Monitoring
 
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
 
Evolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best PracticesEvolution of MongoDB Replicaset and Its Best Practices
Evolution of MongoDB Replicaset and Its Best Practices
 
binary log と 2PC と Group Commit
binary log と 2PC と Group Commitbinary log と 2PC と Group Commit
binary log と 2PC と Group Commit
 
Advanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suiteAdvanced Percona XtraDB Cluster in a nutshell... la suite
Advanced Percona XtraDB Cluster in a nutshell... la suite
 
MySQL Timeout Variables Explained
MySQL Timeout Variables Explained MySQL Timeout Variables Explained
MySQL Timeout Variables Explained
 
Innodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライドInnodb Deep Talk #2 でお話したスライド
Innodb Deep Talk #2 でお話したスライド
 
MySQL Query And Index Tuning
MySQL Query And Index TuningMySQL Query And Index Tuning
MySQL Query And Index Tuning
 
Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication Evolution of MySQL Parallel Replication
Evolution of MySQL Parallel Replication
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0
 

Viewers also liked

MySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD TourMySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD TourRonald Bradford
 
MySQL Server Defaults
MySQL Server DefaultsMySQL Server Defaults
MySQL Server DefaultsMorgan Tocker
 
Hbase源码初探
Hbase源码初探Hbase源码初探
Hbase源码初探zhaolinjnu
 
What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...Sveta Smirnova
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQLMySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQLOlivier DASINI
 
Mysql参数-GDB
Mysql参数-GDBMysql参数-GDB
Mysql参数-GDBzhaolinjnu
 
MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!Vitor Oliveira
 
MySQL High Availability with Group Replication
MySQL High Availability with Group ReplicationMySQL High Availability with Group Replication
MySQL High Availability with Group ReplicationNuno Carvalho
 
Capturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQLCapturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQLRonald Bradford
 
MySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
MySQL InnoDB Cluster and Group Replication - OSI 2017 BangaloreMySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
MySQL InnoDB Cluster and Group Replication - OSI 2017 BangaloreSujatha Sivakumar
 
MySQL High Availability and Disaster Recovery with Continuent, a VMware company
MySQL High Availability and Disaster Recovery with Continuent, a VMware companyMySQL High Availability and Disaster Recovery with Continuent, a VMware company
MySQL High Availability and Disaster Recovery with Continuent, a VMware companyContinuent
 
Using Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisUsing Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisSveta Smirnova
 
MySQL InnoDB 源码实现分析(一)
MySQL InnoDB 源码实现分析(一)MySQL InnoDB 源码实现分析(一)
MySQL InnoDB 源码实现分析(一)frogd
 
淘宝数据库架构演进历程
淘宝数据库架构演进历程淘宝数据库架构演进历程
淘宝数据库架构演进历程zhaolinjnu
 
Inno db internals innodb file formats and source code structure
Inno db internals innodb file formats and source code structureInno db internals innodb file formats and source code structure
Inno db internals innodb file formats and source code structurezhaolinjnu
 

Viewers also liked (20)

MySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD TourMySQL Best Practices - OTN LAD Tour
MySQL Best Practices - OTN LAD Tour
 
MySQL Server Defaults
MySQL Server DefaultsMySQL Server Defaults
MySQL Server Defaults
 
Hbase源码初探
Hbase源码初探Hbase源码初探
Hbase源码初探
 
SQL Outer Joins for Fun and Profit
SQL Outer Joins for Fun and ProfitSQL Outer Joins for Fun and Profit
SQL Outer Joins for Fun and Profit
 
What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...
 
Load Data Fast!
Load Data Fast!Load Data Fast!
Load Data Fast!
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQLMySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
 
Mysql参数-GDB
Mysql参数-GDBMysql参数-GDB
Mysql参数-GDB
 
MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!MySQL Replication Performance Tuning for Fun and Profit!
MySQL Replication Performance Tuning for Fun and Profit!
 
MySQL High Availability with Group Replication
MySQL High Availability with Group ReplicationMySQL High Availability with Group Replication
MySQL High Availability with Group Replication
 
Capturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQLCapturing, Analyzing and Optimizing MySQL
Capturing, Analyzing and Optimizing MySQL
 
MySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
MySQL InnoDB Cluster and Group Replication - OSI 2017 BangaloreMySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
MySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
 
MySQL High Availability and Disaster Recovery with Continuent, a VMware company
MySQL High Availability and Disaster Recovery with Continuent, a VMware companyMySQL High Availability and Disaster Recovery with Continuent, a VMware company
MySQL High Availability and Disaster Recovery with Continuent, a VMware company
 
Using Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisUsing Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data Analysis
 
MySQL InnoDB 源码实现分析(一)
MySQL InnoDB 源码实现分析(一)MySQL InnoDB 源码实现分析(一)
MySQL InnoDB 源码实现分析(一)
 
淘宝数据库架构演进历程
淘宝数据库架构演进历程淘宝数据库架构演进历程
淘宝数据库架构演进历程
 
Inno db internals innodb file formats and source code structure
Inno db internals innodb file formats and source code structureInno db internals innodb file formats and source code structure
Inno db internals innodb file formats and source code structure
 
Requirements the Last Bottleneck
Requirements the Last BottleneckRequirements the Last Bottleneck
Requirements the Last Bottleneck
 
Extensible Data Modeling
Extensible Data ModelingExtensible Data Modeling
Extensible Data Modeling
 
MySQL aio
MySQL aioMySQL aio
MySQL aio
 

Similar to Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks

NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoNoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoData Con LA
 
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)Valeriy Kravchuk
 
Confoo 2021 -- MySQL New Features
Confoo 2021 -- MySQL New FeaturesConfoo 2021 -- MySQL New Features
Confoo 2021 -- MySQL New FeaturesDave Stokes
 
Common schema my sql uc 2012
Common schema my sql uc 2012Common schema my sql uc 2012
Common schema my sql uc 2012Roland Bouman
 
Common schema my sql uc 2012
Common schema my sql uc 2012Common schema my sql uc 2012
Common schema my sql uc 2012Roland Bouman
 
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenOSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenNETWAYS
 
What's New In MySQL 5.6
What's New In MySQL 5.6What's New In MySQL 5.6
What's New In MySQL 5.6Abdul Manaf
 
Memcached Functions For My Sql Seemless Caching In My Sql
Memcached Functions For My Sql Seemless Caching In My SqlMemcached Functions For My Sql Seemless Caching In My Sql
Memcached Functions For My Sql Seemless Caching In My SqlMySQLConference
 
MySQL Best Practices - OTN
MySQL Best Practices - OTNMySQL Best Practices - OTN
MySQL Best Practices - OTNRonald Bradford
 
My sql 5.7-upcoming-changes-v2
My sql 5.7-upcoming-changes-v2My sql 5.7-upcoming-changes-v2
My sql 5.7-upcoming-changes-v2Morgan Tocker
 
Optimizer overviewoow2014
Optimizer overviewoow2014Optimizer overviewoow2014
Optimizer overviewoow2014Mysql User Camp
 
Performance schema and sys schema
Performance schema and sys schemaPerformance schema and sys schema
Performance schema and sys schemaMark Leith
 
DB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQLDB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQLMarcelo Altmann
 
MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012Colin Charles
 
MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 MinutesSveta Smirnova
 
Query Optimization with MySQL 5.6: Old and New Tricks
Query Optimization with MySQL 5.6: Old and New TricksQuery Optimization with MySQL 5.6: Old and New Tricks
Query Optimization with MySQL 5.6: Old and New TricksMYXPLAIN
 
Advanced Query Optimizer Tuning and Analysis
Advanced Query Optimizer Tuning and AnalysisAdvanced Query Optimizer Tuning and Analysis
Advanced Query Optimizer Tuning and AnalysisMYXPLAIN
 

Similar to Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks (20)

NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoNoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
 
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
MariaDB 10.5 new features for troubleshooting (mariadb server fest 2020)
 
Confoo 2021 -- MySQL New Features
Confoo 2021 -- MySQL New FeaturesConfoo 2021 -- MySQL New Features
Confoo 2021 -- MySQL New Features
 
Common schema my sql uc 2012
Common schema my sql uc 2012Common schema my sql uc 2012
Common schema my sql uc 2012
 
Common schema my sql uc 2012
Common schema my sql uc 2012Common schema my sql uc 2012
Common schema my sql uc 2012
 
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert VanderkelenOSMC 2008 | Monitoring MySQL by Geert Vanderkelen
OSMC 2008 | Monitoring MySQL by Geert Vanderkelen
 
What's New In MySQL 5.6
What's New In MySQL 5.6What's New In MySQL 5.6
What's New In MySQL 5.6
 
Memcached Functions For My Sql Seemless Caching In My Sql
Memcached Functions For My Sql Seemless Caching In My SqlMemcached Functions For My Sql Seemless Caching In My Sql
Memcached Functions For My Sql Seemless Caching In My Sql
 
MySQL Best Practices - OTN
MySQL Best Practices - OTNMySQL Best Practices - OTN
MySQL Best Practices - OTN
 
MySQLinsanity
MySQLinsanityMySQLinsanity
MySQLinsanity
 
My sql 5.7-upcoming-changes-v2
My sql 5.7-upcoming-changes-v2My sql 5.7-upcoming-changes-v2
My sql 5.7-upcoming-changes-v2
 
Optimizer overviewoow2014
Optimizer overviewoow2014Optimizer overviewoow2014
Optimizer overviewoow2014
 
Performance schema and sys schema
Performance schema and sys schemaPerformance schema and sys schema
Performance schema and sys schema
 
DB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQLDB Floripa - ProxySQL para MySQL
DB Floripa - ProxySQL para MySQL
 
MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012
 
MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes
 
Query Optimization with MySQL 5.6: Old and New Tricks
Query Optimization with MySQL 5.6: Old and New TricksQuery Optimization with MySQL 5.6: Old and New Tricks
Query Optimization with MySQL 5.6: Old and New Tricks
 
ProxySQL para mysql
ProxySQL para mysqlProxySQL para mysql
ProxySQL para mysql
 
Advanced Query Optimizer Tuning and Analysis
Advanced Query Optimizer Tuning and AnalysisAdvanced Query Optimizer Tuning and Analysis
Advanced Query Optimizer Tuning and Analysis
 
Perf Tuning Short
Perf Tuning ShortPerf Tuning Short
Perf Tuning Short
 

More from Jaime Crespo

Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Jaime Crespo
 
Backing up Wikipedia Databases
Backing up Wikipedia DatabasesBacking up Wikipedia Databases
Backing up Wikipedia DatabasesJaime Crespo
 
Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017Jaime Crespo
 
Scaling and hardware provisioning for databases (lessons learned at wikipedia)
Scaling and hardware provisioning for databases (lessons learned at wikipedia)Scaling and hardware provisioning for databases (lessons learned at wikipedia)
Scaling and hardware provisioning for databases (lessons learned at wikipedia)Jaime Crespo
 
MySQL Schema Design in Practice
MySQL Schema Design in PracticeMySQL Schema Design in Practice
MySQL Schema Design in PracticeJaime Crespo
 
MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation
MySQL at Wikipedia: How we do relational data at the Wikimedia FoundationMySQL at Wikipedia: How we do relational data at the Wikimedia Foundation
MySQL at Wikipedia: How we do relational data at the Wikimedia FoundationJaime Crespo
 
Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)Jaime Crespo
 
MySQL GUI Administration
MySQL GUI AdministrationMySQL GUI Administration
MySQL GUI AdministrationJaime Crespo
 
Software libre para gestión de proyectos
Software libre para gestión de proyectosSoftware libre para gestión de proyectos
Software libre para gestión de proyectosJaime Crespo
 
warptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidadwarptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidadJaime Crespo
 

More from Jaime Crespo (11)

Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
 
Backing up Wikipedia Databases
Backing up Wikipedia DatabasesBacking up Wikipedia Databases
Backing up Wikipedia Databases
 
Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017
 
Scaling and hardware provisioning for databases (lessons learned at wikipedia)
Scaling and hardware provisioning for databases (lessons learned at wikipedia)Scaling and hardware provisioning for databases (lessons learned at wikipedia)
Scaling and hardware provisioning for databases (lessons learned at wikipedia)
 
MySQL Schema Design in Practice
MySQL Schema Design in PracticeMySQL Schema Design in Practice
MySQL Schema Design in Practice
 
MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation
MySQL at Wikipedia: How we do relational data at the Wikimedia FoundationMySQL at Wikipedia: How we do relational data at the Wikimedia Foundation
MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation
 
Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)
 
Why MySQL
Why MySQLWhy MySQL
Why MySQL
 
MySQL GUI Administration
MySQL GUI AdministrationMySQL GUI Administration
MySQL GUI Administration
 
Software libre para gestión de proyectos
Software libre para gestión de proyectosSoftware libre para gestión de proyectos
Software libre para gestión de proyectos
 
warptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidadwarptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidad
 

Recently uploaded

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfPrecisely
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostZilliz
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 

Recently uploaded (20)

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfHyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage CostLeverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 

Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks

  • 1. DBAHire.com Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Jaime Crespo Percona Live London 2014 -3 Nov 2014- dbahire.com/pluk14 1
  • 2. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Agenda - 3 hours 2 1. Introduction 8. Subquery Optimization 2. Break (VM installation) 9. Fulltext Search 3. General Optimizer Improvements 10. GIS Improvements 4. Computed/Virtual Columns 11. Break 5. Query Rewrite Plugins 12. Query Profiling 6. SQL Mode Changes 13. Results and Conclusions 7. JOIN Optimization 14. Q&A
  • 3. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks INTRODUCTION © 2014 DBAHIRE.COM 3
  • 4. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks This is Me Fighting Bad Query Performance © 2014 DBAHIRE.COM • MySQL Consultant at DBAHire.com • Used to work for Oracle (MySQL), Percona • Loves MySQL query optimization and HA 4
  • 5. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL Versions • Oracle MySQL 5.6 (GA) almost 2 years old • MariaDB 10 over 7 months old • Several lab/alpha releases of MySQL 5.7 and MariaDB 10.1 with great new features – We will focus only on SQL-related ones – We will not perform formal benchmarks or comparisons 5
  • 6. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks New Query Optimization Features • Index Condition Pushdown • Subquery Optimizations (materialization and semijoin) • IN-to-EXISTS • JOIN-to-WHERE • Multi-Range Read • Batched Key Access • Persistent InnoDB Statistics • EXPLAIN FORMAT=JSON • EXPLAIN INSERT/UPDATE/DELETE • Hash Join • New cost-based optimizer • Optimizer Trace • Filesort with a short LIMIT © 2014 DBAHIRE.COM • EXISTS-to-IN • EXPLAIN FOR CONNECTION • Faster eq_ref • Extended secondary keys • Engine-Independent statistics • Improved Spatial Indexes (GIS) • Histograms • Virtual columns • PERFORMANCE_SCHEMA improvements • Forced strict SQL mode; functional dependencies in GROUP BY • Query rewrite plugin • Configurable optimizer_search_depth 6
  • 7. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks I Already Mentioned Some of Those Topics © 2014 DBAHIRE.COM One Year Ago • Check my presentation here: http://www.slideshare.net/jynus/query-optimization56 7
  • 8. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Example Application • OSM Database is licensed under the Open DataBase License and is Copyright OpenStreetMap Contributors 8
  • 9. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GENERAL OPTIMIZER IMPROVEMENTS © 2014 DBAHIRE.COM 9
  • 10. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM EXPLAIN FOR CONNECTION (SHOW EXPLAIN FOR) mysql-5.7.5 (osm)> SHOW PROCESSLIST; +----+------+-----------+------+---------+------+-----------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------+------+---------+------+-----------+------------------+ | 18 | root | localhost | osm | Query | 1867 | updating | UPDATE nodes n | | | | | | | | reference | JOIN users u | | | | | | | | tables | ON n.user=u.name | | | | | | | | | SET n.user=u.id | | 19 | root | localhost | NULL | Query | 0 | starting | SHOW PROCESSLIST | +----+------+-----------+------+---------+------+-----------+------------------+ 2 rows in set (0.00 sec) mysql-5.7.5 (osm)> EXPLAIN FOR CONNECTION 18; +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ | id | select | table | parti- | type | possible | key | key | ref | rows | filte- | Extra | | | _type | | tions | | _keys | | _len | | | red | | +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ | 1 | SIMPLE | u | NULL | index | name | name | 258 | NULL | 11271 | 100.00 | Using where; | | | | | | | | | | | | | Using index | | 1 | UPDATE | n | NULL | ref | user | user | 258 | osm.u. | 280 | 100.00 | NULL | | | | | | | | | name | | | | | +----+--------+-------+--------+-------+----------+------+------+--------+-------+--------+--------------+ 2 rows in set (0.00 sec) 10 Use “SHOW EXPLAIN FOR #” with MariaDB 10
  • 11. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks MySQL 5.7.5 Introduced an New Cost- © 2014 DBAHIRE.COM Based Model for Its Optimizer mysql-5.5.40 (osm) > SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G 59 rows in set (1.37 sec) 11 mysql-5.5.40 (osm) > SELECT STRAIGHT_JOIN * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G 59 rows in set (0.86 sec) This condition is very selective, but there is no index available
  • 12. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Why does it take the wrong table order? mysql-5.5.40 (osm) > EXPLAIN EXTENDED SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** © 2014 DBAHIRE.COM id: 1 select_type: SIMPLE table: node_tags type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 839031 filtered: 100.00 Extra: *************************** 2. row **************************** id: 1 select_type: SIMPLE table: nodes type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.node_tags.node_id rows: 1 filtered: 100.00 Extra: Using where 2 rows in set, 1 warning (0.00 sec) 12
  • 13. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Unindexed Columns Are not Accounted mysql-5.5.40 (osm) > EXPLAIN EXTENDED SELECT STRAIGHT_JOIN * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** © 2014 DBAHIRE.COM id: 1 select_type: SIMPLE table: nodes type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2865521 filtered: 100.00 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 1 filtered: 100.00 Extra: 2 rows in set, 1 warning (0.00 sec) 13 The optimizer assumes that all rows will be returned
  • 14. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM What’s new in 5.7? mysql-5.7.5 (osm) > EXPLAIN SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2773853 filtered: 11.11 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 3 filtered: 100.00 Extra: NULL 2 rows in set, 1 warning (0.00 sec) 14 It is actually 0.002%, but it is good enough
  • 15. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM If things go wrong -our Friend optimizer_switch mysql-5.7.5 (osm) > SHOW VARIABLES like 'optimizer_switch'G ******************* 1. row ******************* Variable_name: optimizer_switch Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on, index_merge_intersection=on,engine_condition_pushdown=on,index _condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_l oop=on,batched_key_access=off,materialization=on,semijoin=on,l oosescan=on,firstmatch=on,subquery_materialization_cost_based= on,use_index_extensions=on,condition_fanout_filter=on 1 row in set (0.00 sec) 15
  • 16. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Improved EXPLAIN FORMAT=JSON "nested_loop": [ © 2014 DBAHIRE.COM { "table": { "table_name": "nodes", "access_type": "ALL", "possible_keys": [ "PRIMARY" ], "rows_examined_per_scan": 2773853, "rows_produced_per_join": 308175, "filtered": 11.11, "cost_info": { "read_cost": "512783.58", "eval_cost": "61635.01", "prefix_cost": "574418.60", "data_read_per_join": "21M" }, 16 Cost information is now included Engine statistics are now floats for improved precision
  • 17. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Optimizer Cost Tuning 17 This point depends on the hardware and buffer status
  • 18. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Configurable Costs mysql-5.7.5 ((none))> SELECT * FROM mysql.server_cost; +------------------------------+------------+---------------------+---------+ | cost_name | cost_value | last_update | comment | +------------------------------+------------+---------------------+---------+ | disk_temptable_create_cost | NULL | 2014-10-21 23:03:30 | NULL | | disk_temptable_row_cost | NULL | 2014-10-21 23:03:30 | NULL | | key_compare_cost | NULL | 2014-10-21 23:03:30 | NULL | | memory_temptable_create_cost | NULL | 2014-10-21 23:03:30 | NULL | | memory_temptable_row_cost | NULL | 2014-10-21 23:03:30 | NULL | | row_evaluate_cost | NULL | 2014-10-21 23:03:30 | NULL | +------------------------------+------------+---------------------+---------+ 6 rows in set (0.00 sec) mysql-5.7.5 ((none))> SELECT * FROM mysql.engine_cost; +-------------+-------------+--------------------+------------+---------------------+---------+ | engine_name | device_type | cost_name | cost_value | last_update | comment | +-------------+-------------+--------------------+------------+---------------------+---------+ | default | 0 | io_block_read_cost | NULL | 2014-10-21 23:03:30 | NULL | +-------------+-------------+--------------------+------------+---------------------+---------+ 1 row in set (0.00 sec) 18 The unit is “read of a random data page”
  • 19. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Changing Costs Example mysql-5.7.5 (osm) > EXPLAIN FORMAT=JSON SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'G "cost_info": { "query_cost": "22567.80" 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > UPDATE mysql.server_cost SET cost_value = 10; mysql-5.7.5 (osm) > FLUSH OPTIMIZER_COSTS; <session restart> mysql-5.7.5 (osm) > EXPLAIN ...G "cost_info": { "query_cost": "661371.00" 1 row in set, 1 warning (0.00 sec) 19 More info on usage in the manual: http://dev.mysql.com/doc/refman/5.7/en/cost-model.html
  • 20. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Proposed Features • Technology-dependent costs (HD, SSD, memory) – Some of them could be auto-detected • Buffer “hotness”-aware statistics • Statistics for non-indexed columns/histograms 20 For more on cost optimizer changes and future, read Olav Sandstå’s slides: http://www.slideshare.net/olavsa/mysql-optimizer- cost-model
  • 21. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Engine-Independent Statistics/Histograms © 2014 DBAHIRE.COM in MariaDB 10 mariadb-10.0.14 (osm) > SET histogram_size = 255; Query OK, 0 rows affected (0.00 sec) mariadb-10.0.14 (osm) > SET use_stat_tables = 2; Query OK, 0 rows affected (0.01 sec) mariadb-10.0.14 (osm) > ANALYZE TABLE node_tags; +---------------+---------+----------+-----------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------+---------+----------+-----------------------------------------+ | osm.node_tags | analyze | status | Engine-independent statistics collected | | osm.node_tags | analyze | status | OK | +---------------+---------+----------+-----------------------------------------+ 2 rows in set (3.01 sec) mariadb-10.0.14 (osm) > ANALYZE TABLE nodes; +-----------+---------+----------+-----------------------------------------+ | Table | Op | Msg_type | Msg_text | +-----------+---------+----------+-----------------------------------------+ | osm.nodes | analyze | status | Engine-independent statistics collected | | osm.nodes | analyze | status | OK | +-----------+---------+----------+-----------------------------------------+ 2 rows in set (32.52 sec) mariadb-10.0.14 (osm) > SET optimizer_use_condition_selectivity = 3; (or 4 for histograms) Query OK, 0 rows affected (0.00 sec) 21
  • 22. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Stats! mariadb-10.0.14 (osm) > EXPLAIN EXTENDED SELECT * FROM nodes JOIN node_tags ON node_tags.node_id = nodes.node_id WHERE nodes.latitude BETWEEN 517000000 and 520000000G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: nodes type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 2865312 filtered: 0.39 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: node_tags type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.nodes.node_id rows: 3 filtered: 100.00 Extra: 2 rows in set, 1 warning (0.01 sec) 22 Much better estimate
  • 23. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Other Changes/Bugs Fixed • UNION ALL does not create temporary tables, returns tables faster • (a, b) IN ((1, 2), (2, 3)) can use index ranges • EXPLAIN EXTENDED is now the default behavior • I.C. Pushdown support for partitioned tables • IGNORE clause meaning has been standardized between sentence types • Increased default for optimizer_search_depth 23
  • 24. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks COMPUTED/VIRTUAL COLUMNS © 2014 DBAHIRE.COM 24
  • 25. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Syntax ALTER TABLE nodes ADD COLUMN lon DECIMAL (10, 7) as (longitude/10000000) VIRTUAL, ADD COLUMN lat DECIMAL (9, 7) as (latitude/10000000) VIRTUAL; • They can be used to simplify SELECTS, calculating values on the fly •Non accessed rows are not calculated 25 VIRTUAL is optional (default)
  • 26. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Virtual Columns Performance mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE lat > 51.5000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (1.10 sec) mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE latitude > 515000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (0.32 sec) mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE latitude/10000000 > 51.5000000; +----------+ | count(*) | +----------+ | 1679741 | +----------+ 1 row in set (0.79 sec) © 2014 DBAHIRE.COM 26
  • 27. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Explanation mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE lat > 51.5000000; +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | 1 | SIMPLE | nodes | NULL | ALL | NULL | NULL | NULL | NULL | 2602012 | 33.33 | Using where | +----+-------------+-------+------------+------+---------------+------+---------+------+---------+----------+-------------+ mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE latitude > 515000000; +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | nodes | NULL | range | latitude | latitude | 4 | NULL | 1301006 | 100.00 | Using where; Using index | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE latitude/10000000 > 51.5000000; +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | nodes | NULL | index | NULL | latitude | 8 | NULL | 2602012 | 100.00 | Using where; Using index | +----+-------------+-------+------------+-------+---------------+----------+---------+------+---------+----------+--------------------------+ mysql-5.7.5 (osm) > ALTER TABLE nodes add index(lon); ERROR 1951 (HY000): Key/Index cannot be defined on a non-stored virtual column. 27
  • 28. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Stored Columns mysql-5.7.5 (osm) > ALTER TABLE nodes CHANGE lat lat DECIMAL (9, 7) as (latitude/10000000) STORED; ERROR 1954 (HY000): 'Changing the STORED status' is not yet supported for virtual columns. mysql-5.7.5 (osm) > ALTER TABLE nodes DROP COLUMN lat, ADD COLUMN lat DECIMAL (9, 7) as (latitude/10000000) STORED; Query OK, 2865312 rows affected (4 min 51.05 sec) Records: 2865312 Duplicates: 0 Warnings: 0 28 MariaDB uses the 'PERMANENT' keyword
  • 29. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks A Query That Cannot Be Indexed mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE MONTH(timestamp) = 1; +----------+ | count(*) | +----------+ | 373557 | +----------+ 1 row in set (0.57 sec) © 2014 DBAHIRE.COM mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM nodes WHERE MONTH(timestamp) = 1G ************* 1. row ************* id: 1 select_type: SIMPLE table: nodes partitions: NULL type: index possible_keys: NULL key: nodes_timestamp_idx key_len: 5 ref: NULL rows: 2576970 filtered: 100.00 Extra: Using where; Using index 1 row in set, 1 warning (0.00 sec) 29
  • 30. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks We Add an Indexed Stored Computed © 2014 DBAHIRE.COM Column mysql-5.7.5 (osm) > ALTER TABLE nodes ADD COLUMN `month` tinyint UNSIGNED AS (MONTH(timestamp)) STORED, ADD INDEX(`month`); Query OK, 2865312 rows affected (7 min 9.12 sec) Records: 2865312 Duplicates: 0 Warnings: 0 30
  • 31. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Functional Index Implementation mysql-5.7.5 (osm) > SELECT count(*) FROM nodes WHERE month = 1; +----------+ | count(*) | +----------+ | 373557 | +----------+ 1 row in set (0.05 sec) © 2014 DBAHIRE.COM mysql-5.7.5 (osm) > EXPLAIN SELECT month, count(*) FROM nodes WHERE month = 1G ************* 1. row ************* id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ref possible_keys: month key: month key_len: 2 ref: const rows: 775360 filtered: 100.00 Extra: Using index 1 row in set, 1 warning (0.00 sec) 31
  • 32. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Why Not Just Use Triggers? CREATE TABLE `nodes_trigger` ( `node_id` bigint(64) NOT NULL, `latitude` int(11) NOT NULL, `longitude` int(11) NOT NULL, `changeset_id` bigint(20) NOT NULL, `visible` tinyint(1) NOT NULL, `timestamp` datetime NOT NULL, `tile` int(10) unsigned DEFAULT NULL, `version` bigint(20) NOT NULL, `month` tinyint(3) unsigned DEFAULT NULL, PRIMARY KEY (`node_id`,`version`), KEY `nodes_timestamp_idx` (`timestamp`), KEY `nodes_tile_idx` (`tile`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TRIGGER nodes_trigger_BI BEFORE INSERT ON nodes_trigger FOR EACH ROW SET NEW.month = MONTH(NEW.timestamp); CREATE TRIGGER nodes_trigger_BU BEFORE UPDATE ON nodes_trigger FOR EACH ROW SET NEW.month = MONTH(NEW.timestamp); CREATE TABLE `nodes_computed` ( `node_id` bigint(64) NOT NULL, `latitude` int(11) NOT NULL, `longitude` int(11) NOT NULL, `changeset_id` bigint(20) NOT NULL, `visible` tinyint(1) NOT NULL, `timestamp` datetime NOT NULL, `tile` int(10) unsigned DEFAULT NULL, `version` bigint(20) NOT NULL, `month` tinyint(3) unsigned AS (MONTH(timestamp)) STORED, PRIMARY KEY (`node_id`,`version`), KEY `nodes_timestamp_idx` (`timestamp`), KEY `nodes_tile_idx` (`tile`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; 32
  • 33. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Performance? $ ./sandboxes/msb_5_7_5/my sqldump osm nodes > nodes.sql $ time sed -e 's/nodes/nodes_trigger/' < nodes.sql | ./sandboxes/msb_5_7_5/use osm real 2m21.528s user 0m3.073s sys 0m0.463s $ time sed -e 's/nodes/nodes_computed/' < nodes.sql | ./sandboxes/msb_5_7_5/use osm real 2m24.458s user 0m3.079s sys 0m0.475s mysql-5.7.5 (osm) > INSERT INTO nodes_trigger(node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version) SELECT node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version FROM nodes; Query OK, 2865312 rows affected (1 min 6.84 sec) Records: 2865312 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > INSERT INTO nodes_computed(node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version) SELECT node_id, latitude, longitude, changeset_id, visible, timestamp, tile, version FROM nodes; Query OK, 2865312 rows affected (54.75 sec) Records: 2865312 Duplicates: 0 Warnings: 0 33 No significative improvements seen within a session
  • 34. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Consistency? mysql-5.7.5 (osm) > UPDATE nodes_trigger SET month = 13 WHERE rand() < 0.1; Query OK, 0 rows affected (1.71 sec) Rows matched: 286169 Changed: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT count(*) FROM nodes_trigger WHERE month=13; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.45 sec) mysql-5.7.5 (osm) > UPDATE nodes_computed SET month = 13 WHERE rand() < 0.1; Query OK, 0 rows affected, 65535 warnings (1.31 sec) Rows matched: 286177 Changed: 0 Warnings: 286177 mysql-5.7.5 (osm) > SHOW WARNINGS; +---------+------+-----------------------------------------------------------------------------------+ | Level | Code | Message | +---------+------+-----------------------------------------------------------------------------------+ | Warning | 1953 | The value specified for virtual column 'month' in table 'nodes_computed' ignored. | mysql-5.7.5 (osm) > SELECT count(*) FROM nodes_computed WHERE month=13; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.57 sec) 34 Maybe an error should be produced in strict mode, as MariaDB does?
  • 35. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Features and Limitations • Virtual “non-stored” columns cannot be indexed • “Stored columns” can be PRIMARY, UNIQUE, FOREIGN and MULTI keys • It cannot contain subqueries or other tables or rows references • It cannot contain used-defined stored functions • It can contain server and user-defined variables • It can be computed based on other virtual columns 35
  • 36. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks QUERY REWRITE PLUGINS © 2014 DBAHIRE.COM 36
  • 37. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks New APIs for Query Rewriting • One for pre-parsing rewriting • Another for post-parsing rewriting © 2014 DBAHIRE.COM 37
  • 38. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Post- Plugin Installation $ ./sandboxes/msb_5_7_5/use -u root < 5.7.5/lib/plugin/ install_rewriter_plugin.sql – It is mostly broken in the lab build (for my tests) – Uninstall it before restart, or MySQL will segfault © 2014 DBAHIRE.COM 38 More on this: http://mysqlserverteam.com/the-query-rewrite-plugins/
  • 39. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Query Rewriting Setup mysql-5.7.5 ((none)) > INSERT INTO query_rewrite.rewrite_rules (pattern, replacement) VALUES ('SELECT ?', 'SELECT ? + 1'); Query OK, 1 row affected (0.01 sec) mysql-5.7.5 ((none)) > CALL query_rewrite.flush_rewrite_rules(); Query OK, 1 row affected (0.00 sec) mysql-5.7.5 ((none)) > SELECT 1; +-------+ | 1 + 1 | +-------+ | 2 | +-------+ 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 ((none)) > SHOW WARNINGS; +-------+------+-------------------------------------------------------------------+ | Level | Code | Message | +-------+------+-------------------------------------------------------------------+ | Note | 1105 | Query 'SELECT 1' rewritten to 'SELECT 1 + 1' by plugin: Rewriter. | +-------+------+-------------------------------------------------------------------+ 1 row in set (0.00 sec) 39
  • 40. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Considerations • It cannot correct malformed queries- pattern and replacement must be syntactically correct • Useful for query optimizations for 3rd party applications • User is notified that a rewrite happened with a note-level message • Low overhead (5%); specially for untouched queries • You can do stupid things like: mysql-5.7.5 ((none)) > INSERT INTO query_rewrite.rewrite_rules( pattern, replacement ) VALUES ( 'SELECT 1', 'DROP TABLE test.test' ); Query OK, 1 row affected (0.01 sec) 40
  • 41. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks SQL MODE CHANGES © 2014 DBAHIRE.COM 41
  • 42. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Default SQL Mode Changes • MySQL 5.5 and earlier – '' • MySQL 5.6 (from 5.6.6): – 'NO_ENGINE_SUBSTITUTION' is the default – NO_ENGINE_SUBSTITUTION and STRICT_TRANS_TABLES were suggested in upstream default my.cnf • MySQL 5.7 (from 5.7.5): – 'ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_EN GINE_SUBSTITUTION' is the new default 42 More on this: http://www.tocker.ca/2014/01/14/making-strict-sql_mode-the-default.html
  • 43. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Stricter Defaults mysql> CREATE TABLE `test` ( `id` int(11) NOT NULL, `type` enum('movie','album','videogame') NOT NULL, `date` datetime NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB; mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); 43 What happens?
  • 44. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks MySQL 5.5/5.6 with default settings mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); Query OK, 1 row affected, 3 warnings (0.00 sec) mysql> SHOW WARNINGS; +---------+------+-----------------------------------------------+ | Level | Code | Message | +---------+------+-----------------------------------------------+ | Warning | 1364 | Field 'id' doesn't have a default value | | Warning | 1265 | Data truncated for column 'type' at row 1 | | Warning | 1264 | Out of range value for column 'date' at row 1 | +---------+------+-----------------------------------------------+ 3 rows in set (0.00 sec) mysql> SELECT * FROM test; +----+------+---------------------+ | id | type | date | +----+------+---------------------+ | 0 | | 0000-00-00 00:00:00 | +----+------+---------------------+ 1 row in set (0.00 sec) © 2014 DBAHIRE.COM 44
  • 45. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL 5.7 mysql> INSERT INTO test (type, date) VALUES ('tv show', -1); ERROR 1265 (01000): Data truncated for column 'type' at row 1 mysql> INSERT INTO test (type, date) VALUES ('videogame', -1); ERROR 1292 (22007): Incorrect datetime value: '-1' for column 'date' at row 1 mysql> INSERT INTO test (type, date) VALUES ('movie', now()); ERROR 1364 (HY000): Field 'id' doesn't have a default value mysql> INSERT INTO test (id, type, date) VALUES (1, 'videogame', now()); Query OK, 1 row affected (0.01 sec) 45
  • 46. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GROUP BY Behavior Changes mysql-5.6.21 (osm) > SELECT way_id, count(*), node_id © 2014 DBAHIRE.COM FROM way_nodes GROUP BY way_id ORDER BY count(*) DESC LIMIT 10; +-----------+----------+------------+ | way_id | count(*) | node_id | +-----------+----------+------------+ | 155339744 | 1187 | 1558095871 | | 243986064 | 1156 | 2604713337 | | 87136668 | 1128 | 1013304944 | | 148812873 | 852 | 1618837453 | | 149200774 | 835 | 34921158 | | 183618216 | 826 | 1940223096 | | 273858696 | 824 | 1267549776 | | 261584374 | 770 | 2669122104 | | 227880171 | 704 | 2240011804 | | 193564006 | 684 | 1808872763 | +-----------+----------+------------+ 10 rows in set (1.24 sec) 46 Undeterministic behavior
  • 47. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks With ONLY_FULL_GROUP_BY (default in 5.7) mysql-5.7.5 (osm) > SELECT way_id, count(*), node_id © 2014 DBAHIRE.COM FROM way_nodes GROUP BY way_id ORDER BY count(*) DESC LIMIT 10; ERROR 1055 (42000): Expression #3 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'osm.way_nodes.node_id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by 47
  • 48. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Problem with ONLY_FULL_GROUP_BY in Functional dependency © 2014 DBAHIRE.COM MySQL <= 5.6 mysql-5.6.21 (osm) > SET SQL_mode='ONLY_FULL_GROUP_BY'; Query OK, 0 rows affected (0.00 sec) mysql-5.6.21 (osm) > SELECT u.id as `user id`, u.display_name as `user name`, count(*) as `# changesets` FROM users u JOIN changesets c ON u.id = c.user_id GROUP BY u.id ORDER BY count(*) DESC LIMIT 10; ERROR 1055 (42000): 'osm.u.display_name' isn't in GROUP BY 48 More on this: http://rpbouman.blogspot.com/2014/09/mysql-575-group-by-respects-functional.html
  • 49. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks 5.7 Aims for SQL99 Compliance mysql-5.7.5 (osm) > SELECT u.id as `user id`, u.display_name as `user name`, count(*) as `# changesets` FROM users u JOIN changesets c ON u.id = c.user_id GROUP BY u.id ORDER BY count(*) DESC LIMIT 10; +---------+---------------+--------------+ | user id | user name | # changesets | +---------+---------------+--------------+ | 31257 | Ed Avis | 4333 | | 508 | Welshie | 2696 | | 1016290 | Amaroussi | 2351 | | 352985 | ecatmur | 1450 | | 142807 | SDavies | 1342 | | 736 | Steve Chilton | 1182 | | 346 | Tom Chance | 1175 | | 38784 | Tom Morris | 1165 | | 88718 | UrbanRambler | 1151 | | 1611 | Harry Wood | 1091 | +---------+---------------+--------------+ 10 rows in set (0.09 sec) © 2014 DBAHIRE.COM 49
  • 50. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Backward Compatibility • Some ORMs and frameworks change the default SQL Mode: – Ruby on Rails 4+ sets STRICT_ALL_TABLES – Drupal 7+ sets TRADITIONAL • Other applications do not work in standard-compliance modes: – Wordpress does not work in strict mode: http:// www.xaprb.com/blog/2013/03/15/wordpress-and-mysqls-strict-mode/ – Cacti does not work with ONLY_FULL_GROUP_BY,NO_ZERO_DATE 50
  • 51. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Deprecated Modes • ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, and NO_ZERO_IN_DATE are deprecated and do nothing – Use STRICT_TRANS_TABLES or STRICT_ALL_TABLES, which include those modes and produce an error 51
  • 52. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks JOIN OPTIMIZATIONS © 2014 DBAHIRE.COM 52
  • 53. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM (Block) Nested Loop Join • Until MySQL 5.5, there was only one algorithm to execute a JOIN: 53 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 name London Eye
  • 54. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM “New” JOIN Methods • MySQL 5.6 added: – Batch Key Access* • MariaDB has since 5.3: – Batch Key Access* – Hash Joins – Slightly modified versions of the above (“incremental” buffers for joining 3 or more tables) 54 *I already mentioned BKA last year
  • 55. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Batch Key Access 55 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 amenity cafe Buffer sorted in physical/ PK order
  • 56. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Hash Joins • Only work for equi-joins 56 node_id version lat lon 1 1 52 0.5 1 2 52 0.5 2 1 51 1 3 1 53 1.5 node_id version k v 1 1 name Big Benn 1 1 tourism attraction 1 2 name Big Ben 1 2 tourism attraction 3 1 amenity cafe Hash table for faster access to rows
  • 57. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL Configuration • BKA requires changes of default optimizer configuration: mysql-5.7.5 (osm) > SET optimizer_switch= 'mrr=on'; mysql-5.7.5 (osm) > SET optimizer_switch= 'mrr_cost_based=off'; mysql-5.7.5 (osm) > SET optimizer_switch= 'batch_key_access=on'; 57
  • 58. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MariaDB configuration mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_incremental=on'; mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_hashed=on'; mariadb-10.0.14 (osm) > SET optimizer_switch = 'join_cache_bka=on'; - Enabled by default mariadb-10.0.14 (osm) > SET join_cache_level = 3 (for hash joins) mariadb-10.0.14 (osm) > SET join_cache_level = 5 (for BKA) - Also, configure join_buffer_size appropriately. - Hash joins, like BKA, are highly dependent on disk-bound DBs to be effective due to the extra overhead 58
  • 59. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Nested Loop Join (cold buffers, buffer_pool=100MB) mariadb-10.0.14 (osm) > EXPLAIN SELECT changeset_id, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visibleG ******************** 1. row ******************** id: 1 select_type: SIMPLE table: changesets type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 69115 Extra: Using index; Using temporary; Using filesort ******************** 2. row ******************** id: 1 select_type: SIMPLE table: nodes type: ref possible_keys: changeset_id key: changeset_id key_len: 8 ref: osm.changesets.id rows: 19 Extra: 2 rows in set (0.00 sec) mariadb-10.0.14 (osm) > SELECT visible, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visible; +---------+----------+ | visible | count(*) | +---------+----------+ | 1 | 2865312 | +---------+----------+ 1 row in set (32.86 sec) 59
  • 60. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Hash Join (cold buffers, buffer_pool=100M, © 2014 DBAHIRE.COM join_buffer=4M) mariadb-10.0.14 (osm) > EXPLAIN SELECT changeset_id, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visibleG ******************** 1. row ******************** id: 1 select_type: SIMPLE table: changesets type: index possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: NULL rows: 69115 Extra: Using index; Using temporary; Using filesort ******************** 2. row ******************** id: 1 select_type: SIMPLE table: nodes type: hash_ALL possible_keys: changeset_id key: #hash#changeset_id key_len: 8 ref: osm.changesets.id rows: 2781732 Extra: Using join buffer (flat, BNLH join) 2 rows in set (0.00 sec) mariadb-10.0.14 (osm) > SELECT visible, count(*) FROM changesets JOIN nodes on changesets.id = nodes.changeset_id GROUP BY visible; +---------+----------+ | visible | count(*) | +---------+----------+ | 1 | 2865312 | +---------+----------+ 1 row in set (6.66 sec) 60
  • 61. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks SUBQUERY OPTIMIZATIONS © 2014 DBAHIRE.COM 61
  • 62. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Delayed Materialization of Derived Tables • EXPLAIN performance improved (it does not execute subqueries) • Derived tables can be indexed for improved performance © 2014 DBAHIRE.COM 62
  • 63. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example with a Derived Table mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON n.changeset_id = changesets.id; +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2865312 | | | 1 | PRIMARY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | n.changeset_id | 1 | Using index | | 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where | +----+-------------+------------+--------+---------------+---------+---------+----------------+---------+-------------+ 3 rows in set (1.42 sec) mysql-5.6.21 (osm) > EXPLAIN SELECT count(*) FROM (SELECT * FROM nodes WHERE VISIBLE = 1) n JOIN changesets ON n.changeset_id = changesets.id; +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ | 1 | PRIMARY | changesets | index | PRIMARY | PRIMARY | 8 | NULL | 70917 | Using index | | 1 | PRIMARY | <derived2> | ref | <auto_key0> | <auto_key0> | 8 | osm.changesets.id | 40 | NULL | | 2 | DERIVED | nodes | ALL | NULL | NULL | NULL | NULL | 2853846 | Using where | +----+-------------+------------+-------+---------------+-------------+---------+-------------------+---------+-------------+ 3 rows in set (0.00 sec) © 2014 DBAHIRE.COM 63 No subquery execution Auto-generated index
  • 64. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks A Common 5.5 Performance Problem mysql-5.5.40 (osm) > EXPLAIN SELECT * FROM nodes © 2014 DBAHIRE.COM WHERE nodes.changeset_id IN ( SELECT changesets.id FROM changesets JOIN users ON changesets.user_id = users.id and users.display_name = 'Steve'); +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ | 1 | PRIMARY | nodes | ALL | NULL | NULL | NULL | NULL | 2865521 | Using where | | 2 | DEPENDENT SUBQUERY | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index | | 2 | DEPENDENT SUBQUERY | changesets | eq_ref | PRIMARY | PRIMARY | 8 | func | 1 | Using where | +----+--------------------+------------+--------+--------------------------------+------------------------+---------+-------+---------+-------------+ 3 rows in set (0.00 sec) mysql-5.5.40 (osm) > SELECT ...; +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 | | 109174 | 515364532 | -1457329 | 1875 | 1 | 2006-01-20 00:01:27 | 2062268519 | 1 | | 276538 | 515324296 | -2094688 | 810 | 1 | 2005-11-25 21:42:53 | 2062267078 | 1 | | 442987 | 515449207 | -1275650 | 1941 | 1 | 2006-01-22 23:50:28 | 2062268628 | 1 | | 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 | | 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 | | 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ 7 rows in set (2.60 sec) 64 This means that the subquery is executed almost 3 million times
  • 65. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Semijoin Optimization • The only way to execute certain IN subqueries was to execute them with poor strategy – This forced to rewrite certain queries into JOINS or scalar subqueries, when possible • There are now several additional strategies/ optimizations: – Convert to a JOIN – Materialization (including index creation) – FirstMatch – LooseScan – Duplicate Weedout 65
  • 66. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks The Previous Query is Not a Problem in © 2014 DBAHIRE.COM 5.6/5.7/MariaDB 5.3+ mysql-5.6.21 (osm) > EXPLAIN SELECT * FROM nodes WHERE nodes.changeset_id IN ( SELECT changesets.id FROM change sets JOIN users ON changesets.user_id = users.id and users.display_name = 'Steve'); +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ | 1 | SIMPLE | users | const | PRIMARY,users_display_name_idx | users_display_name_idx | 767 | const | 1 | Using index | | 1 | SIMPLE | changesets | ALL | PRIMARY | NULL | NULL | NULL | 70917 | Using where | | 1 | SIMPLE | nodes | ref | changeset_id | changeset_id | 8 | osm.changesets.id | 21 | NULL | +----+-------------+------------+-------+--------------------------------+------------------------+---------+-------------------+-------+-------------+ 3 rows in set (0.00 sec) mysql-5.6.21 (osm) > SELECT ...; +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ | 99890 | 515276425 | -1497621 | 552 | 1 | 2005-10-25 00:35:24 | 2062268512 | 1 | | 276538 | 515324296 | -2094688 | 810 | 1 | 2005-11-25 21:42:53 | 2062267078 | 1 | | 109174 | 515364532 | -1457329 | 1875 | 1 | 2006-01-20 00:01:27 | 2062268519 | 1 | | 442987 | 515449207 | -1275650 | 1941 | 1 | 2006-01-22 23:50:28 | 2062268628 | 1 | | 442988 | 515449741 | -1272860 | 1941 | 1 | 2006-01-22 23:50:29 | 2062268628 | 1 | | 498803 | 515438432 | -1269436 | 2171 | 1 | 2006-02-03 21:55:17 | 2062268628 | 1 | | 138212838 | 513010180 | -1699929 | 7757299 | 1 | 2011-04-03 18:14:14 | 2062220563 | 6 | +-----------+-----------+-----------+--------------+---------+---------------------+------------+---------+ 7 rows in set (0.02 sec) 66 Executed as a regular JOIN
  • 67. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM First Match Strategy mysql-5.7.5 (osm) > EXPLAIN SELECT * FROM changesets WHERE id IN (SELECT changeset_id FROM nodes)G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: changesets partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 70917 filtered: 100.00 Extra: NULL *************************** 2. row *************************** id: 1 select_type: SIMPLE table: nodes partitions: NULL type: ref possible_keys: changeset_id key: changeset_id key_len: 8 ref: osm.changesets.id rows: 33 filtered: 100.00 Extra: Using index; FirstMatch(changesets) 67 It is converting the ref into an eq_ref
  • 68. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Enabling and disabling materialization/ © 2014 DBAHIRE.COM semijoin mysql-5.7.5 (osm) > SHOW VARIABLES like 'optimizer_switch'G ******************* 1. row ******************* Variable_name: optimizer_switch Value: index_merge=on,index_merge_union=on,index_merge_sort_union=on, index_merge_intersection=on,engine_condition_pushdown=on,index _condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_l oop=on,batched_key_access=off,materialization=on,semijoin=on,l oosescan=on,firstmatch=on,subquery_materialization_cost_based= on,use_index_extensions=on,condition_fanout_filter=on 1 row in set (0.00 sec) 68
  • 69. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks FULLTEXT SEARCH © 2014 DBAHIRE.COM 69
  • 70. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Fuzzy Search of “gloucester/Gloucester’s/ © 2014 DBAHIRE.COM etc” • “Typical” way to solve this: mysql-5.7.5 (osm) > SELECT way_id as id, v FROM way_tags WHERE v like '%gloucester%'; 425 rows in set (0.46 sec) 70 Too slow
  • 71. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Let’s Add an Index mysql-5.7.5 (osm) > ALTER TABLE way_tags ADD INDEX(v); Query OK, 0 rows affected (6.44 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT ...; 425 rows in set (0.38 sec) • Still slow, why? mysql-5.7.5 (osm) > EXPLAIN SELECT 'way' as type, way_id as id, v FROM way_tags WHERE v like '%gloucester%'; +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ | 1 | SIMPLE | way_tags | NULL | index | NULL | v_2 | 767 | NULL | 1333338 | 11.11 | Using where; Using index | +----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+ 1 row in set, 1 warning (0.01 sec) 71
  • 72. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Fulltext Index mysql-5.7.5 (osm) > ALTER TABLE way_tags add FULLTEXT index(v); Query OK, 0 rows affected (3.20 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > SELECT ...; 425 rows in set (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT 'way' as type, way_id as id, v FROM way_tags WHERE MATCH(v) AGAINST ('+gloucester*' IN BOOLEAN MODE); +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ | 1 | SIMPLE | way_tags | NULL | fulltext | v | v | 0 | const | 1 | 100.00 | Using where; Ft_hints: no_ranking | +----+-------------+----------+------------+----------+---------------+------+---------+-------+------+----------+-----------------------------------+ 1 row in set, 1 warning (0.00 sec) 72
  • 73. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Newer Fulltext Optimizations mysql-5.5.40 (osm) > EXPLAIN SELECT count(*) FROM way_tags_myisam WHERE MATCH(v) AGAINST('gloucester'); +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | way_tags_myisam | fulltext | v | v | 0 | | 1 | Using where | +----+-------------+-----------------+----------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec) mysql-5.5.40 (osm) > SHOW STATUS like 'Hand%'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 0 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 0 | | Handler_read_last | 0 | | Handler_read_next | 425 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 16 rows in set (0.00 sec) © 2014 DBAHIRE.COM 73
  • 74. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Newer Fulltext Optimizations (cont.) mysql-5.7.5 (osm) > EXPLAIN SELECT count(*) FROM way_tags WHERE MATCH(v) AGAINST('gloucester'); +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ | 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away | +----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+ 1 row in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > SHOW STATUS like 'Hand%'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 2 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 0 | | Handler_read_last | 0 | | Handler_read_next | 0 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 18 rows in set (0.00 sec) © 2014 DBAHIRE.COM 74 It’s counting directly from the FULLTEXT index
  • 75. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Open Issues and Limitations • No postfix support (wildcards) • Simple Ranking (and different from MyISAM) • No stemming support • Some multi-language limitations © 2014 DBAHIRE.COM 75 More on FULLTEXT InnoDB support: http://www.drdobbs.com/database/full-text- search-with-innodb/231902587
  • 76. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks GIS IMPROVEMENTS © 2014 DBAHIRE.COM 76
  • 77. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Find the Closest Starbucks mysql-5.7.5 (osm) > SET @lat:=51.49353; SET @lon:=-0.18340; mysql-5.7.5 (osm) > SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/10000000 as latitude, sqrt(pow((latitude/10000000 - @lat) * 111257.67, 2) + pow((longitude/10000000 - @lon) * 69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' ORDER BY `distance in metres` ASC LIMIT 1; +-----------+-----------+----------+--------------------+ | node_id | longitude | latitude | distance in metres | +-----------+-----------+----------+--------------------+ | 699693936 | -0.1823 | 51.4945 | 130.9792513096838 | +-----------+-----------+----------+--------------------+ 1 row in set (0.20 sec) 77 You are here
  • 78. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM This Query is Slow mysql-5.7.5 (osm) > EXPLAIN SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/ 10000000 as latitude, sqrt(pow((latitude/10000000- @lat)*111257.67, 2) + pow((longitude/10000000- @lon)*69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' ORDER BY `distance in metres` ASC LIMIT 1G ********************* 1. row ********************* id: 1 select_type: SIMPLE table: n_t1 partitions: NULL type: ALL possible_keys: PRIMARY key: NULL key_len: NULL ref: NULL rows: 832040 filtered: 0.00 Extra: Using where; Using temporary; Using file sort ********************* 2. row ********************* id: 1 select_type: SIMPLE table: n_t2 partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.n_t1.node_id rows: 3 filtered: 1.41 Extra: Using where ********************* 3. row ********************* id: 1 select_type: SIMPLE table: n partitions: NULL type: ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: osm.n_t1.node_id rows: 1 filtered: 100.00 Extra: NULL 3 rows in set, 1 warning (0.00 sec) 78
  • 79. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Can We Optimize it? • We could add a bounding box: mysql-5.7.5 (osm) > ALTER TABLE nodes add index(latitude, longitude); mysql-5.7.5 (osm) > SELECT n.node_id, n.longitude/10000000 as longitude, n.latitude/10000000 as latitude, sqrt(pow((latitude/10000000 - @lat) * 111257.67, 2) + pow((longitude/10000000 - @lon) * 69450.32, 2)) as `distance in metres` FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' and n.latitude BETWEEN ((@lat - 1000/111257.67) * 10000000) AND ((@lat + 1000/111257.67) * 10000000) and n.longitude BETWEEN ((@lon - 1000/69450.32) * 10000000) AND ((@lon + 1000/69450.32) * 10000000) ORDER BY `distance in metres` ASC LIMIT 1; 79 This is not a square, only an approximation
  • 80. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks We Create an Index… and Force It mysql-5.7.5 (osm) > EXPLAIN SELECT ...; +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ | id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra | | | | | | | _keys | | len | | | | | +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ | 1 | SIMPLE | n_t1 | NULL | ALL | PRIMARY | NULL | NULL | NULL | 832040 | 0.00 | Using where; | | | | | | | | | | | | | Using temporary; | | | | | | | | | | | | | Using filesort | | 1 | SIMPLE | n | NULL | ref | PRIMARY, | PRIMARY | 8 | osm.n_t1.node_id | 1 | 5.00 | Using where | | | | | | | latitude | | | | | | | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n_t1.node_id | 3 | 1.41 | Using where | +----+-------------+-------+------------+------+----------+---------+------+------------------+--------+----------+------------------+ 3 rows in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT ... FROM nodes n FORCE INDEX(latitude) ...; +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ | id | select_type | table | partitions | type | possible | key | key_ | ref | rows | filtered | Extra | | | | | | | _keys | | len | | | | | +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ | 1 | SIMPLE | n | NULL | range | latitude | latitude | 8 | NULL | 493666 | 11.11 | Using where; | | | | | | | | | | | | | Using index; | | | | | | | | | | | | | Using filesort | | 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 3 | 1.41 | Using where | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n.node_id | 3 | 1.41 | Using where | +----+-------------+-------+------------+-------+----------+----------+------+---------------+--------+----------+------------------+ 3 rows in set, 1 warning (0.00 sec) mysql-5.7.5 (osm) > SELECT ... FROM nodes n FORCE INDEX(latitude) ...; +-----------+-----------+----------+--------------------+ | node_id | longitude | latitude | distance in metres | +-----------+-----------+----------+--------------------+ | 699693936 | -0.1823 | 51.4945 | 130.9792513096838 | +-----------+-----------+----------+--------------------+ 1 row in set (0.09 sec) © 2014 DBAHIRE.COM 80 Still many rows are examined Most of the gain comes from the covering index, not the filtering MySQL ignores the newly created index, why? Performance improvement is not great
  • 81. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Multiple Range Scans Cannot Be Optimized with BTREE Indexes • We need quadtrees or R-TREE Indexes for indexing in multiple dimensions – The later are implemented in MySQL with the name “SPATIAL indexes”, as they only apply to GIS types • Spatial indexing is available for the first time for InnoDB tables on MySQL 5.7.5 81
  • 82. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Creating a Spatial Index mysql-5.7.5 (osm) > ALTER TABLE nodes ADD COLUMN coord GEOMETRY NOT NULL; Query OK, 0 rows affected (21.80 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql-5.7.5 (osm) > UPDATE nodes SET coord = point(longitude/1000000, latitude/10000000); Query OK, 2865312 rows affected (34.66 sec) Rows matched: 2865312 Changed: 2865312 Warnings: 0 mysql-5.7.5 (osm) > ALTER TABLE nodes add SPATIAL index(coord); Query OK, 0 rows affected (1 min 50.00 sec) Records: 0 Duplicates: 0 Warnings: 0 82 This is new in 5.7
  • 83. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM New Query mysql> SET @area := envelope(linestring(POINT(@lon - 500/69450.32, @lat - 500/111257.67), POINT(@lon + 500/69450.32, @lat + 500/111257.67))); mysql> SELECT n.node_id, x(n.coord) as longitude, y(n.coord) as latitude, st_distance(POINT(@lon, @lat), coord) as distance FROM nodes n JOIN node_tags n_t1 ON n.node_id = n_t1.node_id JOIN node_tags n_t2 ON n.node_id = n_t2.node_id WHERE n_t1.k = 'amenity' and n_t1.v = 'cafe' and n_t2.k = 'name' and n_t2.v = 'Starbucks' and st_within(coord, @area) ORDER BY st_distance(POINT(@lon, @lat), coord) ASC LIMIT 1; 83 We can use any shape we want thanks to 5.6 improvements
  • 84. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Performance mysql-5.7.5 (osm) > SELECT ...; +-----------+------------+------------+-----------------------+ | node_id | longitude | latitude | distance | +-----------+------------+------------+-----------------------+ | 699693936 | -0.1822879 | 51.4944808 | 0.0014631428672541478 | +-----------+------------+------------+-----------------------+ 1 row in set (0.02 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT ...; +----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+ | id | select | table | parti | type | possible | key | key | ref | rows | filtered | Extra | | | _type | | tions | | _keys | | _len | | | | | +----+--------+-------+-------+-------+----------+---------+------+---------+------+----------+----------------+ | 1 | SIMPLE | n | NULL | range | PRIMARY | coord | 34 | NULL | 2 | 100.00 | Using where; | | | | | | | ,coord | | | | | | Using filesort | | 1 | SIMPLE | n_t1 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where | | | | | | | | | | node_id | | | | | 1 | SIMPLE | n_t2 | NULL | ref | PRIMARY | PRIMARY | 8 | osm.n. | 3 | 1.41 | Using where | | | | | | | | | | node_id | | | | +----+--------+-------+-------+-------+----------+---------+------+---------+------+---------------------------+ 3 rows in set, 1 warning (0.00 sec) 84 This field is almost useless
  • 85. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Better Filtering Not using the index: +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 1 | | Handler_read_key | 1914 | | Handler_read_last | 0 | | Handler_read_next | 1954 | | Handler_read_prev | 0 | | Handler_read_rnd | 1 | | Handler_read_rnd_next | 833426 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 1 | +----------------------------+--------+ 18 rows in set (0.00 sec) Using the BTREE index: +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 274 | | Handler_read_last | 0 | | Handler_read_next | 246540 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+--------+ 18 rows in set (0.00 sec) Using the SPATIAL index: +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | | Handler_external_lock | 6 | | Handler_mrr_init | 0 | | Handler_prepare | 0 | | Handler_read_first | 0 | | Handler_read_key | 522 | | Handler_read_last | 0 | | Handler_read_next | 5254 | | Handler_read_prev | 0 | | Handler_read_rnd | 259 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-------+ 18 rows in set (0.00 sec) 85 mysql-5.7.5 (osm) > SHOW STATUS LIKE 'Hand%';
  • 86. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Geohash Functions • mysql-5.7.5 (osm) > SELECT ST_GeoHash(@lon, @lat, 10); +----------------------------+ | ST_GeoHash(@lon, @lat, 10) | +----------------------------+ | gcpugy47w3 | +----------------------------+ 1 row in set (0.00 sec) • Useful to index coordinates with a BTREE – It could be specially useful combined with indexed STORED columns (emulating quadtrees) 86 More on Geohashing: http://mysqlserverteam.com/geohash-functions/
  • 87. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM GeoJSON Functions mysql-5.7.5 (osm) > SELECT nm.v, ST_AsGeoJson(n.coord) FROM node_tags n_t JOIN nodes n USING (node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' AND n_t.v='attraction' AND nm.k='name'; +-----------------------------+--------------------------------------------------------+ | v | ST_AsGeoJson(n.coord) | +-----------------------------+--------------------------------------------------------+ | BedZED | {"type":"Point","coordinates":[-0.1560632,51.3821745]} | | Blewcoat School | {"type":"Point","coordinates":[-0.1360484,51.4983179]} | | Camden / Buck Street Market | {"type":"Point","coordinates":[-0.143193,51.5400398]} | | Camden Lock Village | {"type":"Point","coordinates":[-0.1447181,51.5416552]} | | Wimbledon Windmill | {"type":"Point","coordinates":[-0.2315468,51.4376583]} | . . . 87
  • 88. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM GeoJSON Functions (cont.) $ ./sandboxes/msb_5_7_5/use osm -B -e "SET SESSION group_concat_max_len = 10000; SELECT CONCAT('{"type":"FeatureCollection", "features":[ ', GROUP_CONCAT(CONCAT('{"type":"Feature", "geometry":', ST_AsGeoJson(n.coord), ',"properties":{"name":"',nm.v,'"}}')), ' ]}') FROM node_tags n_t JOIN nodes n USING (node_id, version) JOIN node_tags nm USING (node_id, version) WHERE n_t.k='tourism' and n_t.v='attraction' AND nm.k='name'" http://geojsonlint.com/ 88
  • 89. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Open Issues • The SRID can be set and retrieved, but all operations are done in squared euclidean coordinates: 89 mysql-5.7.5 (osm) > SET @p1 := GeomFromText('POINT(-1 51)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SET @p2 := GeomFromText('POINT(0 51)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SET @p3 := GeomFromText('POINT(-1 52)', 4326); Query OK, 0 rows affected (0.00 sec) mysql-5.7.5 (osm) > SELECT srid(@p1); +-----------+ | srid(@p1) | +-----------+ | 4326 | +-----------+ 1 row in set (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p2); +-----------------------+ | st_distance(@p1, @p2) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.00 sec) mysql-5.7.5 (osm) > SELECT st_distance(@p1, @p3); +-----------------------+ | st_distance(@p1, @p3) | +-----------------------+ | 1 | +-----------------------+ 1 row in set (0.00 sec)
  • 90. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks QUERY PROFILING © 2014 DBAHIRE.COM 90
  • 91. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Which Queries Should I Optimize First? • My two favorite methods: – pt-query-digest – PERFORMANCE_SCHEMA • I prefer pt-query-digest for long-term reports, PERFORMANCE_SCHEMA for more real-time evaluation and fine-tuning – Also, PERFORMANCE_SCHEMA was not “ready” until © 2014 DBAHIRE.COM MySQL 5.6 91
  • 92. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM pt-query-digest • It is a 3rd party tool written in Perl, originally created by Baron Schwartz • It requires activation of the slow log: – SET GLOBAL slow_query_log = 1; – SET long_query_time = 0; • In Percona Server and MariaDB it provides extra information: – SHOW GLOBAL VARIABLES like 'log_slow_verbosity'; 92 Be careful with extra IO and latency!
  • 93. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM pt-query-digest Execution $ pt-query-digest ~/sandboxes/msb_5_7_5/data/localhost-slow.log # 190ms user time, 30ms system time, 25.45M rss, 219.82M vsz # Current date: Sat Nov 1 17:26:58 2014 # Hostname: localhost.localdomain # Files: /home/dbahire/sandboxes/msb_5_7_5/data/localhost-slow.log # Overall: 21 total, 7 unique, 0 QPS, 0x concurrency _____________________ # Attribute total min max avg 95% stddev median # ============ ======= ======= ======= ======= ======= ======= ======= # Exec time 7s 6us 482ms 324ms 455ms 201ms 433ms # Lock time 18ms 0 11ms 840us 224us 2ms 113us # Rows sent 7.08k 0 1.22k 345.43 420.77 300.48 420.77 # Rows examine 19.22M 0 1.28M 937.14k 1.26M 581.94k 1.26M # Query size 1.10k 11 67 53.81 65.89 18.23 65.89 # Profile # Rank Query ID Response time Calls R/Call V/M Item # ==== ================== ============= ===== ====== ===== =============== # 1 0x5060CBA490BE371E 6.4236 94.4% 14 0.4588 0.00 SELECT way_tags 93
  • 94. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks pt-query-digest Execution (cont.) # Query 1: 0 QPS, 0x concurrency, ID 0x5060CBA490BE371E at byte 8795 _____ # This item is included in the report because it matches --limit. # Scores: V/M = 0.00 # Attribute pct total min max avg 95% stddev median # ============ === ======= ======= ======= ======= ======= ======= ======= # Count 66 14 # Exec time 94 6s 435ms 482ms 459ms 477ms 15ms 455ms # Lock time 10 2ms 0 227us 137us 204us 54us 113us # Rows sent 87 6.21k 0 1.22k 453.86 420.77 242.38 420.77 # Rows examine 93 17.94M 1.28M 1.28M 1.28M 1.28M 0 1.28M # Query size 80 912 59 67 65.14 65.89 2.70 65.89 # String: # Databases osm # Hosts localhost # Time 2014-11-01... (1/7%), 2014-11-01... (1/7%)... 12 more # Users msandbox # Query_time distribution # 1us # 10us # 100us # 1ms # 10ms # 100ms ################################################################ # 1s # 10s+ # Tables # SHOW TABLE STATUS FROM `osm` LIKE 'way_tags'G # SHOW CREATE TABLE `osm`.`way_tags`G # EXPLAIN /*!50100 PARTITIONS*/ SELECT way_id as id, v FROM way_tags WHERE v like '%gloucester%'G © 2014 DBAHIRE.COM 94
  • 95. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM PERFORMANCE_SCHEMA • Monitoring schema enabled by default since MySQL 5.6 – performance_schema = 1 (it is not dynamic) • Deprecates the old query profiling • It is way more user-friendly when combined with the SYS schema/ps_helper (a set of views and stored procedures created by Mark Leith) 95
  • 96. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Installation of the SYS Schema $ git clone https://github.com/MarkLeith/mysql-sys.git Cloning into 'mysql-sys'... remote: Counting objects: 926, done. remote: Compressing objects: 100% (73/73), done. remote: Total 926 (delta 35), reused 6 (delta 2) Receiving objects: 100% (926/926), 452.19 KiB | 225.00 KiB/s, done. Resolving deltas: 100% (584/584), done. $ cd mysql-sys/ $ ~/sandboxes/msb_5_7_5/use < sys_57.sql © 2014 DBAHIRE.COM 96
  • 97. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Discovering Unused © 2014 DBAHIRE.COM Indexes mysql-5.7.5 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5; +---------------+-------------------+---------------------------------+ | object_schema | object_name | index_name | +---------------+-------------------+---------------------------------+ | osm | acls | acls_k_idx | | osm | changeset_tags | changeset_tags_id_idx | | osm | current_nodes | current_nodes_timestamp_idx | | osm | current_nodes | current_nodes_tile_idx | | osm | current_relations | current_relations_timestamp_idx | +---------------+-------------------+---------------------------------+ 5 rows in set (0.04 sec) mysql-5.7.5 (osm) > SELECT * FROM current_nodes WHERE tile = 100; ... mysql-5.7.5 (osm) > SELECT * FROM sys.schema_unused_indexes LIMIT 5; +---------------+-------------------+---------------------------------+ | object_schema | object_name | index_name | +---------------+-------------------+---------------------------------+ | osm | acls | acls_k_idx | | osm | changeset_tags | changeset_tags_id_idx | | osm | current_nodes | current_nodes_timestamp_idx | | osm | current_relations | current_relations_timestamp_idx | | osm | current_relations | changeset_id | +---------------+-------------------+---------------------------------+ 5 rows in set (0.03 sec) 97 With enough activity, it can help us clean up our schema
  • 98. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Slow Queries (ordered by © 2014 DBAHIRE.COM server time) ********* 1. row ********* query: SELECT `way_id` AS `id` , `v` FROM `way_tags` WHERE `v` LIKE ? db: osm full_scan: * exec_count: 15 err_count: 0 warn_count: 0 total_latency: 7.83 s max_latency: 1.33 s avg_latency: 521.84 ms lock_latency: 17.94 ms rows_sent: 6779 rows_sent_avg: 452 rows_examined: 20152155 rows_examined_avg: 1343477 rows_affected: 0 rows_affected_avg: 0 tmp_tables: 0 tmp_disk_tables: 0 rows_sorted: 0 sort_merge_passes: 0 digest: 21f90695b1ebf20a5f4d4c1e5e860f58 first_seen: 2014-11-01 17:04:51 last_seen: 2014-11-01 17:05:22 98 mysql-5.7.5 (osm) > SELECT * FROM sys.statement_analysis LIMIT 10G
  • 99. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables mysql-5.7.5 (osm) > SELECT * FROM sys.statements_with_temp_tables WHERE db = 'osm' LIMIT 10G *************************** 1. row *************************** query: SELECT ? AS TYPE , `node_id` A ... gs` WHERE `k` = ? AND `v` = ? db: osm exec_count: 11 total_latency: 7.57 s memory_tmp_tables: 11 disk_tmp_tables: 0 avg_tmp_tables_per_query: 1 tmp_tables_to_disk_pct: 0 first_seen: 2014-11-01 17:33:55 last_seen: 2014-11-01 17:34:45 digest: 5e6e82799b7c7c0e5c57cfe63eb98d5d 99
  • 100. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables (cont.) mysql-5.7.5 (osm) > SELECT DIGEST_TEXT FROM performance_schema.events_statements_summary_by_digest WHERE digest = '5e6e82799b7c7c0e5c57cfe63eb98d5d'G *************************** 1. row *************************** DIGEST_TEXT: SELECT ? AS TYPE , `node_id` AS `id` FROM `node_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS TYPE , `way_id` AS `id` FROM `way_tags` WHERE `k` = ? AND `v` = ? UNION SELECT ? AS TYPE , `relation_id` AS `id` FROM `relation_tags` WHERE `k` = ? AND `v` = ? 1 row in set (0.00 sec) mysql-5.7.5 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'; +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ | 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where | | 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where | | 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where | | NULL | UNION RESULT | <union1,2,3> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary | +----+--------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-----------------+ 4 rows in set, 1 warning (0.01 sec) 100
  • 101. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Example Usage: Top Queries Creating © 2014 DBAHIRE.COM Temporary Tables (cont.) mysql-5.7.5 (osm) > EXPLAIN SELECT 'node' as type, node_id as id FROM node_tags WHERE k='amenity' and v='cafe' UNION ALL SELECT 'way' as type, way_id as id FROM way_tags WHERE k='amenity' and v='cafe' UNION ALL SELECT 'relation' as type, relation_id as id FROM relation_tags WHERE k='amenity' and v='cafe'; +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ | 1 | PRIMARY | node_tags | NULL | ALL | NULL | NULL | NULL | NULL | 851339 | 0.00 | Using where | | 2 | UNION | way_tags | NULL | ALL | NULL | NULL | NULL | NULL | 1331016 | 0.00 | Using where | | 3 | UNION | relation_tags | NULL | ALL | NULL | NULL | NULL | NULL | 63201 | 0.00 | Using where | +----+-------------+---------------+------------+------+---------------+------+---------+------+---------+----------+-------------+ 3 rows in set, 1 warning (0.04 sec) 101 No Temporary tables now in 5.7
  • 102. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks CONCLUSIONS © 2014 DBAHIRE.COM 102
  • 103. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM 5.7/10.1 Still in Alpha • New features usually involve new bugs – I only suffered a single crashing bug – Some of the bleeding edge features are not yet optimized for performance and usability • The greatest problem is finding documentation, as some features are only documented on the developer’s blogs – It will probably catch up slowly • It is great to have the lab builds, though, for testing 103
  • 104. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM MySQL 5.7 New Features • MySQL 5.6 seemed Percona Server-inspired • MySQL 5.7 seems MariaDB-inspired – Competition is always good for consumer 104
  • 105. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Many Optimizer Advantages Have to Be © 2014 DBAHIRE.COM Manually Enabled • Modifying on a per-query basis: SET optimizer_switch='batched_key_access=on'; SET join_cache_level=8; # for MariaDB in order to take advantage of them make the features useless unless you are fine-tuning – I expect that to change in the future 105
  • 106. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks I Herby Declare MyISAM as Dead • All major MyISAM-only features are now on MySQL 5.7 – FULLTEXT – GIS – Transportable tables • There are still reasons to use MyISAM – MyISAM is still required for the mysql © 2014 DBAHIRE.COM schema and non-durable temporary tables (WIP) 106
  • 107. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Benchmarks • Do not trust first party benchmarks – In fact, do not trust 3rd party benchmarks either • Only care about the performance of your application running on your hardware 107
  • 108. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Q&A 108
  • 109. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Knowing More about Query Optimization • Tomorrow: Bill Karwin’s session “Query Patterns, Optimized” • “Optimization, Administration and High Availability with MySQL 5.6” course by DBAHire.com on 15th December (Spain) – More dates coming soon © 2014 DBAHIRE.COM 109
  • 110. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks Talk to me During the Percona Live • Talk to me about: – MySQL/MariaDB/Percona – OpenStreetMap – Game of Thrones and I will offer you 1 hour of free MySQL © 2014 DBAHIRE.COM online consulting by myself* 110 *Subjected to availability
  • 111. Query Optimization With MySQL 5.7 and MariaDB 10: Even Newer Tricks © 2014 DBAHIRE.COM Thank You for Attending! • Special thanks to -in ORDER BY rand(): Bill Karwin, Morgan Tocker, Baron Schwartz, Giuseppe Maxia, Ewen Fortune, Henrik Ingo, Tim “do-not- confuse-with-Mark” Callaghan, David Hildebrandt, Cédric Peintre, Kortney Runyan, Miguel Ángel Nieto, Valerii Kravchuk, Aleksandr Kuzminsky and everyone at the Percona and Oracle-MySQL teams and the Percona Live London organization, committee and sponsors • Do not forget, after the session finishes, to please login with your Percona Live account and go to http:// www.percona.com/live/london-2014/sessions/query-optimization- mysql-57-and-mariadb-10-even-newer-tricks to “Rate This Session” for a chance to win great prizes! 111