SlideShare une entreprise Scribd logo
1  sur  53
Télécharger pour lire hors ligne
Deep dive
into
PostgreSQL
Statistics.
Алексей Лесовский
О чем доклад
Статистика активности postgresql.
Как эффективно использовать статистику.
Решение проблем на основе данных статистики.
https://goo.gl/d0pSYK
Чего НЕ будет
Статистика планировщика запросов.
Обзор инструментов.
Итоговая цель
Умение использовать статистику - это полезно.
Статистика в постгресе - это не страшно.
Какую статистику выбрать для решения частной
задачи.
Черный ящик
$ ps hf -u postgres -o cmd
/usr/pgsql-9.4/bin/postgres -D /var/lib/pgsql/9.4/data
_ postgres: logger process
_ postgres: checkpointer process
_ postgres: writer process
_ postgres: wal writer process
_ postgres: autovacuum launcher process
_ postgres: stats collector process
_ postgres: postgres pgbench [local] idle in transaction
_ postgres: postgres pgbench [local] idle
_ postgres: postgres pgbench [local] UPDATE
_ postgres: postgres pgbench [local] UPDATE waiting
_ postgres: postgres pgbench [local] UPDATE
Как тратится время
Write Ahead Log
Shared
Buffers
Buffers IO Autovacuum Workers
Autovacuum Launcher
Background Workers
Indexes IO
Query Execution
Query Planning
Client Backends Postmaster
Relations IO
Logger Process Stats Collector
Logical
Replication
WAL Sender
Process
Archiver
Process
Background
Writer
Checkpointer
Process
Network Storage
Recovery Process
WAL Receiver Process
Tables/Indexes Data Files
Проблемы
Информации много (~109 метрик для 9.4).
Статистика представлена online счетчиками.
Нет истории (но есть функции сброса статистики).
Отсутствие native инструмента.
Множество сторонних инструментов.
Проблемы
Информации много (~109 метрик для 9.4).
Статистика представлена online счетчиками.
Нет истории (но есть функции сброса статистики).
Отсутствие native инструмента.
Множество сторонних инструментов.
Важно уметь брать статистику напрямую.
Нужны базовые знания SQL.
Что предлагают?
События, происходящие в базе.
Свойства объектов базы (баз, таблиц, индексов).
Время, затраченное событием.
Источники статы
Счетчики в shared memory.
Функции.
Встроенные представления (VIEWs).
Официальный пакет contribs.
Неофициальные contrib'ы.
Источники статы
pg_stat_all_tables
pg_current_xlog_location()
pg_replication_slots
pg_stat_replication
pg_stat_archiver
pg_stat_database_conflicts
pg_buffercache
pg_stat_database
pg_stat_user_tables
pg_stat_database
pg_stat_bgwrier
pgstattuple
pg_table_size()
pg_database_size
pg_index_size()
pg_stat_activity
EXPLAIN
pg_locks
pg_stat_user_functions
pg_stat_all_indexes
pg_statio_all_indexes
pg_statio_all_tables
pg_xlog_location_diff()
pg_stat_kcache
pg_stat_statements
pg_roles
pg_database
pg_class
pg_relation_size()
pg_total_relation_size()
current_setting()
pg_current_xlog_insert_location()
pg_last_xact_replay_timestamp()
pg_last_xlog_replay_location()
pg_tablespace_size()
pg_ls_dir()
Write Ahead Log
Shared
Buffers
Buffers IO Autovacuum Workers
Autovacuum Launcher
Background Workers
Indexes IO
Query Execution
Query Planning
Client Backends Postmaster
Tables IO
Logger Process Stats Collector
Logical
Replication
WAL Sender
Process
Archiver
Process
Background
Writer
Checkpointer
Process
Network Storage
Recovery Process
WAL Receiver Process
Tables/Indexes Data Files
pg_stat_all_tables
pg_current_xlog_location
pg_replication_slots
pg_stat_replication
pg_stat_archiver
pg_stat_database_conflicts
pg_buffercache
pg_stat_database
pg_stat_activity
pg_stat_user_tables
pg_stat_database
pg_stat_bgwrier
pgstattuplepg_table_size
pg_database_size
pg_index_size
pg_stat_activity
EXPLAIN
pg_stat_statements
pg_locks
pg_stat_user_functions
pg_stat_all_indexes
pg_stat_database
pg_statio_all_indexes
pg_statio_all_tables
pg_xlog_location_diff
nicstat* iostat*
pg_stat_kcache
pg_stat_statements
pg_stat_database
$ select * from pg_stat_database where datname = 'shop';
-[ RECORD 1 ]--+-------------------------------------------
datid | 16414
datname | shop
numbackends | 34
xact_commit | 51167155051
xact_rollback | 44781878826
blks_read | 7978770895
blks_hit | 9683551077519
tup_returned | 12507331807583
tup_fetched | 3885840966616
tup_inserted | 2898024857
tup_updated | 3082071349
tup_deleted | 2751363323
conflicts | 0
temp_files | 377675
temp_bytes | 4783712399875
deadlocks | 151
blk_read_time | 214344118.089
blk_write_time | 1260880.747
stats_reset | 2015-05-31 11:37:52.017967+03
Cache hit ratio
$ select * from pg_stat_database;
...
blks_read | 7978770895
blks_hit | 9683551077519
...
$ select
sum(blks_hit)*100/sum(blks_hit+blks_read) as hit_ratio
from pg_stat_database;
Больше = лучше, но не меньше 90%
Аномалии
$ select * from pg_stat_database;
...
xact_commit | 51167155051
xact_rollback | 44781878826
conflicts | 0
temp_files | 377675
temp_bytes | 4783712399875
deadlocks | 151
...
$ select
datname,
(xact_commit*100)/(xact_commit+xact_rollback) as c_ratio,
deadlocks, conflicts,
temp_file, pg_size_pretty(temp_bytes) as temp_size
from pg_stat_database;
Аномалии
$ select * from pg_stat_database;
...
xact_commit | 51167155051 < 95 %
xact_rollback | 44781878826 > 5 %
conflicts | 0 > 10
temp_files | 377675 > 100
temp_bytes | 4783712399875 > 10 GB
deadlocks | 151 > 10
...
$ select
datname,
(xact_commit*100)/(xact_commit+xact_rollback) as c_ratio,
deadlocks, conflicts,
temp_file, pg_size_pretty(temp_bytes) as temp_size
from pg_stat_database;
pg_stat_bgwriter
$ select * from pg_stat_bgwriter;
-[ RECORD 1 ]---------+------------------------------------
checkpoints_timed | 3267
checkpoints_req | 6
checkpoint_write_time | 10416478591
checkpoint_sync_time | 405039
buffers_checkpoint | 2518065526
buffers_clean | 99602799
maxwritten_clean | 157
buffers_backend | 219356924
buffers_backend_fsync | 0
buffers_alloc | 3477374822
stats_reset | 2015-05-31 11:09:48.413185+03
Checkpoint's
$ select * from pg_stat_bgwriter;
...
checkpoints_timed | 3267
checkpoints_req | 6
...
checkpoints_req > checkpoints_timed = плохо
postgresql.conf
●
checkpoint_segments
●
checkpoint_timeout
●
checkpoint_completion_target
Background Writer
$ select * from pg_stat_bgwriter;
...
buffers_clean | 99602799
maxwritten_clean | 157 > 0
buffers_backend_fsync | 0 > 0
...
Обеспечивает бэкенды чистыми страницами.
Разгружает checkpointer.
postgresql.conf
●
bgwriter_delay
●
bgwriter_lru_maxpages
●
bgwriter_lru_multiplier
pg_stat_replication
$ select * from pg_stat_replication;
-[ RECORD 1 ]----+------------------------------------
pid | 26921
usesysid | 15588142
usename | replica
application_name | walreceiver
client_addr | 10.0.0.7
client_hostname |
client_port | 32956
backend_start | 2015-10-01 19:14:42.979377+03
state | streaming
sent_location | 1691/EEE65900
write_location | 1691/EEE65900
flush_location | 1691/EEE65900
replay_location | 1691/EEE658D0
sync_priority | 0
sync_state | async
Лаг репликации
$ select * from pg_stat_replication;
...
sent_location | 1691/EEE65900
write_location | 1691/EEE65900
flush_location | 1691/EEE65900
replay_location | 1691/EEE658D0
...
1692/EEE65900 — позиция в журнале транзакций.
Все значения равны = идеально.
Лаг репликации
Причины отставания:
Networking,
Storage,
CPU.
сколько записано xlog в байтах
$ select
pg_xlog_location_diff(pg_current_xlog_location(),'0/00000000');
лаг репликации в байтах
$ select
client_addr,
pg_xlog_location_diff(pg_current_xlog_location(), replay_location)
from pg_stat_replication;
лаг репликации в секундах
$ select
extract(epoch from now() - pg_last_xact_replay_timestamp());
pg_stat_all_tables
$ select * from pg_stat_all_tables;
-[ RECORD 1 ]-------+------------------------------------
relid | 98221
schemaname | public
relname | clientsession
seq_scan | 192
seq_tup_read | 364544695
idx_scan | 2691939318
idx_tup_fetch | 2669551448
n_tup_ins | 239532851
n_tup_upd | 736119030
n_tup_del | 239898968
n_tup_hot_upd | 497688344
n_live_tup | 3493472
n_dead_tup | 1606
n_mod_since_analyze | 349526
...
Sequential scans
$ select * from pg_stat_all_tables;
...
seq_scan | 192 необязательно плохо
seq_tup_read | 364544695 > 1000 (seq_tup_avg)
...
$ select
relname,
pg_size_pretty(pg_relation_size(relname::regclass)) as size,
seq_scan, seq_tup_read,
seq_scan / seq_tup_read as seq_tup_avg
from pg_stat_user_tables
where seq_tup_read > 0 order by 3,4 desc limit 5;
Размеры таблиц
$ select
relname,
pg_size_pretty(pg_total_relation_size(relname::regclass)) as
full_size,
pg_size_pretty(pg_relation_size(relname::regclass)) as
table_size,
pg_size_pretty(pg_total_relation_size(relname::regclass) -
pg_relation_size(relname::regclass)) as index_size
from pg_stat_user_tables
order by pg_total_relation_size(relname::regclass) desc limit 10;
Также есть метакоманды psql: dt+ и di+
Write activity
$ select * from pg_stat_all_tables;
...
n_tup_upd | 736119030
n_tup_hot_upd | 497688344
...
UPDATE = DELETE + INSERT
UPDATE = TABLE UPDATE + INDEX UPDATE
Write activity
$ select
s.relname,
pg_size_pretty(pg_relation_size(relid)),
coalesce(n_tup_ins,0) + 2 * coalesce(n_tup_upd,0) -
coalesce(n_tup_hot_upd,0) + coalesce(n_tup_del,0) AS total_writes,
(coalesce(n_tup_hot_upd,0)::float * 100 / (case when n_tup_upd > 0
then n_tup_upd else 1 end)::float)::numeric(10,2) AS hot_rate,
(select v[1] FROM regexp_matches(reloptions::text,E'fillfactor=(d+)') as
r(v) limit 1) AS fillfactor
from pg_stat_all_tables s
join pg_class c ON c.oid=relid
order by total_writes desc limit 50;
Что такое Heap-Only Tuples?
HOT не вызывает перестроения индекса.
HOT только для тех значений которые не участвуют в индексе
Большое значение n_tup_hot_upd = хорошо.
Как добиться большого n_tup_hot_upd?
Write activity
| pg_size_pretty | total_writes | hot_rate | fillfactor
-+----------------+--------------+----------+------------
| 9418 MB | 391459091 | 5.03 |
| 417 MB | 285948760 | 0.00 |
| 76 MB | 232031972 | 87.31 | 70
| 7123 MB | 124281107 | 99.36 | 70
Fillfactor определяет резерв свободного места в странице.
Накладные расходы на размер таблиц (индексов).
ALTER TABLE table_name SET (fillfactor = 70);
Autovacuum queue
$ select c.relname,
current_setting('autovacuum_vacuum_threshold') as av_base_thresh,
current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor,
(current_setting('autovacuum_vacuum_threshold')::int +
(current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples))
as av_thresh,
s.n_dead_tup
from pg_stat_user_tables s join pg_class c ON s.relname = c.relname
where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int
+ (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples));
| av_base_thresh | av_scale_factor | av_thresh | n_dead_tup
+----------------+-----------------+-----------+------------
| 50 | 0.01 | 60302 | 85445
| 50 | 0.01 | 50 | 1349
| 50 | 0.01 | 55 | 12479
| 50 | 0.01 | 103 | 3134
| 50 | 0.01 | 52716 | 173100
Autovacuum queue
$ select c.relname,
current_setting('autovacuum_vacuum_threshold') as av_base_thresh,
current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor,
(current_setting('autovacuum_vacuum_threshold')::int +
(current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples))
as av_thresh,
s.n_dead_tup
from pg_stat_user_tables s join pg_class c ON s.relname = c.relname
where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int
+ (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples));
| av_base_thresh | av_scale_factor | av_thresh | n_dead_tup
+----------------+-----------------+-----------+------------
| 50 | 0.01 | 60302 | 60545
| 50 | 0.01 | 50 | 1349
| 50 | 0.01 | 55 | 12479
| 50 | 0.01 | 103 | 3134
| 50 | 0.01 | 52716 | 173100
Avito Experience
https://github.com/eshkinkot/avito/blob/master/munin/vacuum_queue
pg_stat_all_indexes
$ select * from pg_stat_all_indexes where idx_scan = 0;
-[ RECORD 1 ]-+------------------------------------------
relid | 98242
indexrelid | 55732253
schemaname | public
relname | products
indexrelname | products_special2_idx
idx_scan | 0
idx_tup_read | 0
idx_tup_fetch | 0
Лишние индексы
$ select * from pg_stat_all_indexes where idx_scan = 0;
...
indexrelname | products_special2_idx
idx_scan | 0 0 = плохо
...
https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/lo
w_used_indexes.sql
http://www.databasesoup.com/2014/05/new-finding-unused-indexes-q
uery.html
Лишние индексы
$ select * from pg_stat_all_indexes where idx_scan = 0;
...
indexrelname | products_special2_idx
idx_scan | 0 0 = плохо
...
https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/lo
w_used_indexes.sql
http://www.databasesoup.com/2014/05/new-finding-unused-indexes-q
uery.html
Неиспользуемые индексы - плохо.
Занимают место.
Замедляют UPDATE, DELETE, INSERT операции.
Лишняя работа для VACUUM.
pg_stat_activity
$ select * from pg_stat_activity;
-[ RECORD 1 ]----+--------------------------------------------
datid | 16401
datname | ts
pid | 116408
usesysid | 16384
usename | tsagent
application_name | unicorn_rails worker
client_addr | 172.17.0.37
client_hostname |
client_port | 50888
backend_start | 2015-10-14 15:18:03.01039+00
xact_start |
query_start | 2015-10-14 15:21:30.336325+00
state_change | 2015-10-14 15:21:30.33635+00
waiting | f
state | idle
backend_xid |
backend_xmin |
query | COMMIT
Общая активность
$ select * from pg_stat_activity;
...
datname | ts
usename | tsagent
client_addr | 172.17.0.37
...
общий процент подключений
$ select
count(*)*100/(select current_setting('max_connections')::int)
from pg_stat_activity;
Общая активность
$ select * from pg_stat_activity;
...
datname | ts
usename | tsagent
client_addr | 172.17.0.37
...
$ select
client_addr, usename, datname, count(*)
from pg_stat_activity group by 1,2,3 order by 4 desc;
client_addr | usename | datname | count
-------------+-----------+---------+-------
127.0.0.1 | cron_role | ts | 508
127.0.0.1 | tsagent | ts | 60
127.0.0.1 | postgres | ts | 1
127.0.0.1 | pupkin | ts | 1
Долгие запросы
$ select * from pg_stat_activity;
...
backend_start | 2015-10-14 15:18:03.01039+00
xact_start | 2015-10-14 15:21:15.336325+00
query_start | 2015-10-14 15:21:30.336325+00
state_change | 2015-10-14 15:21:30.33635+00
...
Долгие запросы
$ select * from pg_stat_activity;
...
backend_start | 2015-10-14 15:18:03.01039+00
xact_start | 2015-10-14 15:21:15.336325+00
query_start | 2015-10-14 15:21:30.336325+00
state_change | 2015-10-14 15:21:30.33635+00
...
$ select
client_addr, usename, datname,
clock_timestamp() - xact_start as xact_age,
clock_timestamp() - query_start as query_age,
query
from pg_stat_activity order by xact_start, query_start;
Долгие запросы
$ select * from pg_stat_activity;
...
backend_start | 2015-10-14 15:18:03.01039+00
xact_start | 2015-10-14 15:21:15.336325+00
query_start | 2015-10-14 15:21:30.336325+00
state_change | 2015-10-14 15:21:30.33635+00
...
$ select
client_addr, usename, datname,
clock_timestamp() - xact_start as xact_age,
clock_timestamp() - query_start as query_age,
query
from pg_stat_activity order by xact_start, query_start;
clock_timestamp() для вычисления времени работы.
Запросы запоминаем, отстреливаем, оптимизируем (EXPLAIN).
Плохие транзакции
$ select * from pg_stat_activity where state in
('idle in transaction', 'idle in transaction (aborted)';
...
xact_start | 2015-10-14 15:21:21.128192+00
query_start | 2015-10-14 15:21:30.336325+00
state_change | 2015-10-14 15:21:30.33635+00
state | idle in transaction
...
Плохие транзакции
$ select * from pg_stat_activity where state in
('idle in transaction', 'idle in transaction (aborted)';
...
xact_start | 2015-10-14 15:21:21.128192+00
query_start | 2015-10-14 15:21:30.336325+00
state_change | 2015-10-14 15:21:30.33635+00
state | idle in transaction
...
idle in transaction, idle in transaction (aborted) = плохо
Повод для беспокойств: > 5
clock_timestamp() для вычисления времени работы.
Транзакции отстреливаем, приложение оптимизируем.
Блокировки
$ select * from pg_stat_activity where waiting;
...
xact_start | 2015-10-14 15:21:21.128192+00
query_start | 2015-10-14 15:21:30.336325+00
state_change | 2015-10-14 15:21:30.33635+00
waiting | t
...
Блокировки
$ select * from pg_stat_activity where waiting;
...
xact_start | 2015-10-14 15:21:21.128192+00
query_start | 2015-10-14 15:21:30.336325+00
state_change | 2015-10-14 15:21:30.33635+00
waiting | t
...
waiting = true = плохо.
clock_timestamp() для вычисления времени работы.
pg_locks для поиска источника блокировки.
Транзакции отстреливаем, приложение оптимизируем.
Блокировки
https://github.com/lesovsky/uber-scripts/blob/master/postgresql/sql/c4_06_sho
w_locked_queries.sql
http://big-elephants.com/2013-09/exploring-query-locks-in-postgres/
locked_item | products
waiting | t
waiting_query | update products set avail = false where id = 154;
waiting_mode | RowExclusiveLock
waiting_xact_age | 00:00:07.103685
waiting_query_age | 00:00:01.813508
waiting_pid | 5895
waiting_user | app_role
waiting_state | active
waiting | f
locking_query | alter table products drop column avail;
locking_mode | AccessExclusiveLock
locking_xact_age | 00:00:16.416441
locking_query_age | 00:00:14.416441
locking_pid | 5844
locking_user | postgres
locking_state | idle in transaction
pg_stat_statements
$ select * from pg_stat_statements
where blk_read_time <> 0 order by blk_read_time desc;
-[ RECORD 1 ]-------+---------------------------------------
userid | 25078444
dbid | 16411
query | SELECT "id" FROM run_plan_xact(?)
calls | 11165832
total_time | 11743325.6880088
rows | 11165832
shared_blks_hit | 351353214
shared_blks_read | 205557
shared_blks_dirtied | 256053
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 68894
local_blks_dirtied | 68894
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 495425.535999976
blk_write_time | 0
pg_stat_statements
$ select * from pg_stat_statements;
...
query | SELECT "id" FROM run_plan_xact(?)
calls | 11165832
total_time | 11743325.6880088
rows | 11165832
blk_read_time | 495425.535999976
blk_write_time | 0
Cреднее время запроса в милисекундах
$ select (sum(total_time) / sum(calls))::numeric(6,3)
from pg_stat_statements;
Самые активно пишущие (в shared_buffers) запросы
$ select query, shared_blks_dirtied
from pg_stat_statements
where shared_blks_dirtied > 0 order by 2 desc;
Отчеты
query total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%)
сalls: 476 (0.00%) rows: 476,000
avg_time: 118881.54ms (IO: 21.2%)
user: app_user db: ustats
query: select
filepath, type, deviceuid
from imv5event
where
state = ?::eventstate
and servertime between $1 and $2
order by servertime desc LIMIT $3 OFFSET $4
https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/glob
al_reports/query_stat_total.sql
Отчеты
query total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%)
сalls: 476 (0.00%) rows: 476,000
avg_time: 118881.54ms (IO: 21.2%)
user: app_user db: ustats
query: select
filepath, type, deviceuid
from imv5event
where
state = ?::eventstate
and servertime between $1 and $2
order by servertime desc LIMIT $3 OFFSET $4
Используем sum() для подсчет общей статистики.
Вычисляем «вклад» запроса в общую статистику.
Использование ресурсов (CPU, IO).
За кадром
pg_statio_all_tables, pg_statio_all_indexes
pg_stat_user_functions
Функции определения размеров - df *size*
pgstattuple (contrib)
●
точное определение bloat для индексов и таблиц
●
чем больше размер таблицы, тем больше времени уйдет на
оценку
pg_buffercache (contrib)
●
инспекция shared buffers
●
большие накладые расходы (buffers lock)
За кадром
pgfincore
●
низкоуровневые операции с таблицами через mincore().
●
инспекция OS page cache.
pg_stat_kcache
●
использование getrusage() до и после запроса.
●
cpu usage и дисковый ввод-вывод.
●
требует pg_stat_statements и postgresql-9.4.
●
почти не влияет на производительность.
Резюме
●
Умение пользоваться статистикой полезно.
●
Статистика в постгресе - это не сложно.
●
Статистика поможет ответить на вопросы.
●
Экспериментируйте.
Годные ссылки
Автор, пиши ещё
www.blog.postgresql-consulting.com (eng)
www.thislinux.org (rus)
The Statistics Collector
http://www.postgresql.org/docs/current/static/monitoring-stats.html
System Administration Functions
http://www.postgresql.org/docs/current/static/functions-admin.html
Contrib modules
http://www.postgresql.org/docs/current/static/pgstatstatements.html
http://www.postgresql.org/docs/current/static/pgstattuple.html
http://www.postgresql.org/docs/current/static/pgbuffercache.html
https://github.com/klando/pgfincore
https://github.com/dalibo/pg_stat_kcache
SQL utils and sql code examples
https://github.com/PostgreSQL-Consulting/pg-utils
Write Ahead Log
Shared
Buffers
Buffers IO Autovacuum Workers
Autovacuum Launcher
Background Workers
Indexes IO
Query Execution
Query Planning
Client Backends Postmaster
Tables IO
Logger Process Stats Collector
Logical
Replication
WAL Sender
Process
Archiver
Process
Background
Writer
Checkpointer
Process
Network Storage
Recovery Process
WAL Receiver Process
Tables/Indexes Data Files
pg_stat_all_tables
pg_current_xlog_location
pg_replication_slots
pg_stat_replication
pg_stat_archiver
pg_stat_database_conflicts
pg_buffercache
pg_stat_database
pg_stat_activity
pg_stat_user_tables
pg_stat_database
pg_stat_bgwrier
pgstattuplepg_table_size
pg_database_size
pg_index_size
pg_stat_activity
EXPLAIN
pg_stat_statements
pg_locks
pg_stat_user_functions
pg_stat_all_indexes
pg_stat_database
pg_statio_all_indexes
pg_statio_all_tables
pg_xlog_location_diff
nicstat* iostat*
pg_stat_kcache
pg_stat_statements
Спасибо. Вопросы?

Contenu connexe

Tendances

How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
PostgreSQL-Consulting
 

Tendances (19)

Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.
 
Linux tuning for PostgreSQL at Secon 2015
Linux tuning for PostgreSQL at Secon 2015Linux tuning for PostgreSQL at Secon 2015
Linux tuning for PostgreSQL at Secon 2015
 
PostgreSQL Troubleshoot On-line, (RITfest 2015 meetup at Moscow, Russia).
PostgreSQL Troubleshoot On-line, (RITfest 2015 meetup at Moscow, Russia).PostgreSQL Troubleshoot On-line, (RITfest 2015 meetup at Moscow, Russia).
PostgreSQL Troubleshoot On-line, (RITfest 2015 meetup at Moscow, Russia).
 
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
 
collectd & PostgreSQL
collectd & PostgreSQLcollectd & PostgreSQL
collectd & PostgreSQL
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLTop 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
 
PostgreSQL Administration for System Administrators
PostgreSQL Administration for System AdministratorsPostgreSQL Administration for System Administrators
PostgreSQL Administration for System Administrators
 
PostgreSQL and RAM usage
PostgreSQL and RAM usagePostgreSQL and RAM usage
PostgreSQL and RAM usage
 
SCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
SCALE 15x Minimizing PostgreSQL Major Version Upgrade DowntimeSCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
SCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime
 
PostgreSQL Replication Tutorial
PostgreSQL Replication TutorialPostgreSQL Replication Tutorial
PostgreSQL Replication Tutorial
 
GitLab PostgresMortem: Lessons Learned
GitLab PostgresMortem: Lessons LearnedGitLab PostgresMortem: Lessons Learned
GitLab PostgresMortem: Lessons Learned
 
Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
PostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication CheatsheetPostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication Cheatsheet
 
Sessionization with Spark streaming
Sessionization with Spark streamingSessionization with Spark streaming
Sessionization with Spark streaming
 
PostgreSQL: Joining 1 million tables
PostgreSQL: Joining 1 million tablesPostgreSQL: Joining 1 million tables
PostgreSQL: Joining 1 million tables
 
Rmoug ashmaster
Rmoug ashmasterRmoug ashmaster
Rmoug ashmaster
 
Walbouncer: Filtering PostgreSQL transaction log
Walbouncer: Filtering PostgreSQL transaction logWalbouncer: Filtering PostgreSQL transaction log
Walbouncer: Filtering PostgreSQL transaction log
 

En vedette

Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...
Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...
Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...
Ontico
 

En vedette (15)

PostgreSQL: Advanced features in practice
PostgreSQL: Advanced features in practicePostgreSQL: Advanced features in practice
PostgreSQL: Advanced features in practice
 
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 Vienna
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 ViennaAutovacuum, explained for engineers, new improved version PGConf.eu 2015 Vienna
Autovacuum, explained for engineers, new improved version PGConf.eu 2015 Vienna
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practice
 
Linux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performanceLinux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performance
 
Народные средства оптимизации PostgreSQL
Народные средства оптимизации PostgreSQLНародные средства оптимизации PostgreSQL
Народные средства оптимизации PostgreSQL
 
Как HeadHunter удалось безопасно нарушить RFC 793 (TCP) и обойти сетевые лову...
Как HeadHunter удалось безопасно нарушить RFC 793 (TCP) и обойти сетевые лову...Как HeadHunter удалось безопасно нарушить RFC 793 (TCP) и обойти сетевые лову...
Как HeadHunter удалось безопасно нарушить RFC 793 (TCP) и обойти сетевые лову...
 
Highload 2014. PostgreSQL: ups, DevOps.
Highload 2014. PostgreSQL: ups, DevOps.Highload 2014. PostgreSQL: ups, DevOps.
Highload 2014. PostgreSQL: ups, DevOps.
 
PostgreSQL Advanced Queries
PostgreSQL Advanced QueriesPostgreSQL Advanced Queries
PostgreSQL Advanced Queries
 
PostgreSQL Vacuum: Nine Circles of Hell
PostgreSQL Vacuum: Nine Circles of HellPostgreSQL Vacuum: Nine Circles of Hell
PostgreSQL Vacuum: Nine Circles of Hell
 
Linux internals for Database administrators at Linux Piter 2016
Linux internals for Database administrators at Linux Piter 2016Linux internals for Database administrators at Linux Piter 2016
Linux internals for Database administrators at Linux Piter 2016
 
Tuning Linux for your database FLOSSUK 2016
Tuning Linux for your database FLOSSUK 2016Tuning Linux for your database FLOSSUK 2016
Tuning Linux for your database FLOSSUK 2016
 
Nine Circles of Inferno or Explaining the PostgreSQL Vacuum
Nine Circles of Inferno or Explaining the PostgreSQL VacuumNine Circles of Inferno or Explaining the PostgreSQL Vacuum
Nine Circles of Inferno or Explaining the PostgreSQL Vacuum
 
10 things, an Oracle DBA should care about when moving to PostgreSQL
10 things, an Oracle DBA should care about when moving to PostgreSQL10 things, an Oracle DBA should care about when moving to PostgreSQL
10 things, an Oracle DBA should care about when moving to PostgreSQL
 
Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...
Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...
Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...
 
PostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya Kosmodemiansky
PostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya KosmodemianskyPostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya Kosmodemiansky
PostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya Kosmodemiansky
 

Similaire à Deep dive into PostgreSQL statistics.

Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...
Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...
Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...
Ontico
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
Command Prompt., Inc
 
Fatkulin presentation
Fatkulin presentationFatkulin presentation
Fatkulin presentation
Enkitec
 
Save Coding Time with Proc SQL.ppt
Save Coding Time with Proc SQL.pptSave Coding Time with Proc SQL.ppt
Save Coding Time with Proc SQL.ppt
ssuser660bb1
 

Similaire à Deep dive into PostgreSQL statistics. (20)

Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...
Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...
Deep dive into PostgreSQL internal statistics / Алексей Лесовский (PostgreSQL...
 
Explain this!
Explain this!Explain this!
Explain this!
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
Aplicações 10x a 100x mais rápida com o postgre sql
Aplicações 10x a 100x mais rápida com o postgre sqlAplicações 10x a 100x mais rápida com o postgre sql
Aplicações 10x a 100x mais rápida com o postgre sql
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl
Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas FittlMonitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl
Monitoring Postgres at Scale | PostgresConf US 2018 | Lukas Fittl
 
Osol Pgsql
Osol PgsqlOsol Pgsql
Osol Pgsql
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
 
Around the world with extensions | PostgreSQL Conference Europe 2018 | Craig ...
Around the world with extensions | PostgreSQL Conference Europe 2018 | Craig ...Around the world with extensions | PostgreSQL Conference Europe 2018 | Craig ...
Around the world with extensions | PostgreSQL Conference Europe 2018 | Craig ...
 
Percona Live UK 2014 Part III
Percona Live UK 2014  Part IIIPercona Live UK 2014  Part III
Percona Live UK 2014 Part III
 
New features in Performance Schema 5.7 in action
New features in Performance Schema 5.7 in actionNew features in Performance Schema 5.7 in action
New features in Performance Schema 5.7 in action
 
New Ways to Find Latency in Linux Using Tracing
New Ways to Find Latency in Linux Using TracingNew Ways to Find Latency in Linux Using Tracing
New Ways to Find Latency in Linux Using Tracing
 
What’s new in 9.6, by PostgreSQL contributor
What’s new in 9.6, by PostgreSQL contributorWhat’s new in 9.6, by PostgreSQL contributor
What’s new in 9.6, by PostgreSQL contributor
 
Tek tutorial
Tek tutorialTek tutorial
Tek tutorial
 
Linux Systems Performance 2016
Linux Systems Performance 2016Linux Systems Performance 2016
Linux Systems Performance 2016
 
Fatkulin presentation
Fatkulin presentationFatkulin presentation
Fatkulin presentation
 
gumiStudy#2 実践 memcached
gumiStudy#2 実践 memcachedgumiStudy#2 実践 memcached
gumiStudy#2 実践 memcached
 
実践 memcached
実践 memcached実践 memcached
実践 memcached
 
Save Coding Time with Proc SQL.ppt
Save Coding Time with Proc SQL.pptSave Coding Time with Proc SQL.ppt
Save Coding Time with Proc SQL.ppt
 

Plus de Alexey Lesovsky

Отладка и устранение проблем в PostgreSQL Streaming Replication.
Отладка и устранение проблем в PostgreSQL Streaming Replication.Отладка и устранение проблем в PostgreSQL Streaming Replication.
Отладка и устранение проблем в PostgreSQL Streaming Replication.
Alexey Lesovsky
 

Plus de Alexey Lesovsky (9)

Отладка и устранение проблем в PostgreSQL Streaming Replication.
Отладка и устранение проблем в PostgreSQL Streaming Replication.Отладка и устранение проблем в PostgreSQL Streaming Replication.
Отладка и устранение проблем в PostgreSQL Streaming Replication.
 
Call of Postgres: Advanced Operations (part 5)
Call of Postgres: Advanced Operations (part 5)Call of Postgres: Advanced Operations (part 5)
Call of Postgres: Advanced Operations (part 5)
 
Call of Postgres: Advanced Operations (part 4)
Call of Postgres: Advanced Operations (part 4)Call of Postgres: Advanced Operations (part 4)
Call of Postgres: Advanced Operations (part 4)
 
Call of Postgres: Advanced Operations (part 3)
Call of Postgres: Advanced Operations (part 3)Call of Postgres: Advanced Operations (part 3)
Call of Postgres: Advanced Operations (part 3)
 
Call of Postgres: Advanced Operations (part 2)
Call of Postgres: Advanced Operations (part 2)Call of Postgres: Advanced Operations (part 2)
Call of Postgres: Advanced Operations (part 2)
 
Call of Postgres: Advanced Operations (part 1)
Call of Postgres: Advanced Operations (part 1)Call of Postgres: Advanced Operations (part 1)
Call of Postgres: Advanced Operations (part 1)
 
PostgreSQL Streaming Replication
PostgreSQL Streaming ReplicationPostgreSQL Streaming Replication
PostgreSQL Streaming Replication
 
Tuning Linux for Databases.
Tuning Linux for Databases.Tuning Linux for Databases.
Tuning Linux for Databases.
 
Streaming replication in practice
Streaming replication in practiceStreaming replication in practice
Streaming replication in practice
 

Dernier

Activity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdfActivity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdf
ciinovamais
 
Beyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global ImpactBeyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global Impact
PECB
 
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK  LOOKBOOK(1) (1).pdfBASLIQ CURRENT LOOKBOOK  LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
SoniaTolstoy
 
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
fonyou31
 

Dernier (20)

Activity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdfActivity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdf
 
IGNOU MSCCFT and PGDCFT Exam Question Pattern: MCFT003 Counselling and Family...
IGNOU MSCCFT and PGDCFT Exam Question Pattern: MCFT003 Counselling and Family...IGNOU MSCCFT and PGDCFT Exam Question Pattern: MCFT003 Counselling and Family...
IGNOU MSCCFT and PGDCFT Exam Question Pattern: MCFT003 Counselling and Family...
 
APM Welcome, APM North West Network Conference, Synergies Across Sectors
APM Welcome, APM North West Network Conference, Synergies Across SectorsAPM Welcome, APM North West Network Conference, Synergies Across Sectors
APM Welcome, APM North West Network Conference, Synergies Across Sectors
 
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
 
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
 
Holdier Curriculum Vitae (April 2024).pdf
Holdier Curriculum Vitae (April 2024).pdfHoldier Curriculum Vitae (April 2024).pdf
Holdier Curriculum Vitae (April 2024).pdf
 
Beyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global ImpactBeyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global Impact
 
Disha NEET Physics Guide for classes 11 and 12.pdf
Disha NEET Physics Guide for classes 11 and 12.pdfDisha NEET Physics Guide for classes 11 and 12.pdf
Disha NEET Physics Guide for classes 11 and 12.pdf
 
A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy Reform
 
Web & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdfWeb & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdf
 
fourth grading exam for kindergarten in writing
fourth grading exam for kindergarten in writingfourth grading exam for kindergarten in writing
fourth grading exam for kindergarten in writing
 
Student login on Anyboli platform.helpin
Student login on Anyboli platform.helpinStudent login on Anyboli platform.helpin
Student login on Anyboli platform.helpin
 
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK  LOOKBOOK(1) (1).pdfBASLIQ CURRENT LOOKBOOK  LOOKBOOK(1) (1).pdf
BASLIQ CURRENT LOOKBOOK LOOKBOOK(1) (1).pdf
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The Basics
 
Advanced Views - Calendar View in Odoo 17
Advanced Views - Calendar View in Odoo 17Advanced Views - Calendar View in Odoo 17
Advanced Views - Calendar View in Odoo 17
 
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
Ecosystem Interactions Class Discussion Presentation in Blue Green Lined Styl...
 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and Mode
 
Mattingly "AI & Prompt Design: The Basics of Prompt Design"
Mattingly "AI & Prompt Design: The Basics of Prompt Design"Mattingly "AI & Prompt Design: The Basics of Prompt Design"
Mattingly "AI & Prompt Design: The Basics of Prompt Design"
 
Arihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdfArihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdf
 
social pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajansocial pharmacy d-pharm 1st year by Pragati K. Mahajan
social pharmacy d-pharm 1st year by Pragati K. Mahajan
 

Deep dive into PostgreSQL statistics.

  • 2. О чем доклад Статистика активности postgresql. Как эффективно использовать статистику. Решение проблем на основе данных статистики. https://goo.gl/d0pSYK
  • 3. Чего НЕ будет Статистика планировщика запросов. Обзор инструментов.
  • 4. Итоговая цель Умение использовать статистику - это полезно. Статистика в постгресе - это не страшно. Какую статистику выбрать для решения частной задачи.
  • 5. Черный ящик $ ps hf -u postgres -o cmd /usr/pgsql-9.4/bin/postgres -D /var/lib/pgsql/9.4/data _ postgres: logger process _ postgres: checkpointer process _ postgres: writer process _ postgres: wal writer process _ postgres: autovacuum launcher process _ postgres: stats collector process _ postgres: postgres pgbench [local] idle in transaction _ postgres: postgres pgbench [local] idle _ postgres: postgres pgbench [local] UPDATE _ postgres: postgres pgbench [local] UPDATE waiting _ postgres: postgres pgbench [local] UPDATE
  • 6. Как тратится время Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Relations IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files
  • 7. Проблемы Информации много (~109 метрик для 9.4). Статистика представлена online счетчиками. Нет истории (но есть функции сброса статистики). Отсутствие native инструмента. Множество сторонних инструментов.
  • 8. Проблемы Информации много (~109 метрик для 9.4). Статистика представлена online счетчиками. Нет истории (но есть функции сброса статистики). Отсутствие native инструмента. Множество сторонних инструментов. Важно уметь брать статистику напрямую. Нужны базовые знания SQL.
  • 9. Что предлагают? События, происходящие в базе. Свойства объектов базы (баз, таблиц, индексов). Время, затраченное событием.
  • 10. Источники статы Счетчики в shared memory. Функции. Встроенные представления (VIEWs). Официальный пакет contribs. Неофициальные contrib'ы.
  • 12. Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Tables IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files pg_stat_all_tables pg_current_xlog_location pg_replication_slots pg_stat_replication pg_stat_archiver pg_stat_database_conflicts pg_buffercache pg_stat_database pg_stat_activity pg_stat_user_tables pg_stat_database pg_stat_bgwrier pgstattuplepg_table_size pg_database_size pg_index_size pg_stat_activity EXPLAIN pg_stat_statements pg_locks pg_stat_user_functions pg_stat_all_indexes pg_stat_database pg_statio_all_indexes pg_statio_all_tables pg_xlog_location_diff nicstat* iostat* pg_stat_kcache pg_stat_statements
  • 13. pg_stat_database $ select * from pg_stat_database where datname = 'shop'; -[ RECORD 1 ]--+------------------------------------------- datid | 16414 datname | shop numbackends | 34 xact_commit | 51167155051 xact_rollback | 44781878826 blks_read | 7978770895 blks_hit | 9683551077519 tup_returned | 12507331807583 tup_fetched | 3885840966616 tup_inserted | 2898024857 tup_updated | 3082071349 tup_deleted | 2751363323 conflicts | 0 temp_files | 377675 temp_bytes | 4783712399875 deadlocks | 151 blk_read_time | 214344118.089 blk_write_time | 1260880.747 stats_reset | 2015-05-31 11:37:52.017967+03
  • 14. Cache hit ratio $ select * from pg_stat_database; ... blks_read | 7978770895 blks_hit | 9683551077519 ... $ select sum(blks_hit)*100/sum(blks_hit+blks_read) as hit_ratio from pg_stat_database; Больше = лучше, но не меньше 90%
  • 15. Аномалии $ select * from pg_stat_database; ... xact_commit | 51167155051 xact_rollback | 44781878826 conflicts | 0 temp_files | 377675 temp_bytes | 4783712399875 deadlocks | 151 ... $ select datname, (xact_commit*100)/(xact_commit+xact_rollback) as c_ratio, deadlocks, conflicts, temp_file, pg_size_pretty(temp_bytes) as temp_size from pg_stat_database;
  • 16. Аномалии $ select * from pg_stat_database; ... xact_commit | 51167155051 < 95 % xact_rollback | 44781878826 > 5 % conflicts | 0 > 10 temp_files | 377675 > 100 temp_bytes | 4783712399875 > 10 GB deadlocks | 151 > 10 ... $ select datname, (xact_commit*100)/(xact_commit+xact_rollback) as c_ratio, deadlocks, conflicts, temp_file, pg_size_pretty(temp_bytes) as temp_size from pg_stat_database;
  • 17. pg_stat_bgwriter $ select * from pg_stat_bgwriter; -[ RECORD 1 ]---------+------------------------------------ checkpoints_timed | 3267 checkpoints_req | 6 checkpoint_write_time | 10416478591 checkpoint_sync_time | 405039 buffers_checkpoint | 2518065526 buffers_clean | 99602799 maxwritten_clean | 157 buffers_backend | 219356924 buffers_backend_fsync | 0 buffers_alloc | 3477374822 stats_reset | 2015-05-31 11:09:48.413185+03
  • 18. Checkpoint's $ select * from pg_stat_bgwriter; ... checkpoints_timed | 3267 checkpoints_req | 6 ... checkpoints_req > checkpoints_timed = плохо postgresql.conf ● checkpoint_segments ● checkpoint_timeout ● checkpoint_completion_target
  • 19. Background Writer $ select * from pg_stat_bgwriter; ... buffers_clean | 99602799 maxwritten_clean | 157 > 0 buffers_backend_fsync | 0 > 0 ... Обеспечивает бэкенды чистыми страницами. Разгружает checkpointer. postgresql.conf ● bgwriter_delay ● bgwriter_lru_maxpages ● bgwriter_lru_multiplier
  • 20. pg_stat_replication $ select * from pg_stat_replication; -[ RECORD 1 ]----+------------------------------------ pid | 26921 usesysid | 15588142 usename | replica application_name | walreceiver client_addr | 10.0.0.7 client_hostname | client_port | 32956 backend_start | 2015-10-01 19:14:42.979377+03 state | streaming sent_location | 1691/EEE65900 write_location | 1691/EEE65900 flush_location | 1691/EEE65900 replay_location | 1691/EEE658D0 sync_priority | 0 sync_state | async
  • 21. Лаг репликации $ select * from pg_stat_replication; ... sent_location | 1691/EEE65900 write_location | 1691/EEE65900 flush_location | 1691/EEE65900 replay_location | 1691/EEE658D0 ... 1692/EEE65900 — позиция в журнале транзакций. Все значения равны = идеально.
  • 22. Лаг репликации Причины отставания: Networking, Storage, CPU. сколько записано xlog в байтах $ select pg_xlog_location_diff(pg_current_xlog_location(),'0/00000000'); лаг репликации в байтах $ select client_addr, pg_xlog_location_diff(pg_current_xlog_location(), replay_location) from pg_stat_replication; лаг репликации в секундах $ select extract(epoch from now() - pg_last_xact_replay_timestamp());
  • 23. pg_stat_all_tables $ select * from pg_stat_all_tables; -[ RECORD 1 ]-------+------------------------------------ relid | 98221 schemaname | public relname | clientsession seq_scan | 192 seq_tup_read | 364544695 idx_scan | 2691939318 idx_tup_fetch | 2669551448 n_tup_ins | 239532851 n_tup_upd | 736119030 n_tup_del | 239898968 n_tup_hot_upd | 497688344 n_live_tup | 3493472 n_dead_tup | 1606 n_mod_since_analyze | 349526 ...
  • 24. Sequential scans $ select * from pg_stat_all_tables; ... seq_scan | 192 необязательно плохо seq_tup_read | 364544695 > 1000 (seq_tup_avg) ... $ select relname, pg_size_pretty(pg_relation_size(relname::regclass)) as size, seq_scan, seq_tup_read, seq_scan / seq_tup_read as seq_tup_avg from pg_stat_user_tables where seq_tup_read > 0 order by 3,4 desc limit 5;
  • 25. Размеры таблиц $ select relname, pg_size_pretty(pg_total_relation_size(relname::regclass)) as full_size, pg_size_pretty(pg_relation_size(relname::regclass)) as table_size, pg_size_pretty(pg_total_relation_size(relname::regclass) - pg_relation_size(relname::regclass)) as index_size from pg_stat_user_tables order by pg_total_relation_size(relname::regclass) desc limit 10; Также есть метакоманды psql: dt+ и di+
  • 26. Write activity $ select * from pg_stat_all_tables; ... n_tup_upd | 736119030 n_tup_hot_upd | 497688344 ... UPDATE = DELETE + INSERT UPDATE = TABLE UPDATE + INDEX UPDATE
  • 27. Write activity $ select s.relname, pg_size_pretty(pg_relation_size(relid)), coalesce(n_tup_ins,0) + 2 * coalesce(n_tup_upd,0) - coalesce(n_tup_hot_upd,0) + coalesce(n_tup_del,0) AS total_writes, (coalesce(n_tup_hot_upd,0)::float * 100 / (case when n_tup_upd > 0 then n_tup_upd else 1 end)::float)::numeric(10,2) AS hot_rate, (select v[1] FROM regexp_matches(reloptions::text,E'fillfactor=(d+)') as r(v) limit 1) AS fillfactor from pg_stat_all_tables s join pg_class c ON c.oid=relid order by total_writes desc limit 50; Что такое Heap-Only Tuples? HOT не вызывает перестроения индекса. HOT только для тех значений которые не участвуют в индексе Большое значение n_tup_hot_upd = хорошо. Как добиться большого n_tup_hot_upd?
  • 28. Write activity | pg_size_pretty | total_writes | hot_rate | fillfactor -+----------------+--------------+----------+------------ | 9418 MB | 391459091 | 5.03 | | 417 MB | 285948760 | 0.00 | | 76 MB | 232031972 | 87.31 | 70 | 7123 MB | 124281107 | 99.36 | 70 Fillfactor определяет резерв свободного места в странице. Накладные расходы на размер таблиц (индексов). ALTER TABLE table_name SET (fillfactor = 70);
  • 29. Autovacuum queue $ select c.relname, current_setting('autovacuum_vacuum_threshold') as av_base_thresh, current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor, (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)) as av_thresh, s.n_dead_tup from pg_stat_user_tables s join pg_class c ON s.relname = c.relname where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)); | av_base_thresh | av_scale_factor | av_thresh | n_dead_tup +----------------+-----------------+-----------+------------ | 50 | 0.01 | 60302 | 85445 | 50 | 0.01 | 50 | 1349 | 50 | 0.01 | 55 | 12479 | 50 | 0.01 | 103 | 3134 | 50 | 0.01 | 52716 | 173100
  • 30. Autovacuum queue $ select c.relname, current_setting('autovacuum_vacuum_threshold') as av_base_thresh, current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor, (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)) as av_thresh, s.n_dead_tup from pg_stat_user_tables s join pg_class c ON s.relname = c.relname where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)); | av_base_thresh | av_scale_factor | av_thresh | n_dead_tup +----------------+-----------------+-----------+------------ | 50 | 0.01 | 60302 | 60545 | 50 | 0.01 | 50 | 1349 | 50 | 0.01 | 55 | 12479 | 50 | 0.01 | 103 | 3134 | 50 | 0.01 | 52716 | 173100 Avito Experience https://github.com/eshkinkot/avito/blob/master/munin/vacuum_queue
  • 31. pg_stat_all_indexes $ select * from pg_stat_all_indexes where idx_scan = 0; -[ RECORD 1 ]-+------------------------------------------ relid | 98242 indexrelid | 55732253 schemaname | public relname | products indexrelname | products_special2_idx idx_scan | 0 idx_tup_read | 0 idx_tup_fetch | 0
  • 32. Лишние индексы $ select * from pg_stat_all_indexes where idx_scan = 0; ... indexrelname | products_special2_idx idx_scan | 0 0 = плохо ... https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/lo w_used_indexes.sql http://www.databasesoup.com/2014/05/new-finding-unused-indexes-q uery.html
  • 33. Лишние индексы $ select * from pg_stat_all_indexes where idx_scan = 0; ... indexrelname | products_special2_idx idx_scan | 0 0 = плохо ... https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/lo w_used_indexes.sql http://www.databasesoup.com/2014/05/new-finding-unused-indexes-q uery.html Неиспользуемые индексы - плохо. Занимают место. Замедляют UPDATE, DELETE, INSERT операции. Лишняя работа для VACUUM.
  • 34. pg_stat_activity $ select * from pg_stat_activity; -[ RECORD 1 ]----+-------------------------------------------- datid | 16401 datname | ts pid | 116408 usesysid | 16384 usename | tsagent application_name | unicorn_rails worker client_addr | 172.17.0.37 client_hostname | client_port | 50888 backend_start | 2015-10-14 15:18:03.01039+00 xact_start | query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | f state | idle backend_xid | backend_xmin | query | COMMIT
  • 35. Общая активность $ select * from pg_stat_activity; ... datname | ts usename | tsagent client_addr | 172.17.0.37 ... общий процент подключений $ select count(*)*100/(select current_setting('max_connections')::int) from pg_stat_activity;
  • 36. Общая активность $ select * from pg_stat_activity; ... datname | ts usename | tsagent client_addr | 172.17.0.37 ... $ select client_addr, usename, datname, count(*) from pg_stat_activity group by 1,2,3 order by 4 desc; client_addr | usename | datname | count -------------+-----------+---------+------- 127.0.0.1 | cron_role | ts | 508 127.0.0.1 | tsagent | ts | 60 127.0.0.1 | postgres | ts | 1 127.0.0.1 | pupkin | ts | 1
  • 37. Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ...
  • 38. Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ... $ select client_addr, usename, datname, clock_timestamp() - xact_start as xact_age, clock_timestamp() - query_start as query_age, query from pg_stat_activity order by xact_start, query_start;
  • 39. Долгие запросы $ select * from pg_stat_activity; ... backend_start | 2015-10-14 15:18:03.01039+00 xact_start | 2015-10-14 15:21:15.336325+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 ... $ select client_addr, usename, datname, clock_timestamp() - xact_start as xact_age, clock_timestamp() - query_start as query_age, query from pg_stat_activity order by xact_start, query_start; clock_timestamp() для вычисления времени работы. Запросы запоминаем, отстреливаем, оптимизируем (EXPLAIN).
  • 40. Плохие транзакции $ select * from pg_stat_activity where state in ('idle in transaction', 'idle in transaction (aborted)'; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 state | idle in transaction ...
  • 41. Плохие транзакции $ select * from pg_stat_activity where state in ('idle in transaction', 'idle in transaction (aborted)'; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 state | idle in transaction ... idle in transaction, idle in transaction (aborted) = плохо Повод для беспокойств: > 5 clock_timestamp() для вычисления времени работы. Транзакции отстреливаем, приложение оптимизируем.
  • 42. Блокировки $ select * from pg_stat_activity where waiting; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | t ...
  • 43. Блокировки $ select * from pg_stat_activity where waiting; ... xact_start | 2015-10-14 15:21:21.128192+00 query_start | 2015-10-14 15:21:30.336325+00 state_change | 2015-10-14 15:21:30.33635+00 waiting | t ... waiting = true = плохо. clock_timestamp() для вычисления времени работы. pg_locks для поиска источника блокировки. Транзакции отстреливаем, приложение оптимизируем.
  • 44. Блокировки https://github.com/lesovsky/uber-scripts/blob/master/postgresql/sql/c4_06_sho w_locked_queries.sql http://big-elephants.com/2013-09/exploring-query-locks-in-postgres/ locked_item | products waiting | t waiting_query | update products set avail = false where id = 154; waiting_mode | RowExclusiveLock waiting_xact_age | 00:00:07.103685 waiting_query_age | 00:00:01.813508 waiting_pid | 5895 waiting_user | app_role waiting_state | active waiting | f locking_query | alter table products drop column avail; locking_mode | AccessExclusiveLock locking_xact_age | 00:00:16.416441 locking_query_age | 00:00:14.416441 locking_pid | 5844 locking_user | postgres locking_state | idle in transaction
  • 45. pg_stat_statements $ select * from pg_stat_statements where blk_read_time <> 0 order by blk_read_time desc; -[ RECORD 1 ]-------+--------------------------------------- userid | 25078444 dbid | 16411 query | SELECT "id" FROM run_plan_xact(?) calls | 11165832 total_time | 11743325.6880088 rows | 11165832 shared_blks_hit | 351353214 shared_blks_read | 205557 shared_blks_dirtied | 256053 shared_blks_written | 0 local_blks_hit | 0 local_blks_read | 68894 local_blks_dirtied | 68894 local_blks_written | 0 temp_blks_read | 0 temp_blks_written | 0 blk_read_time | 495425.535999976 blk_write_time | 0
  • 46. pg_stat_statements $ select * from pg_stat_statements; ... query | SELECT "id" FROM run_plan_xact(?) calls | 11165832 total_time | 11743325.6880088 rows | 11165832 blk_read_time | 495425.535999976 blk_write_time | 0 Cреднее время запроса в милисекундах $ select (sum(total_time) / sum(calls))::numeric(6,3) from pg_stat_statements; Самые активно пишущие (в shared_buffers) запросы $ select query, shared_blks_dirtied from pg_stat_statements where shared_blks_dirtied > 0 order by 2 desc;
  • 47. Отчеты query total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%) сalls: 476 (0.00%) rows: 476,000 avg_time: 118881.54ms (IO: 21.2%) user: app_user db: ustats query: select filepath, type, deviceuid from imv5event where state = ?::eventstate and servertime between $1 and $2 order by servertime desc LIMIT $3 OFFSET $4 https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/glob al_reports/query_stat_total.sql
  • 48. Отчеты query total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%) сalls: 476 (0.00%) rows: 476,000 avg_time: 118881.54ms (IO: 21.2%) user: app_user db: ustats query: select filepath, type, deviceuid from imv5event where state = ?::eventstate and servertime between $1 and $2 order by servertime desc LIMIT $3 OFFSET $4 Используем sum() для подсчет общей статистики. Вычисляем «вклад» запроса в общую статистику. Использование ресурсов (CPU, IO).
  • 49. За кадром pg_statio_all_tables, pg_statio_all_indexes pg_stat_user_functions Функции определения размеров - df *size* pgstattuple (contrib) ● точное определение bloat для индексов и таблиц ● чем больше размер таблицы, тем больше времени уйдет на оценку pg_buffercache (contrib) ● инспекция shared buffers ● большие накладые расходы (buffers lock)
  • 50. За кадром pgfincore ● низкоуровневые операции с таблицами через mincore(). ● инспекция OS page cache. pg_stat_kcache ● использование getrusage() до и после запроса. ● cpu usage и дисковый ввод-вывод. ● требует pg_stat_statements и postgresql-9.4. ● почти не влияет на производительность.
  • 51. Резюме ● Умение пользоваться статистикой полезно. ● Статистика в постгресе - это не сложно. ● Статистика поможет ответить на вопросы. ● Экспериментируйте.
  • 52. Годные ссылки Автор, пиши ещё www.blog.postgresql-consulting.com (eng) www.thislinux.org (rus) The Statistics Collector http://www.postgresql.org/docs/current/static/monitoring-stats.html System Administration Functions http://www.postgresql.org/docs/current/static/functions-admin.html Contrib modules http://www.postgresql.org/docs/current/static/pgstatstatements.html http://www.postgresql.org/docs/current/static/pgstattuple.html http://www.postgresql.org/docs/current/static/pgbuffercache.html https://github.com/klando/pgfincore https://github.com/dalibo/pg_stat_kcache SQL utils and sql code examples https://github.com/PostgreSQL-Consulting/pg-utils
  • 53. Write Ahead Log Shared Buffers Buffers IO Autovacuum Workers Autovacuum Launcher Background Workers Indexes IO Query Execution Query Planning Client Backends Postmaster Tables IO Logger Process Stats Collector Logical Replication WAL Sender Process Archiver Process Background Writer Checkpointer Process Network Storage Recovery Process WAL Receiver Process Tables/Indexes Data Files pg_stat_all_tables pg_current_xlog_location pg_replication_slots pg_stat_replication pg_stat_archiver pg_stat_database_conflicts pg_buffercache pg_stat_database pg_stat_activity pg_stat_user_tables pg_stat_database pg_stat_bgwrier pgstattuplepg_table_size pg_database_size pg_index_size pg_stat_activity EXPLAIN pg_stat_statements pg_locks pg_stat_user_functions pg_stat_all_indexes pg_stat_database pg_statio_all_indexes pg_statio_all_tables pg_xlog_location_diff nicstat* iostat* pg_stat_kcache pg_stat_statements Спасибо. Вопросы?