高CPU消耗MySQL
Posted
技术标签:
【中文标题】高CPU消耗MySQL【英文标题】：High CPU consumption MySQL 【发布时间】：2018-10-22 00:19:47 【问题描述】：我有一个带有 easyengine + wp + redis 的 wordpress 新闻门户，并且遇到性能问题，我的 mysql 消耗了大量的 cpu 使用率 我有超过 500k 的帖子有人帮助我解决这个问题？
打印CPU https://prnt.sc/jgkpwy
服务器详情 Ubuntu 16.04 - 最小 - RAID1 硬盘 256 GB SSD 2.5" 硬盘 256 GB SSD 2.5" RAM 32 GB CPU Intel Xeon E3-1225v3
mysql配置my.cnf
# MariaDB database server configuration file.
#
# You can copy this file to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
max_connections = 300
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 500
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
#
# * MyISAM
#
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched. On error, make copy and try a repair.
myisam_recover_options = BACKUP
key_buffer_size = 128M
#open-files-limit = 2000
table_open_cache = 600
myisam_sort_buffer_size = 512M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
#
# * Query Cache Configuration
#
# Cache only tiny result sets, so we can fit more in the query cache.
query_cache_limit = 2M
query_cache_size = 256M
query_cache_strip_comments =1
# for more write intensive setups, set to DEMAND or OFF
query_cache_type = 1
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# we do want to know about network errors and such
log_warnings = 2
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log[=0|1]
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
#log_slow_rate_limit = 1000
log_slow_verbosity = query_plan
#log-queries-not-using-indexes
#log_slow_admin_statements
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#report_host = master1
#auto_increment_increment = 2
#auto_increment_offset = 1
#log_bin = /var/log/mysql/mariadb-bin
#log_bin_index = /var/log/mysql/mariadb-bin.index
# not fab for performance, but safer
#sync_binlog = 1
#expire_logs_days = 10
#max_binlog_size = 100M
# slaves
#relay_log = /var/log/mysql/relay-bin
#relay_log_index = /var/log/mysql/relay-bin.index
#relay_log_info_file = /var/log/mysql/relay-bin.info
#log_slave_updates
#read_only
#
# If applications support it, this stricter sql_mode prevents some
# mistakes like inserting invalid dates etc.
#sql_mode = NO_ENGINE_SUBSTITUTION,TRADITIONAL
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
default_storage_engine = InnoDB
# you can't just change log file size, requires special procedure
#innodb_log_file_size = 50M
innodb_buffer_pool_size = 15G
innodb_log_buffer_size = 16M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
innodb_buffer_pool_instances = 15
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
#
# * Galera-related settings
#
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completion
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/【问题讨论】：
您可能有运行缓慢的查询。在 MySQL 中，您可以使用SHOW FULL PROCESSLIST 来捕获长时间运行的查询，或者您可以启用慢查询日志。 （更多信息：a2hosting.com/kb/developer-corner/mysql/…）。您很可能需要为您选择的列添加索引，这将大大提高性能并减少 CPU
可能不相关，尽管几个月前我的一个客户遇到了同样的问题，并且是一个糟糕的插件，它计算了破坏数据库的帖子视图。当视图数量呈指数增长时，数据库表会被锁定并一直在重新索引上消耗资源。我的建议是查看每个插件的性能。
当我检查慢查询时出现了这个
谁删除了昨天可见的 ~ 4 个慢查询？
慢查询pastebin.com/raw/E0vHJ190
【参考方案1】：
建议您的 my.cnf-ini [mysqld] 部分考虑
sort_buffer_size=4M # REMOVE or LEAD with # for default
read_buffer_size=2M # REMOVE or LEAD with # for default
read_rnd_buffer_size=1M # REMOVE or LEAD with # for default
innodb_buffer_pool_instances=8 # from 15 will be adequate
thread_cache_size=100 # from 500 see V8 refman CAP at 100 to avoid OOM
innodb_lru_search_depth=128 # from 1024 to be = innodb_buffer_pool_instances
innodb_purge_threads=4 # from likely 1 to support DELETE cleanup
max_write_lock_count=16 # from HUGE # to conserve CPU cycles
query_cache_min_res_unit=512 # from 4096 to support more QC results
请随时通知我们进展情况。
如果您要发布 ulimit -a 和 SHOW GLOBAL STATUS 的文本结果；并显示全局变量；可以提供其他有用的建议。
【讨论】：
按照链接上的要求 ulimit -a pastebin.com/raw/nf6PLDMT SHOW GLOBAL VARIABLES; pastebin.com/raw/pgRcFfV3 显示全球状态； pastebin.com/raw/JpZpLLvw 我在 Skype Skype 上添加了你：redegenial @deborarodrigues 感谢您提供一般日志。完成后，您确实有没有 CLOSE 的 CONNECT。处理完成后，RESCOURCES 不会被释放。请发布完成 CONNECT 的代码，要求 SELECT .... | ...... |通过 QUIT 请求。 @deborarodrigues - "前缀索引" ( KEYpost_name (post_name(191)),) 通常没用。删除(191)。
@deborarodrigues - 当您拥有INDEX(a,b) 时，无需同时拥有INDEX(a)。当您拥有PRIMARY KEY(ID) 时，拥有INDEX(ID, x) 几乎没有任何用处。【参考方案2】：
对 my.cnf-ini [mysqld] 部分的建议 使用 SET GLOBAL variable_name=value 之后；确保正确和可容忍的限制 在移动到下一个变量之前运行一分钟，始终在下一个 SET GLOBAL 之前检查错误日志以确保没有造成任何伤害，然后当您位于列表末尾时使用成功的变量值编辑您的 my.cnf-ini , 关机/重启。
innodb_buffer_pool_size=18*1024*1024*1024 # from 15G for growth
innodb_io_capacity=1000 # from 400 for additional IOPS capacity
read_rnd_buffer_size=128*1024 # from 256K to reduce handler_read_rnd_next
read_buffer_size=256*1024 # from 128K to reduce handler_read_rnd
tmp_table_size=128*1024*1024 # from 32M to expand tmp RAM tbls
max_heap_table_size=128*1024*1024 # from 32M to reduce created_tmp_disk_tables
innodb_lru_scan_depth=128 # from 1024 to reduce CPU use
query_cache_size=50*1024*1024 # from 256M to reduce QC overhead on CPU
query_cache_limit=512*1024 # from 2M for max query result to be stored
updatable_views_with_limit=0 # from YES to reduce handler_external_lock
UVWL 可能必须保持为 YES，请参阅 refman。
我想通过 SKYPE 和 teamviewer 观察您的测试和实施。今天，如果可能的话（早）。
可能无法动态设置和测试的其他 my.cnf-ini [mysqld] 更改包括：
thread_concurrency=30 # from 10 to allow more concurrency
expire_logs_days=5 # from 0 for limited historical logs
key_cache_age_threshold=64800 # from 300 seconds discard to RD again
key_cache_division_limit=50 # for HOT and WARM usage boundary
key_cache_block_size=16384 # from 1024 to age out larger size
进一步分析会发现更多机会。
【讨论】：
小心...key_cache_block_size 限制为 16K。我不相信改变它；我从未见过有人改变它。
由于 OP 似乎在使用 InnoDB，因此 MyISAM 的更改可能无关紧要。
如果他们确实使用 MyISAM 并且性能更好，将会生效。
@RickJames 感谢 mariadb 对 key_cache_block_size 警告的限制。你是对的，检查了裁判并改变了我的建议。非常感谢，【参考方案3】：
您提到的所有慢查询都涉及wp_postmeta。 WP 对该表的架构非常低效。我在 here 中概述了它的几个修复。如果改进该架构后仍然存在问题，我们可以讨论进一步的步骤。
您无法通过调优方式解决高 CPU 问题。但是，有一个设置太高了：
query_cache_size = 256M -- Change to 50M
为了可读性，这里是慢查询之一：
SELECT wp_posts.ID
FROM wp_posts
LEFT JOIN wp_term_relationships
ON ( wp_posts.ID = wp_term_relationships.object_id)
INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id )
INNER JOIN wp_postmeta AS mt1 ON ( om16_posts.ID = mt1.post_id )
WHERE 1=1
AND ( wp_term_relationships.term_taxonomy_id IN (19,37, 38,72) )
AND ( wp_postmeta.meta_key = '_thumbnail_id'
AND ( mt1.meta_key = 'exibir_menu'
AND mt1.meta_value = '1' )
)
AND wp_posts.post_type = 'post'
AND ((wp_posts.post_status = 'publish'))
GROUP BY wp_posts.ID
ORDER BY wp_posts.post_date DESC
LIMIT 0, 7
使用mysqldump时请使用-s t。
附录
鉴于此：
CREATE TABLE `om16_postmeta` (
`meta_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`post_id` bigint(20) unsigned NOT NULL DEFAULT '0',
`meta_key` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`meta_value` longtext COLLATE utf8_unicode_ci,
PRIMARY KEY (`meta_id`),
KEY `post_id` (`post_id`),
KEY `idx_meta_key` (`meta_key`(50)),
KEY `meta_key` (`meta_key`(191))
这可能会以最少的努力为您带来最大的好处：
ALTER TABLE om16_postmeta
DROP PRIMARY KEY,
DROP INDEX post_id,
ADD PRIMARY KEY(post_id, meta_key), -- much better for queries
ADD INDEX(meta_id), -- sufficient for AUTO_INCREMENT
DROP INDEX idx_meta_key, -- prefix indexing unnecessary an inefficient
DROP INDEX meta_key,
ADD INDEX(meta_key);
注意：执行ALTER 需要一些时间；该表将被阻止。
如果 ALTER 导致“重复密钥”，显然对于给定的“帖子”，您多次拥有相同的“密钥”。要么
ADD PRIMARY KEY(post_id, meta_key, meta_id), 。
【讨论】：
Rick，如果我已经从 wp-postmeta 创建了包含 400 万条信息的表，我应该如何处理您的提示？创建表 wp_postmeta ( post_id BIGINT UNSIGNED NOT NULL, meta_key VARCHAR (255) NOT NULL, meta_value LONGTEXT NOT NULL, PRIMARY KEY (post_id, meta_key) INDEX (meta_key) ) ENGINE = InnoDB; @deborarodrigues - 这是当前模式？如果是这样，我会更深入地挖掘。 @deborarodrigues - 请提供EXPLAIN SELECT ...。
@deborarodrigues - 在我漂亮打印的示例中，LEFT 是否必要？切换到INNER JOIN 将为优化器提供更多选择。
ALTER TABLE 的语法与 CREATE TABLE 不同。 （参见手册。）在ALTER 中，您一一指定更改 -- ALTER TABLE foo MODIFY COLUMN ..., ADD INDEX (...), etc;以上是关于高CPU消耗MySQL的主要内容，如果未能解决你的问题，请参考以下文章