Linux服务器缓存总是不够用
Posted 悟能的师兄
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Linux服务器缓存总是不够用相关的知识，希望对你有一定的参考价值。
一：问题描述
服务的 Buffer/Cache 总是在高位运行，可用的缓存一直不够用
二：问题分析
这个缓存，我用了定时清理方式，但是治标不治本，用户量大了后，还是经常不够用，怎么办嘞？
那就看一看这个 Buffer/Cache 里面到底存了什么数据？
想要看 Buffer/Cache 我们要安装一个神器
[root@iZbp1gp1t778obaz5m8vk8Z ~]# wget https://silenceshell-1255345740.cos.ap-shanghai.myqcloud.com/hcache
[root@iZbp1gp1t778obaz5m8vk8Z ~]# chmod +x hcache
[root@iZbp1gp1t778obaz5m8vk8Z ~]# mv hcache /usr/local/bin/
[root@iZbp1gp1t778obaz5m8vk8Z ~]#
查看前十个占用缓存最大的文件
[root@iZbp1gp1t778obaz5m8vk8Z ~]# hcache -top 10
+-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001847356-0005c705938eb664.journal | 117440512 | 28672 | 16792 | 058.566 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001804329-0005c6c0d699f6b5.journal | 117440512 | 28672 | 15558 | 054.262 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001913bd6-0005c7b5d2a0a068.journal | 117440512 | 28672 | 14873 | 051.873 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001830cc0-0005c6f1d16335e1.journal | 117440512 | 28672 | 14501 | 050.575 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001709205-0005c5ec8d98c3bd.journal | 117440512 | 28672 | 14401 | 050.227 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001695bce-0005c591f4f9b6c2.journal | 117440512 | 28672 | 13798 | 048.124 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000167ef8b-0005c5801512bfe8.journal | 117440512 | 28672 | 13798 | 048.124 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000018b8036-0005c76fcfab5444.journal | 117440512 | 28672 | 13027 | 045.435 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000177c190-0005c64db1792194.journal | 117440512 | 28672 | 12802 | 044.650 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001941e20-0005c7dd75f05239.journal | 117440512 | 28672 | 12740 | 044.434 |
+-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
分析得出是因为我们的日志文件大量占用缓存
CentOS系统中有两个日志服务，分别是传统的 rsyslog 和 systemd-journal
systemd-journald是一个改进型日志管理服务，可以收集来自内核、系统早期启动阶段的日志、系统守护进程在启动和运行中的标准输出和错误信息，还有syslog的日志。
该日志服务仅仅把日志集中保存在单一结构的日志文件/run/log中，由于日志是经历过压缩和格式化的二进制数据，所以在查看和定位的时候很迅速。
默认情况下并不会持久化保存日志，只会保留一个月的日志。另外，一些rsyslog无法收集的日志也会被journal记录到。
rsyslog作为传统的系统日志服务，把所有收集到的日志都记录到/var/log/目录下的各个日志文件中。
原来是系统日志在作祟，那就去看看。
二.一 细化分析
[root@iZbp1gp1t778obaz5m8vk8Z journal]# ll
total 16
drwxr-sr-x+ 2 root systemd-journal 12288 Jul 27 02:49 20190711105006363114529432776998
[root@iZbp1gp1t778obaz5m8vk8Z journal]#
[root@iZbp1gp1t778obaz5m8vk8Z journal]# du -sh
3.9G .
三：处理问题
大气出奇迹，开干，把系统日志的最大控制在500M，物理文件被删除 3.3G
[root@iZbp1gp1t778obaz5m8vk8Z journal]# journalctl --vacuum-size=500M
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000165276b-0005c559b355a8c7.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001668bbf-0005c56d9fdc61f3.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000167ef8b-0005c5801512bfe8.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001695bce-0005c591f4f9b6c2.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000016ad657-0005c5a0f50b116c.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000016c3cf5-0005c5bf172b8aca.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000016da438-0005c5d49cd6b6fd.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000016f11aa-0005c5e198551ec4.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001709205-0005c5ec8d98c3bd.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001720988-0005c5f7f7f78e0f.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017383da-0005c607981114f1.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000174f435-0005c61abbac5d31.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001765e21-0005c631203d3e02.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000177c190-0005c64db1792194.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017927bc-0005c65ff00bbafb.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017a968d-0005c67116a09e54.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017bfd07-0005c685c9cf1389.journal (112.0M).
Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017d6d81-0005c698038e2ad2.journal (112.0M).
Deleted archived journal
...................................................................
Vacuuming done, freed 3.3G of archived journals on disk.
[root@iZbp1gp1t778obaz5m8vk8Z journal]#
再看一下我们的物理路径下面，以及我们的内存状态，唉 物理路径下只有500M文件了，而我们的缓存处给我空出来 1.5G 可用内存！！！
[root@iZbp1gp1t778obaz5m8vk8Z journal]#
[root@iZbp1gp1t778obaz5m8vk8Z journal]# du -sh
497M .
[root@iZbp1gp1t778obaz5m8vk8Z journal]#
[root@iZbp1gp1t778obaz5m8vk8Z journal]#
[root@iZbp1gp1t778obaz5m8vk8Z journal]# free -h
total used free shared buff/cache available
Mem: 15G 9.6G 4.6G 1.5M 831M 5.0G
Swap: 0B 0B 0B
[root@iZbp1gp1t778obaz5m8vk8Z journal]# 再看一下我们的缓存占用情况，结果日志文件还是瑶瑶领先，但是终于看到其他的服务了，先这样OK运行。
[root@iZbp1gp1t778obaz5m8vk8Z journal]#
[root@iZbp1gp1t778obaz5m8vk8Z journal]# hcache -top 10
+-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001913bd6-0005c7b5d2a0a068.journal | 117440512 | 28672 | 14873 | 051.873 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001941e20-0005c7dd75f05239.journal | 117440512 | 28672 | 12740 | 044.434 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001958066-0005c7face4cfe97.journal | 117440512 | 28672 | 12522 | 043.673 |
| /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000192aebd-0005c7c5f63c62af.journal | 117440512 | 28672 | 9982 | 034.814 |
| /var/log/journal/20190711105006363114529432776998/system.journal | 25165824 | 6144 | 6144 | 100.000 |
| /usr/bin/dockerd-current | 33420192 | 8160 | 3934 | 048.211 |
| /usr/local/jdk1.8.0_172/jre/lib/amd64/server/libjvm.so | 17067833 | 4167 | 2507 | 060.163 |
| /usr/local/share/aliyun-assist/2.2.3.221/aliyun-service | 15611914 | 3812 | 1952 | 051.207 |
| /usr/bin/docker-containerd-current | 10806584 | 2639 | 1217 | 046.116 |
| /usr/local/nginx/sbin/nginx | 6099568 | 1490 | 1024 | 068.725 |
+-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
四：总结
从运行方向来看，我这台阿里云服务器的缓存，大部分可用内存均是系统日志占用，但是这个系统日志占用了超大量的内存，这一点阿里云需要改善，毕竟访问量大就日志疯狂的存储占用可用内存，这有点做得可不够好。
然后大家如果遇到同样的问题，可以根据自己的情况适当的配置日志可占用的大小情况。
希望跟大家一起进步！！！
参考资料：
《Linux中清理/var/log/journal/垃圾日志》
五：补充学习
[root@iZbp1f0xuq9rc41s6gdvfyZ log]#
[root@iZbp1f0xuq9rc41s6gdvfyZ log]# ps aux|grep tomcat
root 16583 0.0 0.0 112708 984 pts/1 S+ 09:47 0:00 grep --color=auto tomcat
zcsjw 30375 2.5 40.4 11693112 6576120 ? Sl Jun28 1072:50 /usr/local/jdk1.8.0_172/jre/bin/java -Djava.util.logging.config.file=/usr/local/www/apache-tomcat-8.5.43/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -XX:PermSize=1024M -XX:MaxPermSize=4096m -Xms2048M -Xmx4096M -XX:MaxNewSize=1024m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -server -Xms2048m -Xmx4096m -Dignore.endorsed.dirs= -classpath /usr/local/www/apache-tomcat-8.5.43/bin/bootstrap.jar:/usr/local/www/apache-tomcat-8.5.43/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/www/apache-tomcat-8.5.43 -Dcatalina.home=/usr/local/www/apache-tomcat-8.5.43 -Djava.io.tmpdir=/usr/local/www/apache-tomcat-8.5.43/temp org.apache.catalina.startup.Bootstrap start
[root@iZbp1f0xuq9rc41s6gdvfyZ log]#
[root@iZbp1f0xuq9rc41s6gdvfyZ log]#
[root@iZbp1f0xuq9rc41s6gdvfyZ log]# hcache -pid 30375
+---------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
| Name | Size (bytes) | Pages | Cached | Percent |
|---------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
| /usr/local/jdk1.8.0_172/jre/lib/amd64/libnet.so | 115485 | 29 | 18 | 062.069 |
| /usr/local/www/apache-tomcat-8.5.43/webapps/guides/WEB-INF/lib/aopalliance-1.0.jar | 4467 | 2 | 2 | 100.000 |
| /usr/local/www/apache-tomcat-8.5.43/webapps/guides/WEB-INF/lib/commons-httpclient-3.1.jar | 305001 | 75 | 75 | 100.000 |
| /usr/local/www/apache-tomcat-8.5.43/webapps/guides/WEB-INF/lib/mybatis-spring-1.2.2.jar | 48783 | 12 | 12 | 100.000 |
| /usr/lib/fontconfig/cache/83bf7a3f-cf3f-4a35-a43e-c013d1e66acb-le64.cache-7 | 20592 | 6 | 6 | 100.000 |
| /usr/local/www/apache-tomcat-8.5.43/webapps/guides/WEB-INF/lib/codejson-1.1.1.jar | 102621 | 26 | 26 | 100.000 |
| /usr/lib64/libresolv-2.17.so | 105824 | 26 | 26 | 100.000 |
.......................................................以上是关于Linux服务器缓存总是不够用的主要内容，如果未能解决你的问题，请参考以下文章