Linux 缓存服务varnish

Posted 2020-11-12

Varnish是一款高性能的开源HTTP加速器,Squid服务也是，两者的关系就像apache跟nginx，Varnish更年轻轻量,Squid更成熟稳重。
Varnish4.0工作工作流程如下图

各个点说明
vcl_recv：接收请求
vcl_pass, 跳过缓存
vcl_hit, 缓存命中
vcl_miss, 缓存丢失
vcl_pipe, 非http
vcl_purge, 手动清楚缓存为过期
vcl_synth, 手动清楚缓存过期同步
vcl_deliver 发送给客户端
每个节点上的变量作用域如下图

内建变量：

1、req.*：request，表示由客户端发来的请求报文相关；
req.http.User-Agent, req.http.Referer, ...
2、bereq.*：由varnish发往BE主机的httpd请求相关；
3、 beresp.*：由BE主机响应给varnish的响应报文相关；
4、 resp.*：由varnish响应给client相关；
5、obj.*：存储在缓存空间中的缓存对象的属性；只读；

常用变量：

bereq.http.HEADERS
bereq.request：请求方法；
bereq.url：请求的url；
bereq.proto：请求的协议版本；
bereq.backend：指明要调用的后端主机；
req.http.Cookie：客户端的请求报文中Cookie首部的值；
req.http.User-Agent ~ "chrome"
//-------------------------------------
beresp.http.HEADERS
beresp.status：响应的状态码；
reresp.proto：协议版本；
beresp.backend.name：BE主机的主机名；
beresp.ttl：BE主机响应的内容的余下的可缓存时长；
//-------------------------------------
obj.hits：此对象从缓存中命中的次数；
obj.ttl：对象的ttl值
//-------------------------------------
server.ip
server.hostname
//-------------------------------------
client.ip

---

Varnish的特长体现在缓存命中vcl_hit，以及缓存清理vcl_purge方便。

简单使用说明

1、 安装

[[email protected] ~]# yum install varnish

2、 查看

[[email protected] ~]# rpm -ql varnish
/etc/varnish   #配置文件目录
/etc/varnish/default.vcl      #配置各Child/Cache线程的缓存策略；
/etc/varnish/varnish.params   # 配置varnish服务进程的工作特性，例如监听的地址和端口，缓存机制；
/run/varnish.pid

/usr/bin/varnishadm  #客户端脚本
/usr/bin/varnishtest  #测试工具程序：

#Shared Memory Log交互工具：
/usr/bin/varnishhist
/usr/bin/varnishlog
/usr/bin/varnishncsa
/usr/bin/varnishstat
/usr/bin/varnishtop

/usr/sbin/varnishd  #主程序
/usr/sbin/varnish_reload_vcl  #编译配置文件

#启动脚本
/usr/lib/systemd/system/varnish.service
/usr/lib/systemd/system/varnishlog.service     #持久化日志  二选一
/usr/lib/systemd/system/varnishncsa.service    #持久化日志  二选一

3、配置文件

[[email protected] varnish]# vim /etc/varnish/varnish.params
RELOAD_VCL=1   #启动程序后能否reload配置文件
VARNISH_VCL_CONF=/etc/varnish/default.vcl   #
VARNISH_LISTEN_PORT=80 #默认6081 端口   如果直接当web服务器用改80

# Admin interface listen address and port  管理端口地址
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret  #密钥

#VARNISH_STORAGE="malloc,256M"  #缓存
VARNISH_STORAGE="file,/www/data/varnish,1g"

VARNISH_USER=varnish
VARNISH_GROUP=varnish

[[email protected] data]# mkdir varnish/cache
[[email protected] data]# chown -R varnish.varnish varnish

4、启动

[[email protected] varnish]# systemctl start varnish
[[email protected] varnish]# ss -lntup|grep varnish
tcp    LISTEN     0      128       *:80                    *:*                   users:(("varnishd",pid=1765,fd=7))
tcp    LISTEN     0      10     127.0.0.1:6082                  *:*                   users:(("varnishd",pid=1764,fd=6))
tcp    LISTEN     0      128      :::80                   :::*                   users:(("varnishd",pid=1765,fd=8))

5、缓存策略后台配置

[[email protected] varnish]# vim /etc/varnish/default.vcl
backend default {
    .host = "192.168.1.202";
    .port = "80";
}

6、重载

Loading vcl from /etc/varnish/default.vcl
Current running config name is
Using new config name reload_2018-06-06T09:17:17
VCL compiled.
VCL ‘reload_2018-06-06T09:17:17‘ now active
available       0 boot
active          0 reload_2018-06-06T09:17:17

Done

7、admin管理

[[email protected] varnish]# varnishadm -h
varnishadm: invalid option -- ‘h‘
usage: varnishadm [-n ident] [-t timeout] [-S secretfile] -T [address]:port command [...]
    -n is mutually exlusive with -S and -T

[[email protected] varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
200
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,3.10.0-693.el7.x86_64,x86_64,-sfile,-smalloc,-hcritbit
varnish-4.0.5 revision 07eff4c29

Type ‘help‘ for command list.
Type ‘quit‘ to close CLI session.

varnish> vcl.list  #配置文件版本
200
available       0 boot
active          0 reload_2018-06-06T09:17:17

#切换vcl配置  每次配置文件加载都会保存下来，在这里可以切换
vcl.use boot
200
VCL ‘boot‘ now active
vcl.use reload_2018-06-06T09:17:17
200
VCL ‘reload_2018-06-06T09:17:17‘ now active

#查看默认配置选项
vcl.show -v reload_2018-06-06T09:17:17

#线程数量
param.show thread_pools
200
thread_pools
        Value is: 2 [pools] (default)
        Default is: 2
        Minimum is: 1

param.show thread_pool_max   #min
200
thread_pool_max
        Value is: 5000 [threads] (default)
        Default is: 5000
        Minimum is: 100

#设置参数
varnish> param.set thread_pools 4
200

#存储查看
varnish> storage.list
200
Storage devices:
    storage.Transient = malloc
    storage.s0 = file

#后端查看
varnish> backend.list
200
Backend name                   Refs   Admin      Probe
default(127.0.0.1,,8080)       1      probe      Healthy (no probe)
default(192.168.1.202,,80)     1      probe      Healthy (no probe)

#--------------------------------------------------------------------------
#非交互式
[[email protected] varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 status
Child in state running

规则编写

默认规则查看,很重要，因为varnish先读取客户端配置，在读取默认规则

[[email protected] varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 vcl.list
available       0 boot
available       0 reload_2018-06-06T09:17:17
available       0 reload_2018-06-06T10:43:58
available       0 reload_2018-06-06T11:08:20
available       0 reload_2018-06-06T12:52:47
available       0 reload_2018-06-06T13:04:19
active          0 reload_2018-06-06T13:06:08

[[email protected] varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 vcl.show -v reload_2018-06-06T13:06:08

1、不检查缓存

sub vcl_recv {
        if (req.url ~ "(?i)^/(login|admin)") {
                return(pass);
        }
}

2、合成错误码

sub vcl_recv {
    if (req.http.User-Agent ~ "(?i)curl"){
        return(synth(403));
    }

}

marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.200/admin.php
<!DOCTYPE html>
<html>
  <head>
    <title>403 Forbidden</title>
  </head>
  <body>
    <h1>Error 403 Forbidden</h1>
    <p>Forbidden</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 196684</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

3、默认vcl_recv策略

sub vcl_recv {

       if (req.method != "GET" &&
       req.method != "HEAD" &&
       req.method != "PUT" &&
       req.method != "POST" &&
       req.method != "TRACE" &&
       req.method != "OPTIONS" &&
       req.method != "DELETE") {
                        /* Non-RFC2616 or CONNECT which is weird. */
                        return (pipe);
       }

        if (req.method != "GET" && req.method != "HEAD") {
            /* We only deal with GET and HEAD by default */
            return (pass);
        }
        if (req.http.Authorization || req.http.Cookie) {
            /* Not cacheable by default */
            return (pass);
        }
            return (hash);

}

4、指定资源去除cookie做缓存

sub vcl_recv {
    if (req.url ~ "(?i)\.(jpg|jpge)$" ){
                unset req.http.Cookie;
                return(hash);
    }
}
sub vcl_backend_response {
    if (beresp.http.cache-control !~ "s-maxage") {
        if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif|css|js)$") {
            unset beresp.http.Set-Cookie;
            set beresp.ttl = 3600s;
        }
    }
}  

5、客户端ip

 vcl_recv： 如果多级代理需要加 ，默认是有X-Forwarded-For
 if (req.http.X-Forwarded-For){
              set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip;
        }else {
            set req.http.X-Forwarded-For = client.ip;
        }

6、缓存修剪(实用)

#白名单设置  掩码写在外面
acl purgers{
#       "192.168.1.104";
        "127.0.0.1"/8;
}
sub vcl_recv {
    if(req.method == "PURGE"){
        if (!client.ip ~ purgers){
                return(synth(403));
        }
        return(purge);
    }
}

#默认规则里面的行为    
sub vcl_purge {
    return (synth(200, "Purged"));
}    

以上三次请求，第一次删除缓存，第二次没有命中（添加缓存），第三次命中缓存

在交互模式下(用于临时清理)   这种方式很有用
ban req.url ~ ^/javascript   #清空所有/javascript 开头的
ban req.url ~ ^/            #清空所有

#配置文件清空
if (req.method == "BAN") {
    ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
    # Throw a synthetic page so the request won‘t go to the backend.
    return(synth(200, "Ban added"));
}

7、绑定多个后台

import directors;
#默认是backend default
backend websrv1 {
    .host = "192.168.1.201";
    .port = "80";
}

backend websrv2 {
    .host = "192.168.1.202";
    .port = "80";
}
sub vcl_init {
    #轮询
    new websrvs = directors.round_robin();
    websrvs.add_backend(websrv1);
    websrvs.add_backend(websrv2);

    #权重
    new websrvs3 = directors.random();
    websrvs3.add_backend(websrv1,1);
    websrvs3.add_backend(websrv2,2);

}

#可以区分出来 动态页面跟静态页面
sub vcl_recv {
       if (req.url ~ "(?i)\.php$") {
                set req.backend_hint = websrv1;
        } else {
                set req.backend_hint = websrvs.backend();
        }
}    

#---------------------------------------------------------
#会话绑定
sub vcl_init {
    new h = directors.hash();
    h.add_backend(one, 1);   // backend ‘one‘ with weight ‘1‘
    h.add_backend(two, 1);   // backend ‘two‘ with weight ‘1‘
}

sub vcl_recv {
    // pick a backend based on the cookie header of the client
    set req.backend_hint = h.backend(req.http.cookie);
}

8、健康状态监测

probe www_probe {
    .url = "/index.html";
    .timeout = 1s;
    .interval = 4s;
    .window = 5;   #探测5次 成功3次算在线
    .threshold = 3;
    .expected_response：期望的响应码，默认为200；
}

backend websrv1 {
    .host = "192.168.1.201";
    .port = "80";
    .probe = www_probe;
}

backend websrv2 {
    .host = "192.168.1.202";
    .port = "80";
    .probe = www_probe;
}

[[email protected] varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 backend.list
Backend name                   Refs   Admin      Probe
default(192.168.1.201,,80)     9      probe      Healthy (no probe)
websrv1(192.168.1.201,,80)     5      probe      Healthy 5/5
websrv2(192.168.1.202,,80)     5      probe      Healthy 5/5

在交互模式下手动设置下线
backend.set_health websrv Sick
backend.set_health websrv Healthy

服务进程配置

DAEMON_OPTS="-p thread_pool=3  thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"

thread_pool 2 线程池 小于等于cpu
thread_pool_min 100 每个线程池最小几个线程 默认100
thread_pool_max 5000 最大几个线程  默认5000
thread_pool_timeout 300.000 空闲线程的最长时常  超过thread_pool_min个数后 空闲的超过这个时间就会被杀死
thread_queue_limit 20  线程池队列长度默认20  可以增加
thread_pool_add_delay 0.000 [seconds]  增加不延迟
thread_pool_destroy_delay  1 second 杀死延迟一秒
thread_pool_fail_delay 0.200 [seconds] 创建失败时候，间隔多少时间在重新创建

client：
send_timeout 600.000   发送响应包给客户端
timeout_idle 5     保持连接超时空闲时常  可调增加
timeout_req  2      接收客户端请求报文首部  超过2秒放弃
cli_timeout    admin 管理工具请求超时时常  

backend BE_NAME {
    ...
    .connect_timeout = 3.5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
}

log

1、varnishstat - Varnish Cache statistics

    -1
    -1 -f FILED_NAME 
    -l：可用于-f选项指定的字段名称列表；

    MAIN.cache_hit 
    MAIN.cache_miss

    # varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss
    # varnishstat -l -f MAIN -f MEMPOOL

[[email protected] varnish]# varnishstat -1 #一次显示所有

[[email protected] varnish]# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss
MAIN.cache_hit              63         0.00 Cache hits
MAIN.cache_miss             18         0.00 Cache misses

2、varnishtop - Varnish log entry ranking

    -1     Instead of a continously updated display, print the statistics once and exit.
    -i taglist，可以同时使用多个-i选项，也可以一个选项跟上多个标签；
    -I <[taglist:]regex>
    -x taglist：排除列表
    -X  <[taglist:]regex>

[[email protected] varnish]# varnishtop -i RespStatus  #压测 状态码

3、日志文件，一般不开启，一级代理采用

[[email protected] varnish]# systemctl start varnishncsa
[[email protected] varnish]# tail /var/log/varnish/varnishncsa.log
192.168.1.104 - - [06/Jun/2018:20:54:42 +0800] "GET http://192.168.1.200/robots.txt HTTP/1.1" 404 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"