nginx-lua流量采集及拦截

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了nginx-lua流量采集及拦截相关的知识,希望对你有一定的参考价值。

参考技术A 首先你的环境需要支持:

简单将流量采集分为3个模块:

这部分比较简单,基本信息都在ngx.var ngx.ctx 等中都可以获取到:

ngx.status #状态码
ngx.var.scheme #协议
ngx.var.request_method # 请求方法
ngx.var.request_uri #请求uri
ngx.var.host #请求域名host
ngx.var.remote_addr #请求方IP
ngx.var.hostname #主机hostname
ngx.var.time_iso8601 #时间
ngx.var.server_port #服务端端口
ngx.req.get_headers() #获取请求头
ngx.req.get_body_data() #获取请求体

此处需注意get_headers 和 get_body_data 均需转换成dict形式。

响应体获取不是直接ngx.resp.get_headers()之类可以获取到的,需做一些处理:

ngx.ctx.buffered 即为响应体内容
需配合 header_filter_by_lua_file 引用

ngx.ctx.resp_headers 即为响应头信息
需配合 body_filter_by_lua_file 引用

以下为对内容进行简单拦截。

待更新高级用法。

php类似nginx-lua的扩展,nginx-php中文开发文档

官方php有什么不同

  • 全局变量在每个请求中都不安全
  • 类的静态变量在每个请求中都不安全
  • 不要设计单例模式
  • 本机IO功能可以正常工作,但是会减慢Nginx的速度

运行条件

  • 仅支持 Linux
  • PHP-7.0.* ~ PHP-7.4.*
  • nginx-1.4.7 ~ nginx-1.17.8

安装

编译安装

 

 

$ wget ‘http://php.net/distributions/php-7.3.10.tar.gz‘
$ tar xf php-7.3.10.tar.gz
$ cd php-7.3.10

$ ./configure --prefix=/path/to/php --enable-embed
$ make && make install

$ git clone https://github.com/rryqszq4/ngx_php7.git

$ wget ‘http://nginx.org/download/nginx-1.12.2.tar.gz‘
$ tar -zxvf nginx-1.12.2.tar.gz
$ cd nginx-1.12.2

$ export PHP_CONFIG=/path/to/php/bin/php-config
$ export PHP_BIN=/path/to/php/bin
$ export PHP_INC=/path/to/php/include/php
$ export PHP_LIB=/path/to/php/lib

$ ./configure --user=www --group=www $             --prefix=/path/to/nginx $             --with-ld-opt="-Wl,-rpath,$PHP_LIB" $             --add-module=/path/to/ngx_php7/third_party/ngx_devel_kit $             --add-module=/path/to/ngx_php7
$ make && make install

CentOS / RedHat 7

 

yum -y install https://extras.getpagespeed.com/release-el7-latest.rpm
yum -y install http://rpms.remirepo.net/enterprise/remi-release-7.rpm yum-utils
yum-config-manager --enable remi-php73
yum install nginx-module-php7

编辑 nginx.conf 并在顶部加载所需的模块:

load_module modules/ndk_http_module.so;
load_module modules/ngx_http_php_module.so;

Docker

 

$ docker build -t nginx-php7 .
$ : "app.conf: Create nginx config"
$ docker run -p 80:80 -v $PWD/app.conf:/etc/nginx/conf.d/default.conf nginx-php7

概要

 

worker_processes  auto;

events {
    worker_connections  102400;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    keepalive_timeout  65;
    
    client_max_body_size 64k;   
    client_body_buffer_size 64k;

    php_ini_path /usr/local/php/etc/php.ini;

    server {
        listen       80;
        server_name  localhost;
        default_type ‘application/json; charset=UTF-8‘;
    
        location /php {
            content_by_php_block {
                echo "hello ngx_php7";
            }
        }

        location = /ngx_request {
            content_by_php_block {
                echo ngx_request_document_uri();
            }
        }

        # curl /ngx_get?a=1&b=2
        location = /ngx_get {
            content_by_php_block {
                echo "ngx_query_args()
";
                var_dump(ngx_query_args());
            }
        }

        # curl -d ‘a=1&b=2‘ /ngx_post
        location = /ngx_post {
            content_by_php_block {
                echo "ngx_post_args()
";
                var_dump(ngx_post_args());
            }
        }

        location = /ngx_sleep {
            content_by_php_block {
                echo "ngx_sleep start
";
                yield ngx_sleep(1);
                echo "ngx_sleep end
";
            }
        }

        location = /ngx_socket2 {
            default_type ‘application/json;charset=UTF-8‘;
            content_by_php_block {
                $fd = ngx_socket_create();

                yield ngx_socket_connect($fd, "hq.sinajs.cn", 80);

                $send_buf = "GET /list=s_sh000001 HTTP/1.0

                                            Host: hq.sinajs.cn
Connection: close

";
                yield ngx_socket_send($fd, $send_buf, strlen($send_buf));

                $recv_buf = "";
                yield ngx_socket_recv($fd, $recv_buf);
                var_dump($recv_buf);
                
                yield ngx_socket_close($fd);
            }
        }

        location = /ngx_var {
            set $a 1234567890;
            content_by_php_block {
                $a = ngx_var_get("a");
                var_dump($a);
            }
        }
        
        # set content-type of response headers
        location = /ngx_header {
            content_by_php_block {
                ngx_header_set("Content-Type", "text/html; charset=UTF-8");
            }
        }

        # run a php file
        location = /php {
            content_by_php_block {
                include "name_of_php_file.php";
            }
        }
        
        # run any php file in root
        location = / {
            content_by_php_block {
                include ngx_var_get("uri");
            }
        }

    }
}

测试

使用Test :: Nginx模块的perl进行测试,搜索和发现ngx_php7中的问题。

 

ngx_php7 test ...
nginx version: nginx/1.12.2
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) 
configure arguments: --prefix=/home/travis/build/rryqszq4/ngx_php7/build/nginx --with-ld-opt=-Wl,-rpath,/home/travis/build/rryqszq4/ngx_php7/build/php/lib --add-module=../../../ngx_php7/third_party/ngx_devel_kit --add-module=../../../ngx_php7
t/001-hello.t ..................... ok
t/002-ini.t ....................... ok
t/003-error.t ..................... ok
t/004-ngx_request.t ............... ok
t/005-ngx_log.t ................... ok
t/006-ngx_sleep.t ................. ok
t/007-ngx_socket.t ................ ok
t/008-ngx_exit.t .................. ok
t/009-ngx_query_args.t ............ ok
t/010-ngx_post_args.t ............. ok
t/011-ngx_constants.t ............. ok
t/012-function.t .................. ok
t/013-class.t ..................... ok
t/014-ngx_var.t ................... ok
t/015-ngx_header.t ................ 1/? WARNING: TEST 2: set content-length of response headers - unexpected extra bytes after last chunk in response: "Testing ngx_header!x{0a}"
t/015-ngx_header.t ................ ok
t/016-rewrite_by_php.t ............ ok
t/017-ngx_redirect.t .............. ok
t/018-ngx_mysql.t ................. ok
t/019-php_set.t ................... ok
t/020-ngx_cookie.t ................ ok
t/021-content_by_php_block.t ...... ok
t/022-init_worker_by_php_block.t .. ok
All tests successful.
Files=22, Tests=84, 14 wallclock secs ( 0.09 usr  0.02 sys +  2.19 cusr  0.43 csys =  2.73 CPU)
Result: PASS

以上是关于nginx-lua流量采集及拦截的主要内容,如果未能解决你的问题,请参考以下文章

php类似nginx-lua的扩展,nginx-php中文开发文档

[nginx]lua控制响应头

如何在 nginx 中引用 lua 设置的变量?

滴滴在流量链路检测架构设计及实践

道路交通实时流量监控预测系统

iOS端socket流量采集