配置apache日志，分别记录不同搜索引擎爬虫的访问记录

Posted 2020-06-17

编辑/usr/local/apache/conf/extra/http_vhost.conf，在添加的虚拟主机中配置日志如下：

SetEnvIfNoCase User-Agent Baiduspider baidu_robot   #百度访问日志

    SetEnvIfNoCase User-Agent Googlebot   google_robot    #谷歌访问日志

    SetEnvIfNoCase User-Agent 360Spider   360__robot

    SetEnvIfNoCase User-Agent Iaskspider  xinglang_robot

    SetEnvIfNoCase User-Agent Sogou       sogou_robot

    SetEnvIfNoCase User-Agent YodaoBot    wangyi_robot

    CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.site_%Y%m%d.log 86400" combined env=!image-request

    CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.google_%Y%m%d.log 86400" combined env=google_robot

    CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.baidu_%Y%m%d.log 86400" combined env=baidu_robot

    CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.360_%Y%m%d.log 86400" combined env=360_robot

    CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.xinglang_%Y%m%d.log 86400" combined env=xinglang_robot

    CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.sougou_%Y%m%d.log 86400" combined env=sougou_robot

    CustomLog "|/usr/local/apache2/bin/rotatelogs -l /usr/local/apache2/logs/cn.wangyi_%Y%m%d.log 86400" combined env=wangyi_robot

然后每天生成不同的日志来记录，实现不同的访问日志记录不同搜索引擎爬虫的访问记录。

本文出自 “11083647” 博客，请务必保留此出处http://11093647.blog.51cto.com/11083647/1745341