SYSLOG-NG：将相同的日志发送到弹性搜索中的两个不同索引

Posted 2023-02-22

【中文标题】SYSLOG-NG：将相同的日志发送到弹性搜索中的两个不同索引

【英文标题】：SYSLOG-NG: Sending same log to two different index in elasticsearch

【发布时间】：2021-02-12 13:12:41

【问题描述】：

我正在尝试将相同的日志流发送到两个不同的弹性搜索索引，因为每个索引具有不同角色的用户。

我也使用文件作为目标。这是一个示例：

2021-02-12T14:00:00+01:00 192.168.89.222 <30>1 2021-02-12T14:00:01+01:00 sonda filebeat 474 - - 2021-02-12T14:00:01.364+0100    DEBUG   [input] input/input.go:152      Run input
2021-02-12T14:00:00+01:00 192.168.89.222 <30>1 2021-02-12T14:00:01+01:00 sonda filebeat 474 - - 2021-02-12T14:00:01.364+0100    DEBUG   [input] log/input.go:191        Start next scan
2021-02-12T14:00:00+01:00 192.168.89.222 <30>1 2021-02-12T14:00:01+01:00 sensor filebeat 474 - - 2021-02-12T14:00:01.364+0100    DEBUG   [input] log/input.go:421        Check file for harvesting: /opt/zeek/logs/current/weird.log

当我只为 elasticsearch-http 使用一个目的地时，配置了两个中的任何一个，一切正常，但是当使用 两个目的地时，syslog-ng 无法启动和 systemcl 抱怨。

这是我的/etc/syslog-ng/syslog-ng.conf 文件：

@version: 3.27
@include "scl.conf"
options  chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
      dns_cache(no); owner("root"); group("adm"); perm(0640);
      stats_freq(0); bad_hostname("^gconfd$");
;
source s_net  
    udp( 
        ip(0.0.0.0) 
        port(514)
        flags(no-parse)
    );
;
log 
    source(s_net);
    destination(d_es);
    destination(d_es_other_index);   ######## comment this to avoid the error
    destination(d_file);
;

template t_demo_filetemplate 
    template("$ISODATE $HOST $MESSAGE\n");
;

destination d_file 
    file("/tmp/test.log" template(t_demo_filetemplate));
;

destination d_es
    elasticsearch-http(
        index("syslog-ng-$YEAR-$MONTH-$DAY")
            url("https://192.168.89.44:9200/_bulk" "https://192.168.89.144:9200/_bulk")
        type("")
            user("elastic")
            password("password")
            batch_lines(128)
            batch_timeout(10000)
            timeout(100)
            template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
        time-zone("UTC")
        tls(
            ca-file("/root/elastic_certs/elastic-ca.crt")       
            cert-file("/root/elastic_certs/elastic.crt")
            key-file("/root/elastic_certs/elastic.key")
            peer-verify(no)
        )
    );
;

destination d_es_other_index
    elasticsearch-http(
        index("otherindex-$YEAR-$MONTH-$DAY")
            url("https://192.168.89.44:9200/_bulk" "https://192.168.89.144:9200/_bulk")
        type("")
            user("elastic")
            password("password")
            batch_lines(128)
            batch_timeout(10000)
            timeout(100)
            template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
        time-zone("UTC")
        tls(
            ca-file("/root/elastic_certs/elastic-ca.crt")
            cert-file("/root/elastic_certs/elastic.crt")
            key-file("/root/elastic_certs/elastic.key")
            peer-verify(no)
        )
    );
;

错误我在使用两个 elasticsearch 目标时遇到（journalctl -xe 似乎没有显示相关信息）：

# systemctl restart syslog-ng.service
Job for syslog-ng.service failed because the control process exited with error code.
See "systemctl status syslog-ng.service" and "journalctl -xe" for details.

还有我的 syslog-ng 信息：

$ syslog-ng --version
syslog-ng 3 (3.27.1)
Config version: 3.22
Installer-Version: 3.27.1
Revision: 3.27.1-3build1
Compile-Date: Jul 30 2020 17:56:17
Module-Directory: /usr/lib/syslog-ng/3.27
Module-Path: /usr/lib/syslog-ng/3.27
Include-Path: /usr/share/syslog-ng/include
Available-Modules: syslogformat,afsql,linux-kmsg-format,stardate,affile,dbparser,geoip2-plugin,afprog,kafka,graphite,riemann,tfgetent,json-plugin,cef,hook-commands,basicfuncs,disk-buffer,confgen,timestamp,http,afamqp,mod-python,tags-parser,pseudofile,system-source,afsocket,afsnmp,csvparser,afstomp,appmodel,cryptofuncs,examples,afmongodb,add-contextual-data,afsmtp,afuser,xml,map-value-pairs,kvformat,redis,secure-logging,sdjournal,pacctformat
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on

有没有办法同时做这两个elasticsearch索引？

【参考方案1】：

您可以按照 systemctl 的建议在日志日志中查看确切的错误消息：

See "systemctl status syslog-ng.service" and "journalctl -xe" for details.

或者，您可以在前台启动 syslog-ng：

$ syslog-ng -F --stderr

---

由于匹配的elasticsearch-http() URL，您可能存在持久名称冲突。请尝试添加具有 2 个唯一名称的 persist-name() 选项，例如：

destination d_es 
    elasticsearch-http(
        index("syslog-ng-$YEAR-$MONTH-$DAY")
        url("https://192.168.89.44:9200/_bulk" "https://192.168.89.144:9200/_bulk")
        # ...
        persist-name("d_es")
    );
;

destination d_es_other_index 
    elasticsearch-http(
        index("otherindex-$YEAR-$MONTH-$DAY")
        url("https://192.168.89.44:9200/_bulk" "https://192.168.89.144:9200/_bulk")
        # ...
        persist-name("d_es_other_index")
    );
;