logstash multiline处理csv单元格多行数据的double quotes问题（exception=＞#＜CSV::MalformedCSVError: Unclosed quoted）

Posted 2022-12-28 我怀念的Wu Zhiwei:)

在csv文件里的格式

“field1”,“field2”,“field3”,"line1
line2
line3
",“field5”
可以看到上面field4里面有被换行成几行来显示了。
这是我们用logstash去读取：

input 
  file 
      path => [ "file.csv" ]
      start_position => "beginning"
      sincedb_path => "/dev/null"
  



filter 
  csv 
      separator => ","
      skip_header => "true"
      columns => ["field1","field2","field3","field14,"field5"]
  





output 
    stdout 
    elasticsearch
        hosts => ["https://ip:9200"]
        ssl => true
        cacert => "/etc/logstash/certs/rootCA.crt"
        api_key => "api_key "
        index => "alert-%+YYYY-MM-dd"
    

这时候会遇到下面的错误信息

exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1

解决办法：
在file代码块里面加入下面代码：

codec => multiline 
        pattern => "^\\"XXX"
        negate => "true"
        what => "previous"
      

^是从行的开头开始，
"XXX是匹配每一行开头的字符串
这样就可以解析了，我实测过，可以解析成功。