2.ES----Logstash 安装与测试数据导入

Posted 2021-04-21 爱飞的菜鸟

1.下载Logstash 安装包注意版本对应,解压

2.下载movieton测试数据 ,解压得到 movies.csv 文件,复制到logstash的bin目录下

3.在config目录下 新建文件 logstash.conf文件

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  file {
  #这里指定需要导入的文件的路径 lunix 从/xx/xx
    path => "D:/logstash-7.5.0/logstash-7.5.0/bin/movies.csv"
    start_position => "beginning"
   # 监听文件读取信息记录的位置 win
   #sincedb_path => "H:/ElasticSearch/6.4.3/logstash-6.4.3/test/since_db.txt"
   #Linux
   #sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
   #es地址
     hosts => "http://localhost:9200"
     # 导入文件索引
     index => "movies"
     #数据id
     document_id => "%{id}"
   }
  stdout {}
}

启动 es ,kibana

以logstash.conf作为配置文件启动 logstash, 在logstash的bin目录下执行

logstash.bat　-f  ../config/logstash.conf

成功时,cmd控制台打印: 导入的记录信息

如果未执行成功,第二次导入时一定要先把 sincedb_path指定的文件删除,否则还是导入不了