通过 daemonset 运行的 Fluentd pod 因谷歌容器引擎上的警告而终止

Posted

技术标签:

【中文标题】通过 daemonset 运行的 Fluentd pod 因谷歌容器引擎上的警告而终止【英文标题】：Fluentd pods running via daemonset getting terminated with warnings on google container engine 【发布时间】：2019-02-13 14:28:35 【问题描述】：

我正在尝试将 fluentd 作为 Kubernetes 集群 (GKE) 上的守护程序集运行。配置成功解析，然后插件收到关闭信号，并带有少量警告消息。没有错误消息。我尝试增加详细级别，以下是 pod 的输出：

fluentd-7przp fluentd 2018-09-08 11:02:46 +0000 [info]: #0 fluent/log.rb:322:info: starting fluentd worker pid=9 ppid=1 worker=0
fluentd-7przp fluentd 2018-09-08 11:02:46 +0000 [info]: #0 fluent/log.rb:322:info: fluentd worker is now running worker=0
fluentd-sr764 fluentd 2018-09-08 11:02:50 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:45.151774166 +0000 record="worker"=>0, "message"=>"fluentd worker is now running worker=0"
fluentd-dhwnx fluentd 2018-09-08 11:02:51 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:46.029522363 +0000 record="worker"=>0, "message"=>"fluentd worker is now running worker=0"
fluentd-7przp fluentd 2018-09-08 11:02:51 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.info" time=2018-09-08 11:02:46.538377182 +0000 record="worker"=>0, "message"=>"fluentd worker is now running worker=0"
fluentd-sr764 fluentd 2018-09-08 11:02:55 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.warn" time=2018-09-08 11:02:50.153922217 +0000 record="error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.info", "time"=>2018-09-08 11:02:45.151774166 +0000, "record"=>"worker"=>0, "message"=>"fluentd worker is now running worker=0", "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.info\" time=2018-09-08 11:02:45.151774166 +0000 record=\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\""
fluentd-sr764 fluentd 2018-09-08 11:03:10 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.warn" time=2018-09-08 11:03:05.168427649 +0000 record="error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:03:00.165843014 +0000, "record"=>"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:02:55.156840516 +0000, "record"=>"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.warn", "time"=>2018-09-08 11:02:50.153922217 +0000, "record"=>"error"=>"#<NoMethodError: undefined method `[]' for nil:NilClass>", "location"=>"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'", "tag"=>"fluent.info", "time"=>2018-09-08 11:02:45.151774166 +0000, "record"=>"worker"=>0, "message"=>"fluentd worker is now running worker=0", "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.info\" time=2018-09-08 11:02:45.151774166 +0000 record=\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\"", "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:02:50.153922217 +0000 record=\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\", \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record=\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"\"", "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:02:55.156840516 +0000 record=\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:50.153922217 +0000, \"record\"=>\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\", \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record=\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"\", \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:50.153922217 +0000 record=\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\", \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record=\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"\\\"\"", "message"=>"dump an error event: error_class=NoMethodError error=\"undefined method `[]' for nil:NilClass\" location=\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\" tag=\"fluent.warn\" time=2018-09-08 11:03:00.165843014 +0000 record=\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:55.156840516 +0000, \"record\"=>\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.warn\", \"time\"=>2018-09-08 11:02:50.153922217 +0000, \"record\"=>\"error\"=>\"#<NoMethodError: undefined method `[]' for nil:NilClass>\", \"location\"=>\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\", \"tag\"=>\"fluent.info\", \"time\"=>2018-09-08 11:02:45.151774166 +0000, \"record\"=>\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\", \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.info\\\" time=2018-09-08 11:02:45.151774166 +0000 record=\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\"\", \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:50.153922217 +0000 record=\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\", \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record=\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"\\\"\", \"message\"=>\"dump an error event: error_class=NoMethodError error=\\\"undefined method `[]' for nil:NilClass\\\" location=\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\" tag=\\\"fluent.warn\\\" time=2018-09-08 11:02:55.156840516 +0000 record=\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.warn\\\", \\\"time\\\"=>2018-09-08 11:02:50.153922217 +0000, \\\"record\\\"=>\\\"error\\\"=>\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\", \\\"location\\\"=>\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\", \\\"tag\\\"=>\\\"fluent.info\\\", \\\"time\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\"record\\\"=>\\\"worker\\\"=>0, \\\"message\\\"=>\\\"fluentd worker is now running worker=0\\\", \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.info\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record=\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\"\\\", \\\"message\\\"=>\\\"dump an error event: error_class=NoMethodError error=\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\" location=\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\" tag=\\\\\\\"fluent.warn\\\\\\\" time=2018-09-08 11:02:50.153922217 +0000 record=\\\\\\\"error\\\\\\\"=>\\\\\\\"#<NoMethodError: undefined method `[]' for nil:NilClass>\\\\\\\", \\\\\\\"location\\\\\\\"=>\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\", \\\\\\\"tag\\\\\\\"=>\\\\\\\"fluent.info\\\\\\\", \\\\\\\"time\\\\\\\"=>2018-09-08 11:02:45.151774166 +0000, \\\\\\\"record\\\\\\\"=>\\\\\\\"worker\\\\\\\"=>0, \\\\\\\"message\\\\\\\"=>\\\\\\\"fluentd worker is now running worker=0\\\\\\\", \\\\\\\"message\\\\\\\"=>\\\\\\\"dump an error event: error_class=NoMethodError error=\\\\\\\\\\\\\\\"undefined method `[]' for nil:NilClass\\\\\\\\\\\\\\\" location=\\\\\\\\\\\\\\\"/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'\\\\\\\\\\\\\\\" tag=\\\\\\\\\\\\\\\"fluent.info\\\\\\\\\\\\\\\" time=2018-09-08 11:02:45.151774166 +0000 record=\\\\\\\\\\\\\\\"worker\\\\\\\\\\\\\\\"=>0, \\\\\\\\\\\\\\\"message\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\"fluentd worker is now running worker=0\\\\\\\\\\\\\\\"\\\\\\\"\\\"\""
fluentd-7przp fluentd 2018-09-08 11:03:24 +0000 [debug]: #0 fluent/log.rb:302:debug: preparing shutdown output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:24 +0000 [info]: #0 fluent/log.rb:322:info: shutting down output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-dhwnx fluentd 2018-09-08 11:03:25 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-elasticsearch-2.10.1/lib/fluent/plugin/out_elasticsearch_dynamic.rb:268:in `eval'" tag="fluent.debug" time=2018-09-08 11:03:24.151685730 +0000 record="message"=>"fluentd main process get SIGTERM"
fluentd-sr764 fluentd-dhwnx fluentd-7przp fluentd fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd-dhwnxfluentd-sr764  fluentd 2018-09-08 11:03:25 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0
fluentd 2018-09-08 11:03:25 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0
2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="kubelet_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="apiserver_out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:elasticsearch_dynamic plugin_id="out_es"
fluentd-7przp fluentd 2018-09-08 11:03:25 +0000 [debug]: #0 fluent/log.rb:302:debug: calling terminate on filter plugin type=:parser plugin_id="myapp_filter"
fluentd-7przp fluentd 2018-09-08 11:03:26 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0

fluentd.conf：

@include systemd.conf
@include kubernetes.conf

# Start of fluent.conf
<filter kubernetes.var.log.containers.myapp-provider**.log>
  @type parser
  @id myapp_filter
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type multiline
    format_firstline /^[A-Z]/
    format1 /^(?<level>[^ ]*)[ \t]+\[(?<time>[^\]]*)\] \[(?<thread>[^\]]*)\] \[(?<request>[^\]]*)\] (?<class>[^ ]*): (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S,%L %z
  </parse>
</filter>

<match kubelet>
   @type elasticsearch_dynamic
   @id kubelet_out_es
   log_level info
   include_tag_key true
   host "#ENV['FLUENT_ELASTICSEARCH_HOST']"
   port "#ENV['FLUENT_ELASTICSEARCH_PORT']"
   scheme "#ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'"
   ssl_verify "#ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'"
   #user "#ENV['FLUENT_ELASTICSEARCH_USER']"
   #password "#ENV['FLUENT_ELASTICSEARCH_PASSWORD']"
   reload_connections "#ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'"
   index_name fluentd-$tag_parts[0]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)
   include_timestamp true
   #logstash_prefix "#ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'"
   #logstash_format true
   <buffer>
     flush_thread_count 8
     flush_interval 5s
     chunk_limit_size 2M
     queue_limit_length 32
     retry_max_interval 30
     retry_forever true
   </buffer>
</match>

<match kube-apiserver>
   @type elasticsearch_dynamic
   @id apiserver_out_es
   log_level info
   include_tag_key true
   host "#ENV['FLUENT_ELASTICSEARCH_HOST']"
   port "#ENV['FLUENT_ELASTICSEARCH_PORT']"
   scheme "#ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'"
   ssl_verify "#ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'"
   #user "#ENV['FLUENT_ELASTICSEARCH_USER']"
   #password "#ENV['FLUENT_ELASTICSEARCH_PASSWORD']"
   reload_connections "#ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'"
   index_name fluentd-$tag_parts[0]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)
   include_timestamp true
   #logstash_prefix "#ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'"
   #logstash_format true
   <buffer>
     flush_thread_count 8
     flush_interval 5s
     chunk_limit_size 2M
     queue_limit_length 32
     retry_max_interval 30
     retry_forever true
   </buffer>
</match>

<match **>
   @type elasticsearch_dynamic
   @id out_es
   log_level info
   include_tag_key true
   host "#ENV['FLUENT_ELASTICSEARCH_HOST']"
   port "#ENV['FLUENT_ELASTICSEARCH_PORT']"
   scheme "#ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'"
   ssl_verify "#ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'"
   #user "#ENV['FLUENT_ELASTICSEARCH_USER']"
   #password "#ENV['FLUENT_ELASTICSEARCH_PASSWORD']"
   reload_connections "#ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'"
   index_name fluentd-$record['kubernetes']['container_name']-$Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)
   include_timestamp true
   #logstash_prefix "#ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'"
   #logstash_format true
   <buffer>
     flush_thread_count 8
     flush_interval 5s
     chunk_limit_size 2M
     queue_limit_length 32
     retry_max_interval 30
     retry_forever true
   </buffer>
</match>

谁能帮忙调试一下。提前致谢！

【问题讨论】：

您检查过您的 pod 统计信息吗？转到 Kubernetes Engine 控制台并单击 Workloads，然后查看您的 pod 发生了什么。我假设如果您的 pod 崩溃而没有任何错误，它们可能内存不足，并且正在被杀死。 【参考方案1】：

这可能是此处提到的 kubernetes 错误 - https://github.com/kubernetes/kubernetes/issues/51376

虽然此问题目前已关闭，但我建议您尝试在 default 命名空间中运行相同的 daemonset。 即使我尝试在 kube-system 命名空间中运行它，但它失败并在 default 命名空间中成功运行。

【讨论】：

【参考方案2】：

我将从这个命令开始，以检查频繁重启的 pod：

kubectl get pods --all-namespaces

然后运行以下命令以收集有关容器的最后已知状态和关联的exit code 的更多信息。 Docker 退出代码基于the chroot standard。

kubectl describe pods [podname]

执行“kubectl describe”应该可以为您提供更多关于您的 pod 为何被终止的信息。

【讨论】：

以上是关于通过 daemonset 运行的 Fluentd pod 因谷歌容器引擎上的警告而终止的主要内容，如果未能解决你的问题，请参考以下文章

从 DaemonSet 中的每个 Pod 中抓取指标

K8S 日志收集：在K8S创建fluentd的ds

papertrail 的 fluentd 守护程序集容器无法在 kubernetes 集群中启动

Kubernetes 事件日志到 elasticsearch

使用Fluentd发送告警邮件

深入玩转K8S之最懂实际应用场景的调度神器DaemonSet