fluentd 收集 k8s 到 elasticsearch

Posted 2020-11-25 wh-blog

1. 部署elasticsearch + kibana

apiVersion: v1
kind: PersistentVolume
metadata:
  name: es-data-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
   - ReadWriteOnce
  storageClassName: nfs
  nfs:
    server: 192.168.0.250
    path: /var/nfs/es-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: es-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "nfs"
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: es-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: single-es
  template:
    metadata:
      labels:
        app: single-es
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: single-es
        image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
        ports:
          - containerPort: 9200
            name: http
          - containerPort: 9300
            name: transport
        env:
          - name: discovery.type
            value: single-node
        volumeMounts:
          - mountPath: /usr/share/elasticsearch/data
            name: es-data
      volumes:
        - name: es-data
          persistentVolumeClaim:
            claimName: es-data-pvc 
---
apiVersion: v1
kind: Service
metadata:
  name: single-es-svc
spec:
  selector:
    app: single-es
  type: NodePort
  ports:
  - name: http
    port: 9200
    targetPort: 9200
    nodePort: 30092
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.5.2
        ports:
        - name: http
          containerPort: 5601
        env:
        - name: ELASTICSEARCH_HOSTS
          value: http://single-es-svc:9200
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-svc
spec:
  selector:
    app: kibana
  type: NodePort
  ports:
  - name: http
    port: 5601
    targetPort: 5601
    nodePort: 30561

2. 部署fluentd

参考网站:https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
修改 fluentd-es-configmap.yaml 中的

output.conf: |-
    <match **>
      @id elasticsearch
      @type elasticsearch
      @log_level info
      type_name _doc
      include_tag_key true
      host elasticsearch-logging
      port 9200
      logstash_format true
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 2M
        total_limit_size 500M
        overflow_action block
      </buffer>
    </match>

match 中的es的host改为部署的es的服务名 elasticsearch-logging -> single-es-svc.default // {服务名.命名空间}，如果 es 与 fluentd 在一个命名空间下可省略。