记录一下工作中遇到的一例kibana无法启动的问题

Posted 2021-04-02 DevOps技术实验室

先说一下我这边的环境

kubernetes: 1.17.9
helm: 2.16.x
efk: 7.8.0 官方oss镜像

安装elasticsearch使用的helm官方模板：

https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml

安装kibana使用的deployment yaml文件

apiVersion: apps/v1kind: Deploymentmetadata: name: kibana namespace: log-system labels:    app: kibanaspec: replicas: 1 selector: matchLabels: app: kibana template: metadata: annotations: ## add-dep-hash-annotations dep.secret.hash/kibana-config: "e00d2a0ea4791bd589564b4792ad8461355748ed31ede88256abba4255c23e6e" labels: app: kibana spec: # affinity for best effort to assign pod across nodes # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - kibana topologyKey: "kubernetes.io/hostname" volumes: - name: kibana-config secret: secretName: kibana-config # containers: - name: kibana # port ports: - containerPort: 5601 name: kibana #关闭健康检查 readinessProbe: exec: command: - "echo" - "ok" initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: exec: command: - "echo" - "ok" initialDelaySeconds: 15 periodSeconds: 20        image: "kibana-oss:v7.8.0" imagePullPolicy: Always env: - name: "ELASTICSEARCH_HOSTS" value: "https://dev-elasticsearch.abc.com:443" volumeMounts: - mountPath: /usr/share/kibana/config/kibana.yml name: kibana-config subPath: kibana.yml
 # add-resources resources: # default-resources # overwritten limits limits: cpu: 1 memory: 2Gi # overwritten requests requests: cpu: 1 memory: 2Gi

YAML文件是没啥问题

应用kibana后pod不断重启，查看一下pod日志

{"type":"log","@timestamp":"2020-11-21T01:14:05Z","tags":["warning","plugins-discovery"],"pid":7,"message":"Expect plugin \"id\" in camelCase, but found: apm_oss"}{"type":"log","@timestamp":"2020-11-21T01:14:06Z","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"visTypeXy\" is disabled."}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins-system"],"pid":7,"message":"Setting up [42] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaLegacy,devTools,uiActions,statusPage,share,newsfeed,mapsLegacy,kibanaUtils,kibanaReact,inspector,embeddable,indexPatternManagement,esUiShared,discover,charts,bfetch,expressions,data,home,console,apm_oss,management,advancedSettings,telemetryManagementSection,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeMetric,visTypeTable,visTypeMarkdown,inputControlVis,savedObjects,navigation,visualize,dashboard,savedObjectsManagement]"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","savedobjects-service"],"pid":7,"message":"Starting saved objects migrations"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["fatal","root"],"pid":7,"message":"TypeError: Cannot read property 'properties' of undefined\n at assertIsSupportedIndex (/usr/share/kibana/src/core/server/saved_objects/migrations/core/elastic_index.js:315:32)\n at fetchInfo (/usr/share/kibana/src/core/server/saved_objects/migrations/core/elastic_index.js:70:10)\n at process._tickCallback (internal/process/next_tick.js:68:7)"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins-system"],"pid":7,"message":"Stopping all plugins."}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","savedObjectsManagement"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","dashboard"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visualize"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","inputControlVis"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeMarkdown"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTable"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeMetric"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTagcloud"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTimelion"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTimeseries"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeVega"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeVislib"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visualizations"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","advancedSettings"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","management"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","apm_oss"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","console"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","home"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","data"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","expressions"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","bfetch"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","discover"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","share"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","kibanaLegacy"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","telemetry"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","telemetryCollectionManager"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","usageCollection"],"pid":7,"message":"Stopping plugin"}
 FATAL TypeError: Cannot read property 'properties' of undefined

报错关键信息

FATAL TypeError: Cannot read property 'properties' of undefined

是不是一脸懵，看不出啥问题，就先google了，没任何结果，debug了半天，依然没有任何头绪，第二天一早继续，精神状态贼好

既然网上查不出啥，就慢慢梳理咯，首先构造一个post请求debug一下，看返回什么

curl -X POST "https://dev-elasticsearch.abc.com/_aliases" -H 'Content-Type: application/json' -d' { "actions" : [ { "add" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }' |jq .

我这里是返回一个网关的404页面的信息

果然是网关这边的原因，先看下网关的配置，这里只截取了location关键部分

 location / { # root html; # index 50x.html index.htm;
 proxy_redirect off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-nginx-Proxy true; proxy_set_header Host $http_host; proxy_intercept_errors on;

眼尖的可能已经发现了，是的,就是这个参数`proxy_intercept_errors on;`

先看下nginx官网对这个参数的解释：