[k8s]kube-dns/dashboard排错历险记(含sa加载用法/集群搭建)

Posted 毛台

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了[k8s]kube-dns/dashboard排错历险记(含sa加载用法/集群搭建)相关的知识,希望对你有一定的参考价值。

kube-dns原理

参考:

我是这样部署集群的

http://www.cnblogs.com/iiiiher/p/7888934.html

安装kube-dns

官网下载yaml:

wget https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/kube-dns.yaml.sed
mv kube-dns.yaml.sed kube-dns.yaml
sed -i \'s#gcr.io/google_containers#lanny#g\' kube-dns.yaml
sed -i \'s#$DNS_DOMAIN#cluster.local#g\'  kube-dns.yaml
sed -i \'s#$DNS_SERVER_IP#10.254.0.2#g\'  kube-dns.yaml

3个image
lanny/k8s-dns-kube-dns-amd64:1.14.7
lanny/k8s-dns-dnsmasq-nanny-amd64:1.14.7
lanny/k8s-dns-sidecar-amd64:1.14.7

kubectl create -f  kube-dns.yaml

排错1:kube-dns3个容器都起来了,只能查询nslookup kubernetes 和 nslookup kube-dns.自己新建的svc无法查

开始以为是api启动问题,因为我没有加载任何准入控制器,想着把sa加载进去

无奈,sa搞不好

排错2: 为pod加载sa准入器

kube-apiserver \\
    --service-cluster-ip-range=10.254.0.0/16 \\
    --etcd-servers=http://127.0.0.1:2379 \\
    --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,DefaultStorageClass,ResourceQuota,ServiceAccount \\
    --service-account-key-file=/root/ssl/ca.key \\
    --insecure-bind-address=0.0.0.0 \\
    --v=2
  • 3.api指定key(这里controller一定要加载key,否则单独给api加载key,pod是无法生成token的,切记切记,浪费了一天时间,擦)
kube-controller-manager \\
  --master=http://127.0.0.1:8080 \\
  --service-account-private-key-file=/root/ssl/ca.key \\
  --v=2

接着怀疑flannel host-gw模式问题,遂改给vxlan模式.问题依旧

排错2: pod默认以https来连api的(我发现kube-dns和dashboard都是),报token找不到.

默认有sa情况下 启动容器 /var/run/secrets/kubernetes.io/serviceaccount/token会自动生成的. 目前我们没启动sa.

[root@m1 dns]# kk
NAMESPACE     NAME                              READY     STATUS             RESTARTS   AGE       IP          NODE        LABELS
kube-system   kube-dns-2981639038-f41v9         2/3       CrashLoopBackOff   5          2m        10.2.50.2   n2.ma.com   k8s-app=kube-dns,pod-template-hash=2981639038
[root@m1 dns]# kubectl  logs -f kube-dns-2981639038-f41v9 -n kube-system -c kubedns
I1124 16:24:09.294678      86 dns.go:48] version: 1.14.3-4-gee838f6
F1124 16:24:09.294768      86 server.go:57] Failed to create a kubernetes client: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
rpc error: code = 2 desc = Error: No such container: d72e21f48dd0167dc184c1ddb79a0d88242fff03d0d16463f536f2803e2d2eb0[root@m1 dns]# 

可以看出启动过程需要token.pod以https的方式连apiserver的时候就需要这个token了.默认我启动api的时候是没有加载ServiceAccount组件的.

解决:

  • 方法1: 直接改deploy,kube-dns的args部分添加 pod查找api的地址.(dashboard也是这个原理)
 kubectl -n kube-system edit deployment kube-dns

--kube-master-url=http://192.168.x.x:8080
  • 方法2: 修改yaml args部分添加 --kube-master-url=http://192.168.x.x:8080

那么问题来了: 不同的镜像参数不一样,kube-master-url类似这种连api的参数从哪里找呢?
建议从k8s的github以往的release里yaml里找找.
因为gcr.io里的镜像我发现没dockerfile可以看,至于他们需要什么参数,不太透明

参考他的github可以看下:
https://github.com/denverdino/google-containers

灵感来源: http://jeromeliu.win/2017/04/24/Kubernetes-搭建kube-dns/

curl -k -s -X GET https://gcr.io/v2/google_containers/hyperkube-amd64/tags/list | jq -r \'.tags[]\'
docker search gcr.io/google-containers/hyperkube

提示:这里发现个处理json的小工具,yum install -y jq

贴上kube-dashboard的url

https://github.com/kubernetes/dashboard/blob/master/src/deploy/recommended/kubernetes-dashboard.yaml
我把他精简了下,因为有些东西对于我这个简单的集群没什么用,我还没做多余的认证

官方git下载的,我删改了一些没用的,因为我不需要用证书认证,遵从最小原则,越简单越好.

[root@m1 yaml]# cat kubernetes-dashboard.yaml 
kind: Deployment 
apiVersion: extensions/v1beta1 
metadata: 
  labels: 
    app: kubernetes-dashboard 
  name: kubernetes-dashboard 
  namespace: kube-system 
spec: 
  replicas: 1 
  revisionHistoryLimit: 10 
  selector: 
    matchLabels: 
      app: kubernetes-dashboard 
  template: 
    metadata: 
      labels: 
        app: kubernetes-dashboard 
      # Comment the following annotation if Dashboard must not be deployed on master 
      annotations: 
        scheduler.alpha.kubernetes.io/tolerations: | 
          [ 
            { 
              "key": "dedicated", 
              "operator": "Equal", 
              "value": "master", 
              "effect": "NoSchedule" 
            } 
          ] 
    spec: 
      containers: 
      - name: kubernetes-dashboard 
        image: k8scn/kubernetes-dashboard-amd64:v1.7.1 
        imagePullPolicy: IfNotPresent
        ports: 
        - containerPort: 9090 
          protocol: TCP 
        args: 
          # Uncomment the following line to manually specify Kubernetes API server Host 
          # If not specified, Dashboard will attempt to auto discover the API server and connect 
          # to it. Uncomment only if the default does not work. 
          - --apiserver-host=http://192.168.x.x:8080
        livenessProbe: 
          httpGet: 
            path: / 
            port: 9090
          initialDelaySeconds: 30 
          timeoutSeconds: 30 
--- 
kind: Service 
apiVersion: v1 
metadata: 
  labels: 
    app: kubernetes-dashboard 
  name: kubernetes-dashboard 
  namespace: kube-system 
spec: 
  type: NodePort 
  ports: 
  - port: 80 
    targetPort: 9090
    nodePort: 30090
  selector: 
    app: kubernetes-dashboard

以上是关于[k8s]kube-dns/dashboard排错历险记(含sa加载用法/集群搭建)的主要内容,如果未能解决你的问题,请参考以下文章

k8s医生k8s安全机制之新版rbac详解及排错

K8S 故障排错新手段:kubectl debug 实战

经典k8s日常巡检及排错指南

搞定排错k8s集群日志管理

搞定排错k8s集群日志管理

K8s完整多节点部署(线网实战!含排错!)