juju debug hacks (by quqi99)

Posted 2021-08-23 quqi99

版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 (作者：张华 发表于：2021-07-22)

debug juju-controller

juju ssh -m controller 0
juju_engine_report | pastebinit -f yaml
eg:
instance-poller:
                inputs:
                - api-caller
                - environ-tracker
                - clock
                - valid-credential-flag
                - migration-fortress
                - migration-inactive-flag
                start-count: 19847
                state: starting

上面说instance-poller有问题. instance-poller用于juju-controller定期从maas-api轮循maas machine/instance的数据. 如下面可能和spaces相关.

juju reload-spaces --debug --verbose --show-log
ERROR could not reload spaces: unexpected: Get "http://10.91.42.24/MAAS/api/2.0/spaces/": EOF
16:16:21 DEBUG cmd supercommand.go:537 error stack:
unexpected: Get "http://10.91.42.24/MAAS/api/2.0/spaces/": EOF

检查一下maas端和juju端的spaces信息是否一致, 不一致可试试重启juju-controller试试.

maas admin spaces read
juju spaces

#juju spaces
ceph-access-space 5 10.91.246.0/23
ceph-replica-space 4 10.91.250.0/23
external-space 3 10.91.248.0/23
internal-space 2 10.91.244.0/23
oam-space 1 10.91.42.0/23
#cat maas-admin-space-read.txt | jq '.[] | {name:.name,id:.id,url:.resource_uri,subnet:.subnets[].cidr}' -c
{"name":"oam-space","id":1,"url":"/MAAS/api/2.0/spaces/1/","subnet":"10.91.42.0/23"}
{"name":"internal-space","id":2,"url":"/MAAS/api/2.0/spaces/2/","subnet":"10.91.244.0/23"}
{"name":"external-space","id":3,"url":"/MAAS/api/2.0/spaces/3/","subnet":"10.91.248.0/23"}
{"name":"ceph-replica-space","id":4,"url":"/MAAS/api/2.0/spaces/4/","subnet":"10.91.250.0/23"}
{"name":"ceph-access-space","id":5,"url":"/MAAS/api/2.0/spaces/5/","subnet":"10.91.246.0/23"}
{"name":"undefined","id":-1,"url":"/MAAS/api/2.0/spaces/undefined/","subnet":"192.168.122.0/24"}

# why undefine space is missing, see db
juju ssh -m controller 0
mongo --sslAllowInvalidHostnames --sslAllowInvalidCertificates localhost:37017/admin --ssl -u "$(ls /var/lib/juju/agents/)" -p "$(sudo grep statepassword /var/lib/juju/agents/$(ls /var/lib/juju/agents/)/agent.conf|awk '{print $2}')" --authenticationDatabase admin
use juju;
db.spaces.find().pretty()
mongodump --port 37017 --sslAllowInvalidCertificates --ssl --authenticationDatabase admin -u "$(sudo awk '/^tag:/ { print $2 }' /var/lib/juju/agents/machine-*/agent.conf)" -p "$(sudo awk '/^statepassword:/ { print $2 }' /var/lib/juju/agents/machine-*/agent.conf)" --db juju --gzip
2)tar -czf mongodump-`date '+%F_%T'| tr -s ':' '-'`.tgz dump

不管controller side是不是少一个undefine的space吧，至少：

• instance-poller正在访问/MAAS/api/2.0/machines出了EOF错
• reload-spaces也出了EOF错
  这两点都说明controller和maas的80/5240端口似乎不通．所以接下来可以在maas上安装maas-cli再试以排除网络问题．但是使用’maas admin spaces read’时能看到下列请求的日志，但reload-spaces时似乎看不到．

... regiond: [info] 10.91.42.24 GET /MAAS/api/2.0/spaces/ HTTP/1.1 --> 200 OK (referrer: -; agent: Python-httplib2/0.9.2 (gzip))

难道是连接被代理了吗（ juju model-config -m controller ），抓包试，10.91.43.2是其中一个controoler.

sudo tcpdump -i any src host 10.91.43.2 -w pcap-02.pcap
juju reload-spaces --debug --verbose --show-log

继续抓额外的包：

tcpdump -i any dst host 10.91.42.24 -w pcap-controller.pcap on machine 10.91.43.2
tcpdump -i any src host 10.91.43.2 -w pcap-maas.pcap on machine 10.91.42.24
juju reload-spaces
maas admin spaces read