charm zaza functional test (by quqi99)

Posted 2021-10-23 quqi99

版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 (作者：张华 发表于：2021-07-08)

问题

这篇文档（https://zaza.readthedocs.io/en/latest/runningcharmtests.html）描述了如何测试zaza功能测试．
写了一段charm 代码　－　https://review.opendev.org/c/openstack/charm-octavia/+/787700
也需要写zaza功能测试代码　- https://paste.ubuntu.com/p/mcmtpZNpR9/
如何测试呢？

准备tox代码环境

需要将octavia charm patch与zaza-openstack-tests patch打到系统中．
最开始通过下列从源码编译的方式，

sudo snap install charm --classic
mkdir -p /home/ubuntu/charms
mkdir -p ~/charms/{layers,interfaces}
export JUJU_REPOSITORY=/home/ubuntu/charms
export CHARM_INTERFACES_DIR=$JUJU_REPOSITORY/interfaces
export CHARM_LAYERS_DIR=$JUJU_REPOSITORY/layers
export CHARM_BUILD_DIR=$JUJU_REPOSITORY/layers/builds
cd $CHARM_LAYERS_DIR
git clone https://github.com/openstack/charm-octavia.git octavia
cd octavia

#apply octavia charm patch
git fetch "https://review.opendev.org/openstack/charm-octavia" refs/changes/00/787700/14 && git checkout FETCH_HEAD

#apply zaza-openstack-tests patch
ubuntu@zhhuabj-bastion:~/charms/layers/octavia$ git diff
diff --git a/src/test-requirements.txt b/src/test-requirements.txt
index 520681e..ded9b10 100644
--- a/src/test-requirements.txt
+++ b/src/test-requirements.txt
@@ -12,4 +12,5 @@ keyring<21
 
 # Functional Test Requirements (let Zaza's dependencies solve all dependencies here!)
 git+https://github.com/openstack-charmers/zaza.git#egg=zaza
-git+https://github.com/openstack-charmers/zaza-openstack-tests.git#egg=zaza.openstack
+#git+https://github.com/openstack-charmers/zaza-openstack-tests.git#egg=zaza.openstack
+git+https://github.com/zhhuabj/zaza-openstack-tests@hm-port#egg=zaza.openstack

cd ..
charm build --debug octavia/src/

cd ${JUJU_REPOSITORY}/layers/builds/octavia
tox -e func-noop
source .tox/func-noop/bin/activate

export OS_VIP00=10.5.100.122
functest-run-suite -b focal-ussuri-ha --keep-model

但是它报下列莫名其妙的包依赖问题，

2021-07-07 07:44:17 WARNING unit.octavia/0.install logger.go:60 subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-octavia-0/.venv/bin/pip', 'install', '-U', '--force-reinstall', '--no-index', '--no-cache-dir', '-f', 'wheelhouse', 'tenacity==7.0.0', ...
2021-07-07 07:44:17 ERROR juju.worker.uniter.operation runhook.go:139 hook "install" (via explicit, bespoke hook script) failed: exit status 1

所以最终改成了下列方式：

mkdir ~/charms_pull
charm pull octavia
tox -e func-noop
source .tox/func-noop/bin/activate
# 然后再在octavia目录下打上octavia charm patch, 在.tox/func-noop/lib/python3.8/site-packages/目录下打上zaza-openstack-tests patch

搭建octavia环境

export OS_VIP00=10.5.100.122
functest-run-suite -b focal-ussuri-ha --keep-model

直接使用上面＂functest-run-suite -b focal-ussuri-ha＂一步搭建环境并完成测试会出现很多问题 (注: 后来加–keep-model后成功了）．

还是让我们分步来做吧．先搭建octavia环境（注：运行下列命令需要提升quota, 尤其是SG rule和memory的quota

functest-prepare -m testmodel
juju switch testmodel
export OS_VIP00=10.5.100.122
functest-deploy -m testmodel -b ./tests/bundles/focal-ussuri-ha.yaml #quota exceed

搭建成功后，马上会遇到一个ssl方面的问题，实际上之前遇到过（见：https://blog.csdn.net/quqi99/article/details/107182847），因为这里的certifi全是都过pip3来源码安装的，无法改成通过’apt install python3-certifi"来安装，所以只能通过下列"export OS_CACERT=/etc/ssl/certs/"来bypass问题

source ~/stsstack-bundles/openstack/novarc
export OS_PASSWORD=$(juju run --unit keystone/0 leader-get admin_passwd)
openstack --insecure project list
# https://blog.csdn.net/quqi99/article/details/107182847
export OS_CACERT=/etc/ssl/certs/
openstack project list

接着要解决问题＂Port security must be enabled on the VIP network＂

juju config neutron-api enable-ml2-port-security=true
openstack --insecure network set --enable-port-security $(openstack --insecure network show private -cid -fvalue)

最终的验证命令是：

source ~/stsstack-bundles/openstack/novarc
export OS_PASSWORD=$(juju run --unit keystone/0 leader-get admin_passwd)
export OS_CACERT=/etc/ssl/certs/
openstack project list

配置octavia环境

本来是使用下列命令配置octavia环境：

functest-configure -m testmodel

但实际问题一大堆，上面命令实际运行的是/home/ubuntu/charms_pull/octavia/tests/tests.yaml中的下列configure段，建议通过注释确保一次运行一个来逐一排除问题．

configure:
- zaza.openstack.charm_tests.vault.setup.auto_initialize
#- zaza.openstack.charm_tests.glance_simplestreams_sync.setup.sync_images
#- zaza.openstack.charm_tests.octavia.setup.ensure_lts_images
#- zaza.openstack.charm_tests.octavia.diskimage_retrofit.setup.retrofit_amphora_image
#- zaza.openstack.charm_tests.octavia.setup.configure_octavia
#- zaza.openstack.charm_tests.nova.setup.create_flavors
#- zaza.openstack.charm_tests.nova.setup.manage_ssh_key
#- zaza.openstack.charm_tests.neutron.setup.basic_overcloud_network
#- zaza.openstack.charm_tests.octavia.setup.centralized_fip_network

第一步，运行上面的auto_initialize, 它实际做了下列事情．
vault只能产生intermediate csr(juju run-action --wait vault/0 get-csr)，所以我们需要自己准备ca，然后并将intermediate csr签名. 最后将签名后的intermediate cert及CA一起上传至vault (juju run-action vault-ca/leader upload-signed-csr pem="$(cat./pem-bundle.pem|base64)"root-ca="$(cat ./root-bundle.pem | base64 )" allowed-domains=‘quqi.com’ allowed-subdomains=‘client.quqi.com’ allow-any-name=true --wait

hua@t440p:/bak/work/charms/zaza-openstack-tests$ grep -r 'auto_initialize' zaza/openstack/charm_tests/vault/setup.py -A 16
...
def auto_initialize(cacert=None, validation_application='keystone', wait=True):
    ...
    action = vault_utils.run_get_csr()
    intermediate_csr = action.data['results']['output']
    (cakey, cacertificate) = zaza.openstack.utilities.cert.generate_cert(
        'DivineAuthority',
        generate_ca=True)
    intermediate_cert = zaza.openstack.utilities.cert.sign_csr(
        intermediate_csr,
        cakey.decode(),
        cacertificate.decode(),
        generate_ca=True)
    action = vault_utils.run_upload_signed_csr(
        pem=intermediate_cert,
        root_ca=cacertificate,
        allowed_domains='openstack.local')

用的时候这样用，通过＂juju run-action --format=json vault/leader get-root-ca＂获得CA+intermediate ca即可．

#~/stsstack-bundles/openstack/tools/install_local_ca.sh
model_ca_cert_path=/tmp/tmp.0nfIUL8LzE.xxx-bundles.ssl.b00eaccd-df61-4c40-86c6-9ecf7037a521
juju run-action --format=json vault/leader get-root-ca --wait | jq -r .[].results.output > $model_ca_cert_path
sudo cp ${model_ca_cert_path} /usr/local/share/ca-certificates/cacert.crt
sudo chmod 644 /usr/local/share/ca-certificates/cacert.crt
sudo update-ca-certificates --fresh 1>/dev/null

但因为运行在tox中会遇到一个bug, 需添加OS_CACERT=/etc/ssl/certs/

# https://blog.csdn.net/quqi99/article/details/107182847
export OS_CACERT=/etc/ssl/certs/

第二步，注释auto_initialize，只运行sync_images
第三步，这次依次直到只运行basic_overcloud_network (创建ext_net与private), 但报这种错误：

  File "/home/ubuntu/charms_pull/octavia/.tox/func-noop/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 1039, in configure_gateway_ext_port
    net_id = get_admin_net(neutronclient)['id']
TypeError: 'NoneType' object is not subscriptable

原来是要 source ~/novarc，这样就创建成功了，紧接着：

source ~/stsstack-bundles/openstack/novarc
export OS_PASSWORD=$(juju run --unit keystone/0 leader-get admin_passwd)

juju config neutron-api enable-ml2-port-security=true
# wait above step until 'juju status' is good
openstack --insecure network set --enable-port-security $(openstack --insecure network show private -cid -fvalue)

第四步，只运行centralized_fip_network创建private_lb_fip_network，这步也是用upper openstack novarc|openrc

运行测试

运行测试通过下列命令做：

functest-test -m testmodel

也可通过注释octavia/tests/tests.yaml来只运行你想要的测试

tests:
#- zaza.openstack.charm_tests.octavia.tests.LBAASv2Test
- zaza.openstack.charm_tests.octavia.tests.CharmOperationTest
#- zaza.openstack.charm_tests.policyd.tests.OctaviaTests

其他调试手段:

juju model-config logging-config='unit=DEBUG'
juju debug-log -i octavia/0
juju ssh -m controller 0 

测试代码：

cat << EOF | sudo tee /tmp/test.py
import charms.reactive as reactive
import charm.openstack.api_crud as api_crud
identity_service = reactive.endpoint_from_flag('identity-service.available')
# /var/lib/juju/agents/unit-octavia-1/.venv/bin/pip3.8 install rpdb
# nc 127.0.0.1 4444
import rpdb;rpdb.set_trace()
api_crud.delete_hm_port(identity_service, 'octavia-2')
EOF
juju run --unit octavia/1 --timeout 35m0s -- charm-env python3.8 /tmp/test.py

销毁测试

functest-destroy -m testmodel
source ~/novarc && nova list |grep testmodel |awk '{print $2}' |xargs -i nova delete {}

提交代码

对于这种还未提交的CI patch, 在octavia patch中可以引入（见 https://docs.openstack.org/charm-guide/latest/making-a-change.html#functional-test-changes）：

Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/600

如果不使用Func-Test-Pr由lanchpad来run CI的话, 自己测试除了打diff外,还可以:

hua@t440p:/bak/work/charms/octavia$ git diff src/test-requirements.txt
diff --git a/src/test-requirements.txt b/src/test-requirements.txt
index 520681e..ded9b10 100644
--- a/src/test-requirements.txt
+++ b/src/test-requirements.txt
@@ -12,4 +12,5 @@ keyring<21
 
 # Functional Test Requirements (let Zaza's dependencies solve all dependencies here!)
 git+https://github.com/openstack-charmers/zaza.git#egg=zaza
-git+https://github.com/openstack-charmers/zaza-openstack-tests.git#egg=zaza.openstack
+#git+https://github.com/openstack-charmers/zaza-openstack-tests.git#egg=zaza.openstack
+git+https://github.com/zhhuabj/zaza-openstack-tests@hm-port#egg=zaza.openstack