Transformer课程业务对话机器人Rasa 3.x Testing Your Assistant

Posted 2023-03-18

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Transformer课程业务对话机器人Rasa 3.x Testing Your Assistant相关的知识，希望对你有一定的参考价值。

RASA 官网

https://rasa.com/

Transformer课程

Testing Your Assistant

Rasa开源允许您通过运行测试故事来验证和测试端到端的对话。此外，还可以分别测试对话管理和消息处理（NLU）。

Validating Data and Stories

验证数据和故事#

数据验证可验证您的域、NLU数据或故事数据中是否未出现错误或重大不一致。要验证数据，请运行以下命令：

rasa data validate

运行结果如下

Transformer课程

如果将max_history 值传递给配置config.yml 文件中的一个或多个策略，提供这些值中的最小值，如下所示：

rasa data validate --max-history <max_history>

运行结果如下

Transformer课程

如果数据验证导致错误，那么训练模型也可能失败或产生糟糕的性能，因此在训练模型之前运行此检查总是好的。通过包含–fail-on-warnings标志，此步骤将在指示更多次要问题的警告时失败。

运行 rasa data validate数据验证不会测试您的规则是否与您的故事一致。但是，在训练期间，RulePolicy会检查规则和故事之间的冲突。任何此类冲突都将中止训练。

要阅读有关验证器和所有可用选项的更多信息，请参阅rasa数据验证文档。

Writing Test Stories

在测试故事中测试您的训练模型是对您的助手在某些情况下的行为有信心的最佳方式。测试故事以修改后的故事格式编写，允许您提供完整的对话，并测试在给定特定用户输入的情况下，您的模型将以预期的方式运行。当您开始从用户对话中引入更复杂的故事时，这一点尤为重要。

测试故事与训练数据中的故事相似，但也包括用户消息。
以下是一些例子：

Basics
tests/test_stories.yml

stories:
- story: A basic story test
  steps:
  - user: |
      hello
    intent: greet
  - action: utter_ask_howcanhelp
  - user: |
     show me [chinese]"entity": "cuisine" restaurants
    intent: inform
  - action: utter_ask_location
  - user: |
      in [Paris]"entity": "location"
    intent: inform
  - action: utter_ask_price

Custom Actions
tests/test_stories.yml

stories:
- story: A test where a custom action returns events
  steps:
  - user: |
      hey
    intent: greet
  - action: my_custom_action
  - slot_was_set:
    - my_slot: "value added by custom action"
  - action: utter_ask_age
  - user: |
      thanks
    intent: thankyou
  - action: utter_no_worries

Forms Happy Path
tests/test_stories.yml

stories:
- story: A test story with a form
  steps:
  - user: |
      hi
    intent: greet
  - action: utter_greet
  - user: |
      im looking for a restaurant
    intent: request_restaurant
  - action: restaurant_form
  - active_loop: restaurant_form
  - user: |
      [afghan](cuisine) food
    intent: inform
  - action: restaurant_form
  - active_loop: null
  - action: utter_slots_values
  - user: |
      thanks
    intent: thankyou
  - action: utter_no_worries

Forms Unhappy Path

tests/test_stories.yml

stories:
- story: A test story with unexpected input during a form
  steps:
  - user: |
      hi
    intent: greet
  - action: utter_greet
  - user: |
      im looking for a restaurant
    intent: request_restaurant
  - action: restaurant_form
  - active_loop: restaurant_form
  - user: |
      Hows the weather?
    intent: chitchat
  - action: utter_chitchat
  - action: restaurant_form
  - active_loop: null
  - action: utter_slots_values
  - user: |
      thanks
    intent: thankyou
  - action: utter_no_worries

默认情况下，该命令将对任何名称以test_开头的文件中的故事运行测试。您还可以使用–stories参数提供特定的测试故事文件或目录。您可以通过运行以下命令来测试助手：

rasa test

test_stories.yml

#### This file contains tests to evaluate that your bot behaves as expected.
#### If you want to learn more, please see the docs: https://rasa.com/docs/rasa/testing-your-assistant

stories:
- story: happy path 1
  steps:
  - user: |
      hello there!
    intent: greet
  - action: utter_greet
  - user: |
      amazing
    intent: mood_great
  - action: utter_happy

- story: happy path 2
  steps:
  - user: |
      hello there!
    intent: greet
  - action: utter_greet
  - user: |
      amazing
    intent: mood_great
  - action: utter_happy
  - user: |
      bye-bye!
    intent: goodbye
  - action: utter_goodbye

- story: sad path 1
  steps:
  - user: |
      hello
    intent: greet
  - action: utter_greet
  - user: |
      not good
    intent: mood_unhappy
  - action: utter_cheer_up
  - action: utter_did_that_help
  - user: |
      yes
    intent: affirm
  - action: utter_happy

- story: sad path 2
  steps:
  - user: |
      hello
    intent: greet
  - action: utter_greet
  - user: |
      not good
    intent: mood_unhappy
  - action: utter_cheer_up
  - action: utter_did_that_help
  - user: |
      not really
    intent: deny
  - action: utter_goodbye

- story: sad path 3
  steps:
  - user: |
      hi
    intent: greet
  - action: utter_greet
  - user: |
      very terrible
    intent: mood_unhappy
  - action: utter_cheer_up
  - action: utter_did_that_help
  - user: |
      no
    intent: deny
  - action: utter_goodbye

- story: say goodbye
  steps:
  - user: |
      bye-bye!
    intent: goodbye
  - action: utter_goodbye

- story: bot challenge
  steps:
  - user: |
      are you a bot?
    intent: bot_challenge
  - action: utter_iamabot

运行结果如下

Transformer课程

intent_report.json


  "deny": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 7,
    "confused_with": 
  ,
  "mood_unhappy": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 14,
    "confused_with": 
  ,
  "greet": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 13,
    "confused_with": 
  ,
  "mood_great": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 14,
    "confused_with": 
  ,
  "bot_challenge": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 4,
    "confused_with": 
  ,
  "affirm": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 6,
    "confused_with": 
  ,
  "goodbye": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 10,
    "confused_with": 
  ,
  "accuracy": 1.0,
  "macro avg": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 68
  ,
  "weighted avg": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 68

story_report.json


  "goodbye": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 2
  ,
  "utter_cheer_up": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 3
  ,
  "utter_did_that_help": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 3
  ,
  "utter_iamabot": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 1
  ,
  "mood_unhappy": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 3
  ,
  "greet": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 5
  ,
  "utter_greet": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 5
  ,
  "mood_great": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 2
  ,
  "deny": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 2
  ,
  "bot_challenge": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 1
  ,
  "utter_happy": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 3
  ,
  "utter_goodbye": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 4
  ,
  "affirm": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 1
  ,
  "action_listen": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 16
  ,
  "accuracy": 1.0,
  "macro avg": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 51
  ,
  "weighted avg": 
    "precision": 1.0,
    "recall": 1.0,
    "f1-score": 1.0,
    "support": 51
  ,
  "conversation_accuracy": 
    "accuracy": 1.0,
    "correct": 7,
    "with_warnings": 0,
    "total": 7

会话测试只与包含的测试用例一样准确，因此您应该在改进助手的同时继续增加测试用例集。一个很好的经验法则是，你应该让你的测试故事代表真实对话的真实分布。RASAX使基于真实对话添加测试对话变得容易。

rasa test

有关更多配置选项，请参阅rasa测试的CLI文档。

测试自定义操作

Custom Actions自定义操作不会作为测试故事的一部分执行。如果您的自定义操作将任何事件附加到对话中，这必须反映在您的测试故事中（例如，通过向您的测试故事中添加slot_was_set事件）。

要测试自定义操作的代码，应该为它们编写单元测试，并将这些测试包括在CI/CD管道中。

Evaluating an NLU Model

除了测试故事外，还可以单独测试自然语言理解（NLU）模型。一旦您的助手部署到现实世界中，它将处理训练数据中未显示的消息。为了模拟这种情况，您应该始终留出部分数据用于测试。您可以使用以下方法将NLU数据拆分为训练集和测试集：

rasa data split nlu

Transformer课程

test_data.yml

version: "3.0"
nlu:
- intent: bot_challenge
  examples: |
    - are you a bot?
- intent: affirm
  examples: |
    - of course
- intent: deny
  examples: |
    - no way
    - n
- intent: goodbye
  examples: |
    - have a nice day
    - cu
- intent: greet
  examples: |
    - lets go
    - hi
- intent: mood_great
  examples: |
    - so perfect
    - great
    - wonderful
- intent: mood_unhappy
  examples: |
    - Im so sad
    - very sad
    - so saad

training_data.yml

version: "3.0"
nlu:
- intent: bot_challenge
  examples: |
    - are you a human?
    - am I talking to a human?
    - am I talking to a bot?
- intent: affirm
  examples: |
    - correct
    - indeed
    - y
    - that sounds good
    - yes
- intent: deny
  examples: |
    - never
    - I dont think so
    - not really
    - no
    - dont like that
- intent: goodbye
  examples: |
    - cee you later
    - good night
    - good by
    - goodbye
    - bye bye
    - see you around
    - see you later
    - bye
- intent: greet
  examples: |
    - hello there
    - good afternoon
    - good morning
    - goodevening
    - hey there
    - goodmorning
    - hey
    - hey dude
    - moin
    - hello
    - good evening
- intent: mood_great
  examples: |
    - super stoked
    - I am going to save the world
    - I am feeling very good
    - so good
    - I am great
    - perfect
    - extremely good
    - amazing
    - so so perfect
    - I am amazing
    - feeling like a king
- intent: mood_unhappy
  examples: |
    - my day was horrible
    - I am disappointed
    - not very good
    - not good
    - so sad
    - unhappy
    - I am sad
    - I dont feel very well
    - super sad
    - extremly sad
    - sad

接下来，您可以使用以下方法查看经过训练的NLU模型对生成的测试集数据的预测效果：

rasa test nlu
    --nlu train_test_split/test_data.yml

Transformer课程

要更广泛地测试模型，请使用交叉验证，它会自动创建多个训练/测试拆分：

rasa test nlu
    --nlu data/nlu.yml
    --cross-validation

运行结果如下

Transformer课程

Rasa系列博客：

业务对话机器人Rasa 3.x Internals及Rasa框架定制实战
业务对话机器人Rasa核心算法DIET及TED论文详解
业务对话机器人Rasa 3.x部署安装初体验
业务对话机器人Rasa 3.x Playground
业务对话机器人Rasa 3.x Command Line Interface
业务对话机器人Rasa 3.x 命令 rasa shell 及rasa run
业务对话机器人Rasa 3.x 命令rasa run actions、rasa test、rasa data split 、rasa data convert nlu
业务对话机器人Rasa 3.x 命令rasa data migrate、rasa data validate、rasa export、rasa evaluate markers、rasa x
业务对话机器人Rasa 3.x 会话驱动开发（Conversation-Driven Development）
业务对话机器人Rasa 3.x 生成自然语言理解NLU数据
业务对话机器人Rasa 3.x 写入会话数据（Writing Conversation Data）
业务对话机器人Rasa 3.x 聊天和FAQs常见问题解答
业务对话机器人Rasa 3.x docker安装部署
业务对话机器人Rasa 3.x Handling Business Logic
业务对话机器人Rasa 3.x Fallback and Human Handoff
业务对话机器人Rasa 3.x Handling Unexpected Input
业务对话机器人Rasa 3.x Contextual Conversations
业务对话机器人Rasa 3.x Reaching Out to the User
业务对话机器人Rasa 3.x 微信+Rasa机器人实战案例
业务对话机器人Rasa 3.x Tuning Your NLU Model
业务对话机器人Rasa 3.x Tuning Your NLU Model（二）

Transformer课程业务对话机器人Rasa 3.x 持续集成和持续部署

Continuous Integration and Continuous Deployment

即使开发上下文助手不同于开发传统软件，您仍然应该遵循软件开发最佳实践。设置持续集成和持续部署管道可确保对bot的增量更新能够改善它，而不会损害它。

Overview

持续集成是频繁合并代码更改并在提交更改时自动测试更改的实践。连续部署意味着自动将集成的更改部署到暂存或生产环境。它们使您能够更频繁地改进助手，并有效地测试和部署这些更改。

本指南将涵盖特定于Rasa项目的持续集成、持续部署管道中应包含的内容。建议选择一个与您使用的任何Git存储库集成的工具。

Continuous Integration

改进助手的最佳方法是经常进行增量更新。无论变化有多小，您都要确保它不会带来新问题或对助理的绩效产生负面影响。

通常最好在合并/拉取请求或提交时运行检查。大多数测试都足够快，可以在每次更改时运行。但是，仅当某些文件已更改或存在其他指示器时，才可以选择运行资源密集型测试。例如，如果您的代码托管在Github上，则只有在拉取请求具有特定标签（例如“需要NLU测试”）的情况下，才能进行测试运行。

Continuous Integration Pipeline Overview

持续集成管道应包括模型训练和测试，作为简化部署过程的步骤。保存新训练数据后的第一步是启动管道。这可以手动启动，也可以在创建或更新请求时启动。

接下来，您需要运行各种测试集，以查看更改的影响。这包括为数据验证、NLU交叉验证和故事测试运行测试。有关测试的更多信息，请参阅测试助手。

最后一步是检查测试结果，如果测试成功，则推动更改。一旦新模型经过训练和测试，就可以使用连续部署管道自动部署它。

GitHub Actions CI Pipeline

您可以在持续集成管道中使用Rasa Train Test Github操作自动执行数据验证、训练和测试。

使用Github操作的持续集成管道示例如下所示：

jobs:
  training-testing:
    name: Training and Testing
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - name: Rasa Train and Test GitHub Action
        uses: RasaHQ/rasa-train-test-gha@main
        with:
          requirements_file: requirements.txt
          data_validate: true
          rasa_train: true
          cross_validation: true
          rasa_test: true
          test_type: all
          publish_summary: true
          github_token: $ secrets.GITHUB_TOKEN 
      - name: Upload model
        if: github.ref == refs/heads/main
        uses: actions/upload-artifact@master
        with:
          name: model
          path: models

在此管道中，Rasa Train Test Github操作在第一步执行数据验证、模型训练和故事测试，在第二步将模型文件上传。

存储库的自述文件中提供了Rasa Train Test Github操作的完整可配置参数列表。

当publish_summary设置为true时，此操作将自动将模型的测试结果作为注释发布到关联的Pull请求：

Transformer课程

可以根据评估结果批准或拒绝拉取请求，并且在许多情况下，如果所有检查都通过，您将希望自动化模型的部署。您可以继续到下一节，以了解有关连续部署的更多信息。

Continuous Deployment

要经常向用户提供改进，您需要尽可能多地自动化部署过程。

一旦检查成功，持续部署步骤通常在推送或合并到某个分支时运行。

Deploying Your Rasa Model

如果您在持续集成管道中运行了测试故事，那么您已经拥有了一个经过训练的模型。如果持续集成结果令人满意，您可以设置持续部署管道，将经过训练的模型上载到Rasa服务器。例如，要将模型上载到Rasa X：

curl -k -F "model=@models/my_model.tar.gz" "https://example.rasa.com/api/projects/default/models?api_token=your_api_token"

如果您使用的是Rasa X，则还可以将上载的模型标记为生产（或使用多个部署环境时要标记的任何部署）：

curl -X PUT "https://example.rasa.com/api/projects/default/models/my_model/tags/production"

行动守则的更新

如果更新包括对模型和操作代码的更改，并且这些更改以任何方式相互依赖，则不应自动将模型标记为生产。您首先需要构建和部署更新的操作服务器，以便新模型不会调用更新前操作服务器中不存在的操作。

Deploying Your Action Server

对于操作代码的每次更新，您都可以自动为操作服务器构建新映像并将其上载到映像存储库。如上所述，如果操作服务器与当前生产模型不兼容，请小心在生产中自动部署新的映像标记。

作为示例，请参见Rara Rasa助手以及Carbon Bot。两者都将Github操作用作持续集成和持续部署工具。

这些例子只是许多可能性中的两个。如果您有自己喜欢的持续集成和持续部署设置，请与论坛上的Rasa社区共享。

https://github.com/RasaHQ/carbon-bot/blob/master/.github/workflows/model_ci.yml

name: Model CI
on:
  push:
    branches:
    - master
  pull_request:
    types: [opened, synchronize, reopened]

env:
  DOMAIN: carbon.rasa.com
  GITHUB_TOKEN: $ secrets.GITHUB_TOKEN 
  RELEASE_NAME: carbon-assistant
  NAMESPACE: carbon-assistant
  RASA_X_IMAGE_NAME: $ secrets.RASA_X_IMAGE_NAME 
  RASA_X_USERNAME: admin
  RASA_X_PASSWORD: $ secrets.RASA_X_ADMIN_PASSWORD 

jobs:
  build-model:
    name: Build, test, and upload model
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2

    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7

    - name: Install dependencies
      run: |
        python -m pip install --upgrade "pip<20"
        pip install -r requirements.txt
    - name: Check stories are consistent
      run: |
        rasa data validate stories --max-history 5 --fail-on-warning
    - name: Train model
      run: |
        rasa train
    - name: Run Through Test Stories
      run: |
        rasa test core --stories test_stories/stories.yml --fail-on-prediction-errors
    - name: Cross-validate NLU model
      id: cvnlu
      if: github.event_name == pull_request
      run: |
        rasa test nlu -f 2 --cross-validation
        python format_results.py
    - name: Upload Cross Validation Results
      if: github.event_name == pull_request
      uses: actions/upload-artifact@v2
      with:
        name: cross-validation-result
        path: results.md

    - name: Upload model
      if: github.event_name == push && (startsWith(github.event.ref, refs/tags) || github.ref == refs/heads/master)
      uses: actions/upload-artifact@v2
      with:
        name: model
        path: models

    - name: Post cross-val results to PR
      if: steps.cvnlu.outcome == success
      uses: samsucik/comment-on-pr@comment-file-contents
      continue-on-error: true
      with:
        msg: results.md

  build-images:
    name: Build and Push Images
    runs-on: ubuntu-latest
    env:
      IS_PUSH_EVENT: $ github.event_name == push 

    steps:
    - uses: actions/checkout@v2

    - name: Set image tag
      run: |
        if [[ $IS_PUSH_EVENT == "false" ]]
        then
          IMAGE_TAG=$ github.head_ref 
        else
          IMAGE_TAG=$(basename $ github.ref )
        fi
        echo "IMAGE_TAG=$IMAGE_TAG-$ github.sha " >> $GITHUB_ENV
    # Setup gcloud CLI
    - uses: google-github-actions/setup-gcloud@v0.2.1
      name: Google Auth
      with:
        service_account_key: $ secrets.GKE_SA_KEY 
        project_id: $ secrets.GKE_PROJECT 

    # Configure docker to use the gcloud command-line tool as a credential helper
    - run: |-
        gcloud --quiet auth configure-docker
    - name: Build and push the Docker image
      run: |
        # Read and export variables from .env file
        set -o allexport; source .env; set +o allexport
        docker build . \\
          --build-arg RASA_SDK_IMAGE=rasa/rasa-sdk:$RASA_SDK_VERSION \\
          --tag gcr.io/$ secrets.GKE_PROJECT /carbon-bot-actions:$IMAGE_TAG
        docker push gcr.io/$ secrets.GKE_PROJECT /carbon-bot-actions:$IMAGE_TAG
  deploy-to-cluster:
    name: Re-deploy the cluster with the latest action server
    runs-on: ubuntu-latest

    needs:
    - build-images

    if: github.event_name == push && (startsWith(github.event.ref, refs/tags) || github.ref == refs/heads/master)

    steps:
    # Checkout repository because we need the content of the `.env` file later
    - uses: actions/checkout@v2

    - name: Set image tag
      env:
        IS_PUSH_EVENT: $ github.event_name == push 
      run: |
        if [[ $IS_PUSH_EVENT == "false" ]]
        then
          IMAGE_TAG=$ github.head_ref 
        else
          IMAGE_TAG=$(basename $ github.ref )
        fi
        echo "IMAGE_TAG=$IMAGE_TAG-$ github.sha " >> $GITHUB_ENV
    # Setup gcloud CLI
    - uses: google-github-actions/setup-gcloud@v0.2.1
      name: Google Auth
      with:
        service_account_key: $ secrets.GKE_SA_KEY 
        project_id: $ secrets.GKE_PROJECT 

    # Configure docker to use the gcloud command-line tool as a credential helper
    - run: |-
        gcloud --quiet auth configure-docker
    # Get the GKE credentials so we can deploy to the cluster
    - uses: google-github-actions/get-gke-credentials@v0.2.1
      with:
        cluster_name: $ secrets.GKE_CLUSTER 
        location: $ secrets.GKE_ZONE 
        credentials: $ secrets.GKE_SA_KEY 

    - name: Install Helm and helmfile ⛑
      run: |
        curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
        chmod 700 get_helm.sh
        ./get_helm.sh
        sudo curl -fsSL https://github.com/roboll/helmfile/releases/download/v0.138.7/helmfile_linux_amd64 --output /usr/local/bin/helmfile
        sudo chmod +x /usr/local/bin/helmfile
    - name: Install Chart
      env:
        DB_PASSWORD: $ secrets.DB_PASSWORD 
        RASA_TOKEN: $ secrets.RASA_TOKEN 
        RASA_X_TOKEN: $ secrets.RASA_X_TOKEN 
        JWT_SECRET: $ secrets.JWT_SECRET 
        PASSWORD_SALT: $ secrets.PASSWORD_SALT 
        REDIS_PASSWORD: $ secrets.REDIS_PASSWORD 
        RABBITMQ_PASSWORD: $ secrets.RABBITMQ_PASSWORD 
        RASA_X_ADMIN_PASSWORD: $ secrets.RASA_X_ADMIN_PASSWORD 
        GKE_PROJECT: $ secrets.GKE_PROJECT 
        FACEBOOK_VERIFY_TOKEN: $ secrets.FACEBOOK_VERIFY_TOKEN 
        FACEBOOK_APP_SECRET: $ secrets.FACEBOOK_APP_SECRET 
        FACEBOOK_PAGE_ACCESS_TOKEN: $ secrets.FACEBOOK_PAGE_ACCESS_TOKEN 
        CLIMATIQ_API_KEY: $ secrets.CLIMATIQ_API_KEY 

      run: |
        # Install helm v3
        curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
        # Read and export variables from .env file
        set -o allexport; source .env; set +o allexport
        kubectl create ns $NAMESPACE || true
        cd .github/deployments; helmfile sync
        cat <<EOF | kubectl apply --namespace $NAMESPACE -f  -
        apiVersion: networking.gke.io/v1beta1
        kind: ManagedCertificate
        metadata:
          generation: 1
          name: rasa-bots-certificate
        spec:
          domains:
          - $ env.DOMAIN 
        EOF
  upload-model:
    name: Upload the trained model to Rasa X
    needs:
    - deploy-to-cluster
    - build-model
    env:
      MODEL_DIRECTORY: "models"

    if: github.event_name == push && (startsWith(github.event.ref, refs/tags) || github.ref == refs/heads/master)
    runs-on: ubuntu-latest

    steps:
    - name: Download Model
      uses: actions/download-artifact@v2
      with:
        name: model
        path: $ env.MODEL_DIRECTORY 

    - name: Get path to model
      run: |
        ls -R
        echo "MODELNAME=$ env.MODEL_DIRECTORY /$(ls $ env.MODEL_DIRECTORY )" >> $GITHUB_ENV
    - name: Upload Model to Rasa
      run: |
        # Get token
        RASA_X_TOKEN=$(curl -s --header "Content-Type: application/json" \\
          --request POST \\
          --data "\\"username\\":\\"$RASA_X_USERNAME\\",\\"password\\":\\"$RASA_X_PASSWORD\\"" \\
          https://$ env.DOMAIN /api/auth | jq -r .access_token)
        # Upload model
        curl -k --fail -H "Authorization: Bearer $RASA_X_TOKEN" -F "model=@$MODELNAME" https://$ env.DOMAIN /api/projects/default/models
        # ensure model is ready and tag as production
        sleep 5
        export MODEL=$(basename $MODELNAME .tar.gz)
        curl --fail -XPUT -H "Authorization: Bearer $RASA_X_TOKEN" https://$ env.DOMAIN /api/projects/default/models/$MODEL/tags/production

https://github.com/RasaHQ/rasa-demo/blob/main/.github/workflows/continuous_integration.yml

name: Continuous Integration

on: [pull_request]
env:
GDRIVE_CREDENTIALS: $ secrets.GDRIVE_CREDENTIALS
MAILCHIMP_LIST: $ secrets.MAILCHIMP_LIST
MAILCHIMP_API_KEY: $ secrets.MAILCHIMP_API_KEY
ALGOLIA_APP_ID: $ secrets.ALGOLIA_APP_ID
ALGOLIA_SEARCH_KEY: $ secrets.ALGOLIA_SEARCH_KEY
ALGOLIA_DOCS_INDEX: $ secrets.ALGOLIA_DOCS_INDEX
RASA_X_HOST: $ secrets.RASA_X_DOMAIN
RASA_X_PASSWORD: $ secrets.RASA_X_PASSWORD
RASA_X_USERNAME: $ secrets.RASA_X_USERNAME
RASA_X_HOST_SCHEMA: $ secrets.RASA_X_HOST_SCHEMA
GITHUB_TOKEN: $ secrets.GITHUB_TOKEN
TRACKER_DB_URL: $ secrets.TRACKER_DB_URL
# Due to the issue with openssl library for Google Cloud SDK (gcloud)
# (https://github.com/GoogleCloudPlatform/github-actions/issues/128)
# we use 297.0.01 version
GCLOUD_VERSION: "297.0.1"

concurrency:
group: ci-$ github.ref
cancel-in-progress: true

jobs:
check_changed_files:
name: Check for file changes
runs-on: ubuntu-20.04
outputs:
nlu: $ steps.changed-files.outputs.nlu
core: $ steps.changed-files.outputs.core
training: $ steps.changed-files.outputs.training
actions: $ steps.changed-files.outputs.actions
steps:
# Due to an issue with checking out a wrong commit, we make sure
# to checkout HEAD commit for a pull request.
# More details: https://github.com/actions/checkout/issues/299
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == pull_request
with:
ref: $ github.event.pull_request.head.sha

- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != pull_request

- uses: RasaHQ/pr-changed-files-filter@c4f7116a04b8a4596313469429e2ad235f59d9c4
id: changed-files
with:
token: $ secrets.GITHUB_TOKEN
filters: .github/change_filters.yml
base: $ github.ref
lint-testing:
name: Code Formatting Tests
runs-on: ubuntu-latest
steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == pull_request
with:
ref: $ github.event.pull_request.head.sha

- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != pull_request

- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements-dev.txt
- name: Code Formatting Tests
run: |
echo "------------------------------------"
echo "/usr/bin/git log -1 --format=%H"
/usr/bin/git log -1 --format=%H
echo "------------------------------------"
make lint
type-testing:
name: Type Tests
runs-on: ubuntu-latest
steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == pull_request
with:
ref: $ github.event.pull_request.head.sha

- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != pull_request

- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7

- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements-dev.txt
- name: Type Checking
run: |
pip list
make types
action-unit-tests:
needs:
- lint-testing
- type-testing
name: Custom Action Unit Tests
runs-on: ubuntu-latest

steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == pull_request
with:
ref: $ github.event.pull_request.head.sha

- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != pull_request

- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7

- name: Install dependencies
run: |
make install-dev
- name: Unit Tests
run: |
make test-actions
data-validation:
name: Data Validation
runs-on: ubuntu-latest

steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == pull_request
with:
ref: $ github.event.pull_request.head.sha

- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != pull_request

- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7

- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements-dev.txt
- name: Rasa Data Validation
run: |
rasa data validate --debug
training-testing:
name: Test Model
runs-on: ubuntu-latest
needs:
- data-validation
- check_changed_files
if: $ needs.check_changed_files.outputs.training == true

steps:
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.8.0
with:
access_token: $ github.token

- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == pull_request
with:
ref: $ github.event.pull_request.head.sha

- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != pull_request

- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7

- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements-dev.txt
rasa --version
- name: Cross-validate NLU model
id: cvnlu
if: contains( github.event.pull_request.labels.*.name, nlu_testing_required )
run: |
rasa --version
rasa test nlu -f 3 --cross-validation --config config_nlu_testing.yml
python .github/workflows/format_results.py
- name: post cross-val results to PR
if: steps.cvnlu.outcome == success
uses: amn41/comment-on-pr@comment-file-contents
continue-on-error: true
with:
msg: results.md

- name: Train Model
run: |
rasa --version
rasa train
- name: Test End 2 End Stories
if: $ needs.check_changed_files.outputs.core == true
run: |
rasa --version
rasa test core --stories tests/test_conversations.yml --fail-on-prediction-errors
- name: Wait for the conclusion of all other workflow runs
# upload model from PR
if: github.event_name == pull_request
id: check-runs-conclusion
env:
WAIT_INTERVAL_SECS: 10
timeout-minutes: 20
run: |
while true; do
# Get a list of checks information, excluding training-testing and build-images
CHECKS_LIST=$(gh api /repos/$ github.repository /commits/$ github.event.pull_request.head.sha /check-runs --jq .check_runs.[] | select(.name != "Test Model" and .name != "Build Action Server Image"))
# Get the status and conclusion of echo jobs
STATUS_LIST=$(echo $CHECKS_LIST | jq -r .status)
CONCLUSION_LIST=$(echo $CHECKS_LIST | jq -r .conclusion)
# Make sure all other check runs are completed
if [[ "$(echo $STATUS_LIST | tr \\n | sort | uniq)" == "completed" ]]; then
# Check the conclusion of all other check runs
# Fail the step if there is any failture
if [[ "$(echo CONCLUSION_LIST | tr \\n | sort | uniq)" =~ "failure" ]]; then
echo "::error:: Some check runs failed. Skip uploading model."
exit 1
else
echo "All other check runs are successed."
echo "::set-output name=upload-model::true"
exit 0
fi
fi
sleep $WAIT_INTERVAL_SECS
echo "Wait for $WAIT_INTERVAL_SECS seconds..."
done
- name: Set model name from Rasa version
if: |
github.event_name == pull_request &&
steps.check-runs-conclusion.outputs.upload-model == true
run: |
python -c "import rasa; open(rasaversion.txt,w+).write(rasa.__version__)"
rasa_version=`cat rasaversion.txt`
model_path=`ls models/*.tar.gz | head -n 1`
model_timestamp=$(basename "$model_path" .tar.gz)
model_name="$model_timestamp"_rasa"$rasa_version"
renamed_model_path=models/"$model_name".tar.gz
mv $model_path $renamed_model_path
echo "MODEL_NAME=$model_name" >> $GITHUB_ENV
echo "MODEL_PATH=$renamed_model_path" >> $GITHUB_ENV
- uses: google-github-actions/setup-gcloud@master
if: |
(github.event_name == pull_request &&
contains( github.event.pull_request.labels.*.name, upload_model )) ||
steps.check-runs-conclusion.outputs.upload-model == true
name: Authenticate with gcloud

以上是关于Transformer课程业务对话机器人Rasa 3.x Testing Your Assistant的主要内容，如果未能解决你的问题，请参考以下文章

Transformer课程业务对话机器人Rasa 3.x 持续集成和持续部署

Gavin老师Transformer直播课感悟 - Rasa对话机器人项目实战之教育领域Education Bot项目架构运行测试流程分析及Rasa interactive实验分析（六十)

Rasa 3.x 学习系列-Rasa 3.1+ ElasticSearch 7.15.0 对话机器人实战八 Elasticsearch 可视化及SpacyNLP安装部署

报时机器人的rasa shell执行流程分析

rasa学习

以命令行模式记录Rasa