如何使用任务命中在 azure 批处理节点中运行的烧瓶应用程序？

Posted 2023-02-24

【中文标题】如何使用任务命中在 azure 批处理节点中运行的烧瓶应用程序？

【英文标题】：how to hit a flask app running in azure batch node using a task?

【发布时间】：2020-01-19 09:11:47

【问题描述】：

我能够配置以下设置，

[]

我可以在作业中添加一个任务来运行 testapi.py 文件，该文件将一个简单的字符串作为输入并将其发送到在 docker 容器内运行的烧瓶应用程序。

但是，

任务执行报错

> Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 183, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe6ff71e630>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=6789): Max retries exceeded with url: /upload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6ff71e630>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "preload_testapi.py", line 37, in <module>
    r = requests.post(url, json=got_json)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=6789): Max retries exceeded with url: /upload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6ff71e630>: Failed to establish a new connection: [Errno 111] Connection refused',))

我做过的事情

我确保启动任务（运行烧瓶应用程序）是 由我创建的 sudo 用户（不是 _azbatch 用户）执行，任务 里面的作业也由同一个用户执行。

我通过 ssh 到 与我创建的用户登录相同的节点并且命令运行 很好。

当我从 _azbatch 运行命令时，它会引发相同的错误。

这不是烧瓶端口问题。

@fpark 我的启动任务执行已正确完成，在节点创建期间，我从 Azure 容器注册表下载图像，从 azure 存储 blob 下载 2 个文件作为资源文件，启动任务是运行其中一个文件（shell 脚本）制作一个容器，从该图像在其中运行一个烧瓶应用程序。添加的任务是使用第二个文件（testapi.py）为每个任务使用不同的参数来访问正在运行的烧瓶应用程序。

【问题讨论】：

你有机会使用 docker 吗？ Same issue 在这里讨论并通过 chaning dockerfile 解决。

Another discussion 建议使用requests 而不是urlib3，如果可能的话。

@ŠimonKocúrek 我在我的 testapi.py 文件中使用请求

@simon 对不起，如果我没有说清楚，但我没有使用 azure cli，使用 azure python sdk Link

您能否用相关的启动任务和任务配置扩展您的问题？很难理解您对现有问题的处理方式。

【参考方案1】：

您可以使用任务访问烧瓶 API。按照这种方式：

为您的计算节点提供一个包含两个任务的作业，其中任务 1 运行烧瓶 API，任务 2 命中 API。并在尝试之前为您的节点启用并行任务执行。

希望对你有帮助(～￣▽￣)～