Observability：在容器里运行 Elastic Agent - Elastic Stack 8.x

Posted Elastic 中国社区官方博客

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Observability：在容器里运行 Elastic Agent - Elastic Stack 8.x相关的知识，希望对你有一定的参考价值。

你可以在容器内运行 Elastic Agent — 使用 Fleet Server 或独立运行。 可从 Elastic Docker registry 获取所有版本的 Elastic Agent 的 Docker 映像。如果你想单独安装 Elastic Agent 及 Fleet Server，请参考我之前的文章 “Observability：使用 Elastic Agent 来摄入日志及指标 - Elastic Stack 8.0”。

当我们考虑把 Elastic Agent 或 Fleet Server 安装于容器时，我们需要考虑：

  • 当 Elastic Agent 在容器内运行时，它无法通过 Fleet 升级，因为它期望容器本身会升级。
  • 注册和运行 Elastic Agent 通常是一个两步过程。 但是，这在容器中不起作用，因此调用了一个特殊的子命令 container。 此命令允许环境变量配置所有属性，并将注册和运行命令作为单个命令运行。

下拉镜像

Elastic Agent 有两个镜像：elastic-agent 和 elastic-agent-complete。 elastic-agent 映像包含用于运行 Beats 的所有二进制文件，而 elastic-agent-complete 映像包含这些二进制文件以及通过 Elastic Synthetics 运行浏览器监视器通过 Elastic Synthetics 运行浏览器监视器的其他依赖项。 有关更多信息，请参阅通过 Elastic Agent 和 Fleet 进行Synthetic 监控。

我们可以通过如下的方法来进行下载镜像：

docker pull docker.elastic.co/beats/elastic-agent:version

如果要运行 Synthetic 测试，请运行 docker pull 命令来获取 elastic-agent-complete 映像：

docker pull docker.elastic.co/beats/elastic-agent-complete:version

在今天的展示中，我们将使用最新的 Elastic Agent 8.2.0 来进行展示。那么我们将使用如下的命令来进行下载镜像：

docker pull docker.elastic.co/beats/elastic-agent:8.2.0

在上面提到我们将使用 container 子命令来进行安装。关于 container 子命令的描述，我们可以通过下载一个本地安装的 Elastic Agent 来进行查询。我首先到地址 Download Elastic Agent Free | Elastic 下载相应平台的 tar 包，并进行接压缩：

tar xzf elastic-agent-8.2.0-darwin-x86_64.tar.gz
cd elastic-agent-8.2.0-darwin-x86_64

我们在命令行中打入如下的命令：

elastic-agent container -h
$ pwd
/Users/liuxg/elastic/elastic-agent-8.2.0-darwin-x86_64
$ ./elastic-agent container -h
This should only be used as an entrypoint for a container. This will prepare the Elastic Agent using
environment variables to run inside of the container.

The following actions are possible and grouped based on the actions.

* Elastic Agent Fleet Enrollment
  This enrolls the Elastic Agent into a Fleet Server. It is also possible to have this create a new enrollment token
  for this specific Elastic Agent.

  FLEET_ENROLL - set to 1 for enrollment into fleet-server. If not set, Elastic Agent is run in standalone mode.
  FLEET_URL - URL of the Fleet Server to enroll into
  FLEET_ENROLLMENT_TOKEN - token to use for enrollment. This is not needed in case FLEET_SERVER_ENABLED and FLEET_ENROLL is set. Then the token is fetched from Kibana.
  FLEET_CA - path to certificate authority to use with communicate with Fleet Server [$KIBANA_CA]
  FLEET_INSECURE - communicate with Fleet with either insecure HTTP or unverified HTTPS

  The following vars are need in the scenario that Elastic Agent should automatically fetch its own token.

  KIBANA_FLEET_HOST - kibana host to enable create enrollment token on [$KIBANA_HOST]
  FLEET_TOKEN_NAME - token name to use for fetching token from Kibana. This requires Kibana configs to be set.
  FLEET_TOKEN_POLICY_NAME - token policy name to use for fetching token from Kibana. This requires Kibana configs to be set.

* Bootstrapping Fleet Server
  This bootstraps the Fleet Server to be run by this Elastic Agent. At least one Fleet Server is required in a Fleet
  deployment for other Elastic Agent to bootstrap. In case the Elastic Agent is run without fleet-server. These variables
  are not needed.

  If FLEET_SERVER_ENABLE and FLEET_ENROLL is set but no FLEET_ENROLLMENT_TOKEN, the token is automatically fetched from Kibana.

  FLEET_SERVER_ENABLE - set to 1 enables bootstrapping of Fleet Server inside Elastic Agent (forces FLEET_ENROLL enabled)
  FLEET_SERVER_ELASTICSEARCH_HOST - elasticsearch host for Fleet Server to communicate with [$ELASTICSEARCH_HOST]
  FLEET_SERVER_ELASTICSEARCH_CA - path to certificate authority to use with communicate with elasticsearch [$ELASTICSEARCH_CA]
  FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT - The sha-256 fingerprint value of the certificate authority to trust
  FLEET_SERVER_ELASTICSEARCH_INSECURE - disables cert validation for communication with Elasticsearch
  FLEET_SERVER_SERVICE_TOKEN - service token to use for communication with elasticsearch
  FLEET_SERVER_POLICY_ID - policy ID for Fleet Server to use for itself ("Default Fleet Server policy" used when undefined)
  FLEET_SERVER_HOST - binding host for Fleet Server HTTP (overrides the policy). By default this is 0.0.0.0.
  FLEET_SERVER_PORT - binding port for Fleet Server HTTP (overrides the policy)
  FLEET_SERVER_CERT - path to certificate to use for HTTPS endpoint
  FLEET_SERVER_CERT_KEY - path to private key for certificate to use for HTTPS endpoint
  FLEET_SERVER_INSECURE_HTTP - expose Fleet Server over HTTP (not recommended; insecure)

* Preparing Kibana for Fleet
  This prepares the Fleet plugin that exists inside of Kibana. This must either be enabled here or done externally
  before Fleet Server will actually successfully start. All the Kibana variables are not needed in case Elastic Agent
  should not setup Fleet. To manually trigger KIBANA_FLEET_SETUP navigate to Kibana -> Fleet -> Agents and enabled it.

  KIBANA_FLEET_SETUP - set to 1 enables the setup of Fleet in Kibana by Elastic Agent. This was previously FLEET_SETUP.
  KIBANA_FLEET_HOST - Kibana host accessible from fleet-server. [$KIBANA_HOST]
  KIBANA_FLEET_USERNAME - kibana username to service token [$KIBANA_USERNAME]
  KIBANA_FLEET_PASSWORD - kibana password to service token [$KIBANA_PASSWORD]
  KIBANA_FLEET_CA - path to certificate authority to use with communicate with Kibana [$KIBANA_CA]
  KIBANA_REQUEST_RETRY_SLEEP - specifies sleep duration taken when agent performs a request to kibana [default 1s]
  KIBANA_REQUEST_RETRY_COUNT - specifies number of retries agent performs when executing a request to kibana [default 30]

The following environment variables are provided as a convenience to prevent a large number of environment variable to
be used when the same credentials will be used across all the possible actions above.

  ELASTICSEARCH_HOST - elasticsearch host [http://elasticsearch:9200]
  ELASTICSEARCH_USERNAME - elasticsearch username [elastic]
  ELASTICSEARCH_PASSWORD - elasticsearch password [changeme]
  ELASTICSEARCH_CA - path to certificate authority to use with communicate with elasticsearch
  KIBANA_HOST - kibana host [http://kibana:5601]
  KIBANA_FLEET_USERNAME - kibana username to enable Fleet [$ELASTICSEARCH_USERNAME]
  KIBANA_FLEET_PASSWORD - kibana password to enable Fleet [$ELASTICSEARCH_PASSWORD]
  KIBANA_CA - path to certificate authority to use with communicate with Kibana [$ELASTICSEARCH_CA]


By default when this command starts it will check for an existing fleet.yml. If that file already exists then
all the above actions will be skipped, because the Elastic Agent has already been enrolled. To ensure that enrollment
occurs on every start of the container set FLEET_FORCE to 1.

Usage:
  elastic-agent container [flags]

Flags:
  -h, --help   help for container

Global Flags:
  -c, --c string                     Configuration file, relative to path.config (default "elastic-agent.yml")
  -d, --d string                     Enable certain debug selectors
  -e, --e                            Log to stderr and disable syslog/file output
      --environment environmentVar   set environment being ran in (default default)
      --path.config string           Config path is the directory Agent looks for its config file (default "/Users/liuxg/elastic/elastic-agent-8.2.0-darwin-x86_64")
      --path.downloads string        Downloads path contains binaries Agent downloads
      --path.home string             Agent root path (default "/Users/liuxg/elastic/elastic-agent-8.2.0-darwin-x86_64")
      --path.install string          Install path contains binaries Agent extracts
      --path.logs string             Logs path contains Agent log output (default "/Users/liuxg/elastic/elastic-agent-8.2.0-darwin-x86_64")
  -v, --v                            Log at INFO level

上面的命令列出来的详细信息，对我们下面的配置非常有用。特别是上面列出来的一些环境变量，这个在下面的 docker 命令中将被使用到。我们需要详细阅读。

在下面的展示中，我们将针对自管型的 Elastic Stack 安装来进行配置。

安装

我们首先来安装好自己的 Elasticsearch 及 Kibana。如果你还没有安装好它们，请根据自己的平台按照我之前的教程：

请注意文章中的 8.x 的安装部分。为了能够让 fleet 正常工作，内置的 API service 必须启动。我们必须为 Elasticsearch 的配置文件 config/elasticsearch.yml 文件配置：

xpack.security.authc.api_key.enabled: true

配置完后，我们再重新启动 Elasticsearch。针对 Kibana，我们也需要做一个额外的配置。我们需要修改 config/kibana.yml 文件。在这个文件的最后面，添加如下的一行：

 xpack.encryptedSavedObjects.encryptionKey: 'fhjskloppd678ehkdfdlliverpoolfcr'

如果你不想使用上面的这个设置，你可以在 Kibana 的安装目录下使用如下的方式来获得：

./bin/kibana-encryption-keys generate

从上面的输出中，我们可以看出来，有三个输出的 key。我们可以把这三个同时拷贝，并添加到 config/kibana.yml 文件的后面。当然，我们也可以只拷贝其中的一个也可。我们再重新启动 Kibana。

除此之外，Kibana 需要 Internet 连接才能从 Elastic Package Registry 下载集成包。 确保 Kibana 服务器可以连接到https://epr.elastic.co 的端口 443 上 。如果你的环境有网络流量限制，有一些方法可以解决此要求。 有关详细信息，请参阅气隙环境。

目前，Fleet 只能被具有 superuser role 的用户所使用。

这样我们的 Elasticsearch 及 Kibana 就安装好了。我们的配资如下：

 

配置 Fleet

接下来，我们来配置 Fleet。在 Elastic Agent 能够正常运行起来之前，我们必须安装 Fleet Server。如果你已经有一个 Fleet Server，那么就可以跳过不安装。否则，我们必须安装一个 Fleet Server。

Fleet Server 和 Elastic Agent 之间非安全连接

 

 

 

 

 

 

我们第一次选择 Quick start，也就是说，我们使用如下的配置：

 

我们在 Elastic Agent 和 Fleet Server 之间采取 http 连接。正如上面所说的，我们必须使用 insecure 标志。

期末点击上面的 Generate service token，并拷贝这个 token：

上面显示了如何 Start Fleet Server。显然这个并不适合我们的情况，因为我们要使用 container 来进行启动。

我们在一个 terminal 中打入如下的命令：

docker run \\
  -p 8200:8200 \\
  --env FLEET_SERVER_ENABLE=true \\
  --env FLEET_SERVER_ELASTICSEARCH_HOST=https://192.168.0.3:9200 \\
  --env FLEET_SERVER_SERVICE_TOKEN=AAAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NTI3NjQ1MDk3NzY6ZVZwbk1UYWxReWVqNW1jX2lfQVI4Zw  \\
  --env FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT=1d84ebeb2d99075edf73cc94bfa8c3e22fdccbbd72d632608b70b2b5be90ca79 \\
  --env FLEET_SERVER_INSECURE_HTTP \\
  --env FLEET_SERVER_POLICY_ID=fleet-server-policy \\
  --rm docker.elastic.co/beats/elastic-agent:8.2.0

请注意：

  • FLEET_SERVER_SERVICE_TOKEN 是我们在上面生成的 token。
  • FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT 是在上面的 Start Fleet Server 中的命令行中 fleet-server-es-ca-trusted-fingerprint 所定义的。关于这些变量，你可以参考我上面的 container 子命令的帮助。

 

点击上面的 Continue 按钮：

 

从上面的输出中，我们可以看出来我们的安装是成功的。

Fleet Server 和 Elastic Agent 之间安全连接

在进行下面的安装之前，我们可以取消之前的注册：

直到我们完全没有之前的注册被取消：

 

 我们在生产环境中，必须保证 Fleet Server 和 Elastic Agent 之间的安全连接：

在这种情况下，我们必须生产相应的证书以保证这个连接的安全性。

我们回到 Elasticsearch 的安装目录下，并打入如下的命令：

./bin/elasticsearch-certutil ca --pem
$ pwd
/Users/liuxg/elastic/elasticsearch-8.2.0
$ ./bin/elasticsearch-certutil ca --pem
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.zip]: 
$ ls
LICENSE.txt          bin                  elastic-stack-ca.zip logs
NOTICE.txt           config               jdk.app              modules
README.asciidoc      data                 lib                  plugins
$ unzip elastic-stack-ca.zip 
Archive:  elastic-stack-ca.zip
   creating: ca/
  inflating: ca/ca.crt               
  inflating: ca/ca.key

从上面的输出中，我们可以看到已经生成了一个 ca.crt 的证书。我们把这个证书拷贝到一个自定义的目录中：

$ cp ca/ca.crt ~/certs/
overwrite /Users/liuxg/certs/ca.crt? (y/n [n]) y

接下来，我们使用 CA 来为 Fleet Server 生成证书：

./bin/elasticsearch-certutil cert \\
  --name fleet-server \\
  --ca-cert /path/to/ca/ca.crt \\
  --ca-key /path/to/ca/ca.key \\
  --dns your.host.name.here \\
  --ip 192.0.2.1 \\
  --pem

针对我的情况：

./bin/elasticsearch-certutil cert \\
  --name fleet-server \\
  --ca-cert ca/ca.crt \\
  --ca-key ca/ca.key \\
  --dns localhost \\
  --ip 127.0.0.1 \\
  --pem
$ ./bin/elasticsearch-certutil cert \\
>   --name fleet-server \\
>   --ca-cert ca/ca.crt \\
>   --ca-key ca/ca.key \\
>   --dns localhost \\
>   --ip 127.0.0.1 \\
>   --pem
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file

    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.


    * All certificates generated by this tool will be signed by a certificate authority (CA)
      unless the --self-signed command line option is specified.
      The tool can automatically generate a new CA for you, or you can provide your own with
      the --ca or --ca-cert command line options.


By default the 'cert' mode produces a single PKCS#12 output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate

If you specify any of the following options:
    * -pem (PEM formatted output)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Please enter the desired output file [certificate-bundle.zip]: 

Certificates written to /Users/liuxg/elastic/elasticsearch-8.2.0/certificate-bundle.zip

This file should be properly secured as it contains the private key for 
your instance.
After unzipping the file, there will be a directory for each instance.
Each instance has a certificate and private key.
For each Elastic product that you wish to configure, you should copy
the certificate, key, and CA certificate to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
$ unzip certificate-bundle.zip 
Archive:  certificate-bundle.zip
   creating: fleet-server/
  inflating: fleet-server/fleet-server.crt  
  inflating: fleet-server/fleet-server.key

从上面的输出中，我们可以看到在 fleet-server 的目录下，我们解压缩看到 fleet server 的证书。

我们接下来通过如下的命令来把把这个证书拷贝到上面 ca.crt 所在的目录中：

$ cp fleet-server/fleet-server.* ~/certs
overwrite /Users/liuxg/certs/fleet-server.crt? (y/n [n]) y
overwrite /Users/liuxg/certs/fleet-server.key? (y/n [n]) y

这样在我们的 ~/certs 目录下有如下的文件：

$ pwd
/Users/liuxg/certs
$ ls
ca.crt           fleet-server.crt fleet-server.key

我们接下来进行如下步骤的配置：

 

 

这一次，我们把 Fleet Server 的地址换成是 https 而不是之前的 http 的。

 

 

 

 

 

这次我们选择  Production 而不之前的 Quick Start：

 

我们接下来生成 service token：

 

我们首先拷贝 service token，并按照上面的安装指令来寻找相应的 container 环境变量。我们使用如下的命令：

 

docker run \\
  -p 8200:8200 \\
  -v /Users/liuxg/certs/ca.crt:/usr/share/elastic-agent/ca.crt \\
  -v /Users/liuxg/certs/fleet-server.crt:/usr/share/elastic-agent/fleet-server.crt \\
  -v /Users/liuxg/certs/fleet-server.key:/usr/share/elastic-agent/fleet-server.key \\
  --env FLEET_SERVER_ENABLE=true \\
  --env FLEET_SERVER_ELASTICSEARCH_HOST=https://192.168.0.3:9200 \\
  --env FLEET_SERVER_SERVICE_TOKEN=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NTI3NjY2NjI3ODU6eGZodW4tOGJRb3liOHlMbndxckp1QQ \\
  --env FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT=1d84ebeb2d99075edf73cc94bfa8c3e22fdccbbd72d632608b70b2b5be90ca79 \\
  --env FLEET_URL=https://127.0.0.1:8220 \\
  --env FLEET_CA=/usr/share/elastic-agent/ca.crt \\
  --env FLEET_SERVER_CERT=/usr/share/elastic-agent/fleet-server.crt \\
  --env FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/fleet-server.key \\
  --env FLEET_SERVER_POLICY_ID=fleet-server-policy \\
  --rm docker.elastic.co/beats/elastic-agent:8.2.0

请注意上面的 证书挂载。我们需要根据自己的安装替换如下的变量：

  • FLEET_SERVER_SERVICE_TOKEN
  • FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT
  • FLEET_SERVER_ELASTICSEARCH_HOST

上面的值可以在安装指令中找到。

 

 

从上面的图中，我们可以看出来 Fleet Server 的创建以及 Elastic Agent 的注册是成功。 

我们可以登录到 container 并使用如下的命令来检查 Fleet Server 的状态：

$ docker ps
CONTAINER ID   IMAGE                                         COMMAND                  CREATED         STATUS         PORTS                    NAMES
853fa75133f2   docker.elastic.co/beats/elastic-agent:8.2.0   "/usr/bin/tini -- /u…"   2 minutes ago   Up 2 minutes   0.0.0.0:8200->8200/tcp   hungry_jang
$ docker exec -it 853fa75133f2 bash
elastic-agent@853fa75133f2:~$ curl --cacert /usr/share/elastic-agent/ca.crt https://127.0.0.1:8220/api/status
"name":"fleet-server","status":"HEALTHY"elastic-agent@853fa75133f2:~$

参考：

【1】Run Elastic Agent in a container | Fleet and Elastic Agent Guide [master] | Elastic

以上是关于Observability：在容器里运行 Elastic Agent - Elastic Stack 8.x的主要内容，如果未能解决你的问题，请参考以下文章

Observability：日志监控和非结构化日志数据，超越 tail -f

Observability：Elastic 可观测性是什么？

Observability：使用 Elastic Agent 来进行 Uptime 监控

Observability：从零开始创建 Java 微服务并监控它

Observability：从零开始创建 Java 微服务并监控它

Observability：在 Elastic Observability 部署中添加免费和开放的 Elastic APM