JMX_exporter 在 Prometheus 和 Grafana 中显示错误

Posted 2023-02-15

【中文标题】JMX_exporter 在 Prometheus 和 Grafana 中显示错误

【英文标题】：JMX_exporter shows error in Prometheus & Grafana

【发布时间】：2018-05-11 21:44:54

【问题描述】：

我已经使用 JMX exporter 来监控基于 jetty 部署的 Java 应用程序。

我已经下载了jmx_prometheus_javaagent-0.1.0.jar 文件

使用 jmx_exporter 命令部署 java 应用程序

nohup java -javaagent:./jmx_prometheus_javaagent-0.1.0.jar=7101:config.yaml -Dorg.eclipse.jetty.server.Request.maxFormContentSize=10000000 -Xms256m -Xmx256m -Djava.io.tmpdir=epoch_temp_dir -jar jetty-runner-9.0.7.v20131107.jar --log yyyy_mm_dd-java-application-1-request.log --out yyyy_mm_dd-java-application-1-output.log --port 8091 --path /java-application-1 java-app1.war >> java-application-1.log 2>&1 &

config.yaml

 #cat config.yaml
---
startDelaySeconds: 0
jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:7101/jmxrmi
ssl: false
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
- pattern: ".*"

Prometheus 在状态页面显示连接超时

输出日志：已部署的 java 应用程序

io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector collect
SEVERE: JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error du
ring JRMP connection establishment; nested exception is:
        java.net.SocketTimeoutException: Read timed out]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
        at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:106)
        at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:415)
        at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:180)
        at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:213)
        at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:134)
        at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
        at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59)
        at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
        at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
        at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
        at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
        at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.net.SocketTimeoutException: Read timed out]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136)
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
        at javax.naming.InitialContext.lookup(InitialContext.java:417)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
        ... 17 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        java.net.SocketTimeoutException: Read timed out
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)

在客户端服务器中打开 7101 端口，并授予对 Prometheus 服务器的访问权限。

Prometheus 服务器内存使用情况：

# free -h
              total        used        free      shared  buff/cache   available
Mem:           4.8G        1.9G        118M        256M        2.8G        2.3G
Swap:            0B          0B          0B

客户端服务器内存：

# free -h
              total        used        free      shared  buff/cache   available
Mem:           9.8G        3.7G        435M         16M        5.6G        5.7G
Swap:            0B          0B          0B

客户端服务器中的Curl localhost:7101没有响应

【参考方案1】：

从配置中删除 jmxUrl，这不是代理使用所必需的。

【讨论】：

嗨，布莱恩。我得到了完全相同的错误，我的配置中没有那个 jmxUrl。知道这里出了什么问题吗？

您还应该删除“hostPort”配置参数。因为，如果存在，它将用于构建 jmxUrl，正如 brian 所报告的，代理使用不需要。