如何在 Grafana 的 Hawkular 指标中监控应用程序的线程数?

Posted

技术标签:

【中文标题】如何在 Grafana 的 Hawkular 指标中监控应用程序的线程数?【英文标题】:How to monitor thread count for apps in Hawkular metrics in Grafana? 【发布时间】:2019-07-18 18:52:08 【问题描述】:

我正在创建一个仪表板,我可以在其中监控来自 openshift 的 hawkular-metrics 中存在的应用程序的线程数。 (数据源已配置)。

我确实有一个 json 文件,用于为我们的 openshift 项目中的所有 pod 创建用于内存使用的仪表板。但是如何修改它以在 hawkular 指标中显示 pod 的线程数。目前它正在为 openshift 项目中的所有应用程序显示平均、最小和最大内存等。

json文件内标签中用于内存使用的描述符名称为memory/usage。 线程计数的描述符名称是什么?有什么我应该做的修改。 Hawkular 真的会导出线程数相关的指标吗?

这是用于内存使用的 json 文件:


    "id": null,
    "title": "Openshift Metrics",
    "tags": [],
    "style": "dark",
    "timezone": "browser",
    "editable": true,
    "hideControls": false,
    "sharedCrosshair": false,
    "rows": [
      
        "collapse": false,
        "editable": true,
        "height": "322px",
        "panels": [
          
            "content": "<center><p style='font-size: 40pt'>$app</p></center>",
            "editable": true,
            "error": false,
            "id": 23,
            "isNew": true,
            "links": [],
            "mode": "html",
            "repeatIteration": 1476706310439,
            "scopedVars": ,
            "span": 2,
            "style": 
              "font-size": "36pt"
            ,
            "title": "",
            "type": "text"
          ,
          
            "aliasColors": ,
            "bars": false,
            "datasource": "Hawk-DS",
            "editable": true,
            "error": false,
            "fill": 1,
            "grid": 
              "threshold1": null,
              "threshold1Color": "rgba(216, 200, 27, 0.27)",
              "threshold2": null,
              "threshold2Color": "rgba(234, 112, 112, 0.22)"
            ,
            "id": 9,
            "isNew": true,
            "legend": 
              "avg": false,
              "current": false,
              "max": false,
              "min": false,
              "show": true,
              "total": false,
              "values": false
            ,
            "lines": true,
            "linewidth": 2,
            "links": [],
            "nullPointMode": "connected",
            "percentage": false,
            "pointradius": 5,
            "points": false,
            "renderer": "flot",
            "repeatIteration": 1476706310439,
            "scopedVars": ,
            "seriesOverrides": [],
            "span": 6,
            "stack": false,
            "steppedLine": false,
            "targets": [
              
                "queryBy": "tags",
                "rate": false,
                "refId": "A",
                "seriesAggFn": "none",
                "tags": [
                  
                    "name": "container_name",
                    "value": "$app"
                  ,
                  
                    "name": "descriptor_name",
                    "value": "memory/usage"
                  
                ],
                "target": "select metric",
                "tagsQL": "container_name IN [$app] AND descriptor_name='memory/usage'",
                "timeAggFn": "avg",
                "type": "gauge"
              
            ],
            "timeFrom": null,
            "timeShift": null,
            "title": "Memory usage",
            "tooltip": 
              "msResolution": true,
              "shared": true,
              "sort": 0,
              "value_type": "cumulative"
            ,
            "type": "graph",
            "xaxis": 
              "show": true
            ,
            "yaxes": [
              
                "format": "bytes",
                "label": null,
                "logBase": 1,
                "max": null,
                "min": 0,
                "show": true
              ,
              
                "format": "short",
                "label": null,
                "logBase": 1,
                "max": null,
                "min": null,
                "show": true
              
            ]
          ,
          
            "cacheTimeout": null,
            "colorBackground": true,
            "colorValue": false,
            "colors": [
              "rgba(50, 172, 45, 0.97)",
              "rgba(237, 129, 40, 0.89)",
              "rgba(245, 54, 54, 0.9)"
            ],
            "datasource": "Hawk-DS",
            "editable": true,
            "error": false,
            "format": "bytes",
            "gauge": 
              "maxValue": 100,
              "minValue": 0,
              "show": false,
              "thresholdLabels": false,
              "thresholdMarkers": true
            ,
            "height": "100px",
            "id": 12,
            "interval": null,
            "isNew": true,
            "links": [],
            "mappingType": 1,
            "mappingTypes": [
              
                "name": "value to text",
                "value": 1
              ,
              
                "name": "range to text",
                "value": 2
              
            ],
            "maxDataPoints": 100,
            "nullPointMode": "connected",
            "nullText": null,
            "postfix": "",
            "postfixFontSize": "50%",
            "prefix": "",
            "prefixFontSize": "50%",
            "rangeMaps": [
              
                "from": "null",
                "text": "N/A",
                "to": "null"
              
            ],
            "repeatIteration": 1476706310439,
            "scopedVars": ,
            "span": 2,
            "sparkline": 
              "fillColor": "rgba(31, 118, 189, 0.18)",
              "full": false,
              "lineColor": "rgb(31, 120, 193)",
              "show": false
            ,
            "targets": [
              
                "queryBy": "tags",
                "rate": false,
                "refId": "A",
                "seriesAggFn": "sum",
                "tags": [
                  
                    "name": "container_name",
                    "value": "$app"
                  ,
                  
                    "name": "descriptor_name",
                    "value": "memory/usage"
                  
                ],
                "target": "select metric",
                "tagsQL": "container_name IN [$app] AND descriptor_name='memory/usage'",
                "timeAggFn": "live",
                "type": "gauge"
              
            ],
            "thresholds": "140000000,180000000",
            "title": "Live, all pods",
            "type": "singlestat",
            "valueFontSize": "80%",
            "valueMaps": [
              
                "op": "=",
                "text": "N/A",
                "value": "null"
              
            ],
            "valueName": "avg"
          ,
          
            "cacheTimeout": null,
            "colorBackground": true,
            "colorValue": false,
            "colors": [
              "rgba(50, 172, 45, 0.97)",
              "rgba(237, 129, 40, 0.89)",
              "rgba(245, 54, 54, 0.9)"
            ],
            "datasource": "Hawk-DS",
            "editable": true,
            "error": false,
            "format": "bytes",
            "gauge": 
              "maxValue": 100,
              "minValue": 0,
              "show": false,
              "thresholdLabels": false,
              "thresholdMarkers": true
            ,
            "height": "100px",
            "id": 15,
            "interval": null,
            "isNew": true,
            "links": [],
            "mappingType": 1,
            "mappingTypes": [
              
                "name": "value to text",
                "value": 1
              ,
              
                "name": "range to text",
                "value": 2
              
            ],
            "maxDataPoints": 100,
            "nullPointMode": "connected",
            "nullText": null,
            "postfix": "",
            "postfixFontSize": "50%",
            "prefix": "",
            "prefixFontSize": "50%",
            "rangeMaps": [
              
                "from": "null",
                "text": "N/A",
                "to": "null"
              
            ],
            "repeatIteration": 1476706310439,
            "scopedVars": ,
            "span": 2,
            "sparkline": 
              "fillColor": "rgba(31, 118, 189, 0.18)",
              "full": false,
              "lineColor": "rgb(31, 120, 193)",
              "show": false
            ,
            "targets": [
              
                "queryBy": "tags",
                "rate": false,
                "refId": "A",
                "seriesAggFn": "avg",
                "tags": [
                  
                    "name": "container_name",
                    "value": "$app"
                  ,
                  
                    "name": "descriptor_name",
                    "value": "memory/usage"
                  
                ],
                "target": "select metric",
                "tagsQL": "container_name IN [$app] AND descriptor_name='memory/usage'",
                "timeAggFn": "live",
                "type": "gauge"
              
            ],
            "thresholds": "140000000,180000000",
            "title": "Live per pod",
            "type": "singlestat",
            "valueFontSize": "80%",
            "valueMaps": [
              
                "op": "=",
                "text": "N/A",
                "value": "null"
              
            ],
            "valueName": "avg"
          ,
          
            "cacheTimeout": null,
            "colorBackground": true,
            "colorValue": false,
            "colors": [
              "rgba(50, 172, 45, 0.97)",
              "rgba(237, 129, 40, 0.89)",
              "rgba(245, 54, 54, 0.9)"
            ],
            "datasource": "Hawk-DS",
            "editable": true,
            "error": false,
            "format": "bytes",
            "gauge": 
              "maxValue": 100,
              "minValue": 0,
              "show": false,
              "thresholdLabels": false,
              "thresholdMarkers": true
            ,
            "height": "100px",
            "id": 10,
            "interval": null,
            "isNew": true,
            "links": [],
            "mappingType": 1,
            "mappingTypes": [
              
                "name": "value to text",
                "value": 1
              ,
              
                "name": "range to text",
                "value": 2
              
            ],
            "maxDataPoints": 100,
            "nullPointMode": "connected",
            "nullText": null,
            "postfix": "",
            "postfixFontSize": "50%",
            "prefix": "",
            "prefixFontSize": "50%",
            "rangeMaps": [
              
                "from": "null",
                "text": "N/A",
                "to": "null"
              
            ],
            "repeatIteration": 1476706310439,
            "scopedVars": ,
            "span": 2,
            "sparkline": 
              "fillColor": "rgba(31, 118, 189, 0.18)",
              "full": false,
              "lineColor": "rgb(31, 120, 193)",
              "show": false
            ,
            "targets": [
              
                "queryBy": "tags",
                "rate": false,
                "refId": "A",
                "seriesAggFn": "sum",
                "tags": [
                  
                    "name": "container_name",
                    "value": "$app"
                  ,
                  
                    "name": "descriptor_name",
                    "value": "memory/usage"
                  
                ],
                "target": "select metric",
                "tagsQL": "container_name IN [$app] AND descriptor_name='memory/usage'",
                "timeAggFn": "avg",
                "type": "gauge"
              
            ],
            "thresholds": "140000000,180000000",
            "title": "Average, all pods",
            "type": "singlestat",
            "valueFontSize": "80%",
            "valueMaps": [
              
                "op": "=",
                "text": "N/A",
                "value": "null"
              
            ],
            "valueName": "avg"
          ,
          
            "cacheTimeout": null,
            "colorBackground": true,
            "colorValue": false,
            "colors": [
              "rgba(50, 172, 45, 0.97)",
              "rgba(237, 129, 40, 0.89)",
              "rgba(245, 54, 54, 0.9)"
            ],
            "datasource": "Hawk-DS",
            "editable": true,
            "error": false,
            "format": "bytes",
            "gauge": 
              "maxValue": 100,
              "minValue": 0,
              "show": false,
              "thresholdLabels": false,
              "thresholdMarkers": true
            ,
            "height": "100px",
            "id": 13,
            "interval": null,
            "isNew": true,
            "links": [],
            "mappingType": 1,
            "mappingTypes": [
              
                "name": "value to text",
                "value": 1
              ,
              
                "name": "range to text",
                "value": 2
              
            ],
            "maxDataPoints": 100,
            "nullPointMode": "connected",
            "nullText": null,
            "postfix": "",
            "postfixFontSize": "50%",
            "prefix": "",
            "prefixFontSize": "50%",
            "rangeMaps": [
              
                "from": "null",
                "text": "N/A",
                "to": "null"
              
            ],
            "repeatIteration": 1476706310439,
            "scopedVars": ,
            "span": 2,
            "sparkline": 
              "fillColor": "rgba(31, 118, 189, 0.18)",
              "full": false,
              "lineColor": "rgb(31, 120, 193)",
              "show": false
            ,
            "targets": [
              
                "queryBy": "tags",
                "rate": false,
                "refId": "A",
                "seriesAggFn": "avg",
                "tags": [
                  
                    "name": "container_name",
                    "value": "$app"
                  ,
                  
                    "name": "descriptor_name",
                    "value": "memory/usage"
                  
                ],
                "target": "select metric",
                "tagsQL": "container_name IN [$app] AND descriptor_name='memory/usage'",
                "timeAggFn": "avg",
                "type": "gauge"
              
            ],
            "thresholds": "140000000,180000000",
            "title": "Average per pod",
            "type": "singlestat",
            "valueFontSize": "80%",
            "valueMaps": [
              
                "op": "=",
                "text": "N/A",
                "value": "null"
              
            ],
            "valueName": "avg"
          ,
          
            "cacheTimeout": null,
            "colorBackground": true,
            "colorValue": false,
            "colors": [
              "rgba(50, 172, 45, 0.97)",
              "rgba(237, 129, 40, 0.89)",
              "rgba(245, 54, 54, 0.9)"
            ],
            "datasource": "Hawk-DS",
            "editable": true,
            "error": false,
            "format": "bytes",
            "gauge": 
              "maxValue": 100,
              "minValue": 0,
              "show": false,
              "thresholdLabels": false,
              "thresholdMarkers": true
            ,
            "height": "100px",
            "id": 11,
            "interval": null,
            "isNew": true,
            "links": [],
            "mappingType": 1,
            "mappingTypes": [
              
                "name": "value to text",
                "value": 1
              ,
              
                "name": "range to text",
                "value": 2
              
            ],
            "maxDataPoints": 100,
            "nullPointMode": "connected",
            "nullText": null,
            "postfix": "",
            "postfixFontSize": "50%",
            "prefix": "",
            "prefixFontSize": "50%",
            "rangeMaps": [
              
                "from": "null",
                "text": "N/A",
                "to": "null"
              
            ],
            "repeatIteration": 1476706310439,
            "scopedVars": ,
            "span": 2,
            "sparkline": 
              "fillColor": "rgba(31, 118, 189, 0.18)",
              "full": false,
              "lineColor": "rgb(31, 120, 193)",
              "show": false
            ,
            "targets": [
              
                "queryBy": "tags",
                "rate": false,
                "refId": "A",
                "seriesAggFn": "sum",
                "tags": [
                  
                    "name": "container_name",
                    "value": "$app"
                  ,
                  
                    "name": "descriptor_name",
                    "value": "memory/usage"
                  
                ],
                "target": "select metric",
                "tagsQL": "container_name IN [$app] AND descriptor_name='memory/usage'",
                "timeAggFn": "max",
                "type": "gauge"
              
            ],
            "thresholds": "140000000,180000000",
            "title": "Max, all pods",
            "type": "singlestat",
            "valueFontSize": "80%",
            "valueMaps": [
              
                "op": "=",
                "text": "N/A",
                "value": "null"
              
            ],
            "valueName": "avg"
          ,
          
            "cacheTimeout": null,
            "colorBackground": true,
            "colorValue": false,
            "colors": [
              "rgba(50, 172, 45, 0.97)",
              "rgba(237, 129, 40, 0.89)",
              "rgba(245, 54, 54, 0.9)"
            ],
            "datasource": "Hawk-DS",
            "editable": true,
            "error": false,
            "format": "bytes",
            "gauge": 
              "maxValue": 100,
              "minValue": 0,
              "show": false,
              "thresholdLabels": false,
              "thresholdMarkers": true
            ,
            "height": "100px",
            "id": 14,
            "interval": null,
            "isNew": true,
            "links": [],
            "mappingType": 1,
            "mappingTypes": [
              
                "name": "value to text",
                "value": 1
              ,
              
                "name": "range to text",
                "value": 2
              
            ],
            "maxDataPoints": 100,
            "nullPointMode": "connected",
            "nullText": null,
            "postfix": "",
            "postfixFontSize": "50%",
            "prefix": "",
            "prefixFontSize": "50%",
            "rangeMaps": [
              
                "from": "null",
                "text": "N/A",
                "to": "null"
              
            ],
            "repeatIteration": 1476706310439,
            "scopedVars": ,
            "span": 2,
            "sparkline": 
              "fillColor": "rgba(31, 118, 189, 0.18)",
              "full": false,
              "lineColor": "rgb(31, 120, 193)",
              "show": false
            ,
            "targets": [
              
                "queryBy": "tags",
                "rate": false,
                "refId": "A",
                "seriesAggFn": "avg",
                "tags": [
                  
                    "name": "container_name",
                    "value": "$app"
                  ,
                  
                    "name": "descriptor_name",
                    "value": "memory/usage"
                  
                ],
                "target": "select metric",
                "tagsQL": "container_name IN [$app] AND descriptor_name='memory/usage'",
                "timeAggFn": "max",
                "type": "gauge"
              
            ],
            "thresholds": "140000000,180000000",
            "title": "Max per pod",
            "type": "singlestat",
            "valueFontSize": "80%",
            "valueMaps": [
              
                "op": "=",
                "text": "N/A",
                "value": "null"
              
            ],
            "valueName": "avg"
          
        ],
        "repeat": "app",
        "scopedVars": 
          "app": 
            "text": "aloha",
            "value": "aloha",
            "selected": true
          
        ,
        "title": "New row"
      
    ],
    "time": 
      "from": "now-30m",
      "to": "now"
    ,
    "timepicker": 
      "refresh_intervals": [
        "5s",
        "10s",
        "30s",
        "1m",
        "5m",
        "15m",
        "30m",
        "1h",
        "2h",
        "1d"
      ],
      "time_options": [
        "5m",
        "15m",
        "1h",
        "6h",
        "12h",
        "24h",
        "2d",
        "7d",
        "30d"
      ]
    ,
    "templating": 
      "list": [
        
          "current": ,
          "datasource": "Hawk-DS",
          "hide": 0,
          "includeAll": true,
          "label": "Application",
          "multi": true,
          "name": "app",
          "options": [],
          "query": "tags/container_name:*",
          "refresh": 1,
          "regex": "",
          "type": "query"
        
      ]
    ,
    "annotations": 
      "list": []
    ,
    "schemaVersion": 12,
    "version": 32,
    "links": [],
    "gnetId": null

请提供任何有用的建议。 是否有任何文档可以查看 hawkular 提供的指标列表?

【问题讨论】:

【参考方案1】:

我不知道 openshift w/hawkular 中的任何线程计数指标。

它在后台使用 heapster,请在此处查看指标:https://github.com/kubernetes-retired/heapster/blob/master/docs/storage-schema.md#metrics。

但我不能 100% 确定它们是否都被报告给 hawkular。

【讨论】:

以上是关于如何在 Grafana 的 Hawkular 指标中监控应用程序的线程数?的主要内容,如果未能解决你的问题,请参考以下文章

如何在 Grafana 表中汇总每天的指标?

Grafana + Prometheus 如何在查询指标中使用正则表达式?

如何在 grafana 上显示指标时用标签替换目标

如何在grafana指标中连接两个字符串

如何将 Reactjs 日志和指标推送到 Grafana

如何配置 Grafana 以捕捉 Prometheus 指标的急剧下降?