yaml：使用 go 时解组错误

Posted 2023-02-16

技术标签:

【中文标题】yaml：使用 go 时解组错误【英文标题】：yaml: unmarshal errors when using go 【发布时间】：2021-09-15 23:15:31 【问题描述】：

我运行代码：https://github.com/prometheus-community/json_exporter 通过更改配置将 json 数据转换为 prometheus 指标数据类型

错误是：

level=error ts=2021-07-05T06:19:52.714Z caller=main.go:58 msg="Error loading config" err="yaml: unmarshal errors:\n  line 7: cannot unmarshal !!str `id:\".id\"` into map[string]string"

这是我的 yaml 配置：

metrics:
- name: running_applications
  type: object 
  help: Information on running applications
  path: '.apps.app[?(@.state=="RUNNING")]'
  labels:
        id:'.id'
  values:
        memory: '.allocatedMB'
        vcores: '.allocatedVCores'

这是json的一部分：

"apps":
"app":
["id":"application_1622098898935_1480","user":"hive","name":"HIVE-b5770b0d-fdd0-4ee4-86f8-1684df70dae9","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1480/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624665783612 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626080103_7098ab3b-ea51-4780-ba5d-aa823335b102,userid=user-app_ebupt","priority":0,"startedTime":1624665666232,"finishedTime":1624666419590,"elapsedTime":753358,"amContainerLogs":"http://tst02.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1480_01_000001/hive","amHostHttpAddress":"tst02.bj2.sdbmcc.com:8042","masterNodeId":"tst02.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3752447,"vcoreSeconds":915,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":"entry":"key":"memory-mb","value":"3752447","entry":"key":"vcores","value":"915","preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3752447,"preemptedVcoreSeconds":915,"preemptedResourceSecondsMap":"entry":"key":"memory-mb","value":"3752447","entry":"key":"vcores","value":"915","logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":"timeout":["type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1],
"id":"application_1622098898935_1479","user":"hive","name":"HIVE-db17be65-b129-497b-a89c-28b001acf3fc","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1479/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624663983139 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626073104_b7560acd-42db-46b8-bbd2-72e4f0599624,userid=user-app_ebupt","priority":0,"startedTime":1624663872431,"finishedTime":1624664618529,"elapsedTime":746098,"amContainerLogs":"http://tst06.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1479_01_000001/hive","amHostHttpAddress":"tst06.bj2.sdbmcc.com:8042","masterNodeId":"tst06.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3802102,"vcoreSeconds":927,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":"entry":"key":"memory-mb","value":"3802102","entry":"key":"vcores","value":"927","preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3802102,"preemptedVcoreSeconds":927,"preemptedResourceSecondsMap":"entry":"key":"memory-mb","value":"3802102","entry":"key":"vcores","value":"927","logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":"timeout":["type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1],
"id":"application_1622098898935_1478","user":"hive","name":"HIVE-2805816c-9723-4d32-90f7-146d7bf61946","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1478/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624662198199 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626070103_77d85e4a-ead5-43d1-b36f-8c766719c551,userid=user-app_ebupt","priority":0,"startedTime":1624662071009,"finishedTime":1624662813506,"elapsedTime":742497,"amContainerLogs":"http://tst02.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1478_01_000001/hive","amHostHttpAddress":"tst02.bj2.sdbmcc.com:8042","masterNodeId":"tst02.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3947826,"vcoreSeconds":962,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":"entry":"key":"memory-mb","value":"3947826","entry":"key":"vcores","value":"962","preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3947826,"preemptedVcoreSeconds":962,"preemptedResourceSecondsMap":"entry":"key":"memory-mb","value":"3947826","entry":"key":"vcores","value":"962","logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":"timeout":["type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1],
"id":"application_1622098898935_1477","user":"hive","name":"HIVE-df9d6d21-5011-41f7-8482-dc3f214b6d38","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1477/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624660381598 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626063103_be7a2ca1-35a3-4435-8db5-60e72b97c076,userid=user-app_ebupt","priority":0,"startedTime":1624660266652,"finishedTime":1624661012324,"elapsedTime":745672,"amContainerLogs":"http://tst06.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1477_01_000001/hive","amHostHttpAddress":"tst06.bj2.sdbmcc.com:8042","masterNodeId":"tst06.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3813440,"vcoreSeconds":930,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":"entry":"key":"memory-mb","value":"3813440","entry":"key":"vcores","value":"930","preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3813440,"preemptedVcoreSeconds":930,"preemptedResourceSecondsMap":"entry":"key":"memory-mb","value":"3813440","entry":"key":"vcores","value":"930","logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":"timeout":["type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1],
"id":"application_1622098898935_1476","user":"hive","name":"HIVE-114bdd08-b399-4c2f-8ec9-6320985f612a","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1476/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624658578652 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626060103_1d7f2f72-62de-4d27-be50-0625bbc10409,userid=user-app_ebupt","priority":0,"startedTime":1624658466988,"finishedTime":1624659212326,"elapsedTime":745338,"amContainerLogs":"http://tst02.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1476_01_000001/hive","amHostHttpAddress":"tst02.bj2.sdbmcc.com:8042","masterNodeId":"tst02.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3818593,"vcoreSeconds":931,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":"entry":"key":"memory-mb","value":"3818593","entry":"key":"vcores","value":"931","preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3818593,"preemptedVcoreSeconds":931,"preemptedResourceSecondsMap":"entry":"key":"memory-mb","value":"3818593","entry":"key":"vcores","value":"931","logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":"timeout":["type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1]
]

【问题讨论】：

尝试加空格：id:'.id' -> id: '.id' 该错误表明您直接将 yaml 模板文件提供给 prometheus。您应该首先从您的 yaml 模板 + data.json 生成一个配置文件，并将 生成的配置文件 提供给 prometheus。 【参考方案1】：

根据marco.m的建议，修改yaml配置文件如下：

metrics:
- name: running_applications
  type: object 
  help: Information on running applications
  path: '.apps.app[?(@.state=="RUNNING")]'
  labels:
        id: '.id'
  values:
        memory: '.allocatedMB'
        vcores: '.allocatedVCores'

【讨论】：

以上是关于yaml：使用 go 时解组错误的主要内容，如果未能解决你的问题，请参考以下文章

关于将 yaml 解组为结构的错误

json 错误，无法将对象解组为 Go 值

json：无法将字符串解组为 MyMap.map 类型的 Go 值

使用 Go 解组 PubSub 请求数据 []bytes

带有 Supabase 和 Flutter 的 Twilio 将数字解组到 Go 结构字段 SmsStatus.status

将 json 数组解组为 go struct（数组在 JSON 字符串的中间