logstash：wso api manager 日志到弹性

Posted 2023-02-19

【中文标题】logstash：wso api manager 日志到弹性

【英文标题】：logstash: wso api manager logs to elastic

【发布时间】：2022-01-15 20:35:00

【问题描述】：

我的日志如下所示，日志中的每个参数都是静态的。我尝试通过logstash发送弹性并使用Grok模式。但我看不到常规日志。我不擅长 grok。我怎样才能编写合适的 grok 代码？ 在输出中我不想看到年份、蒙特信息和空的 GREEDYDATA 部分。

[2021-12-05 20:57:40,513]  INFO - LogCounterMetric Metric Name : apim:response Metric Value: apiName=ApiExp, proxyResponseCode=200, errorType=null, destination=http://1.1.1.1:11/service.asmx, apiCreatorTenantDomain=carbon.super, platform=Other, apiMethod=POST, apiVersion=1.0, gatewayType=SYNAPSE, apiCreator=admin, responseCacheHit=false, backendLatency=229, correlationId=244a4509-52a6-4f37-921d-5d2fc6e40ca3, requestMediationLatency=365, keyType=SANDBOX, apiId=32154b48-70dc5-4cx5-9dgd-7er32e31bacc, applicationName=DefaultApplication, targetResponseCode=200, requestTimestamp=2021-12-05T20:57:39.823Z, applicationOwner=admin, userAgent=Other, eventType=response, apiResourceTemplate=/*, responseLatency=596, regionId=default, responseMediationLatency=2, userIp=1.1.1.1, applicationId=37d453cc-4425-fd31-gf9c-fcdfae89bbd, apiType=SOAP

这是我写的 grok 代码。

\[%TIMESTAMP_ISO8601:timestamp\]%SPACE%LOGLEVEL:level - LogCounterMetric Metric Name : apim:response Metric Value: %GREEDYDATAapiName=%GREEDYDATA:apiName, %GREEDYDATAproxyResponseCode=%GREEDYDATA:proxyResponseCode, %GREEDYDATAerrorType=%GREEDYDATA:errorType, %GREEDYDATAdestination=%GREEDYDATA:destination, %GREEDYDATAapiCreatorTenantDomain=%GREEDYDATA:apiCreatorTenantDomain, %GREEDYDATAplatform=%GREEDYDATA:platform, %GREEDYDATAapiMethod=%GREEDYDATA:apiMethod, %GREEDYDATAapiVersion=%GREEDYDATA:apiVersion, %GREEDYDATAgatewayType=%GREEDYDATA:gatewayType, %GREEDYDATAapiCreator=%GREEDYDATA:apiCreator, %GREEDYDATAresponseCacheHit=%GREEDYDATA:responseCacheHit, %GREEDYDATAbackendLatency=%GREEDYDATA:backendLatency, %GREEDYDATAcorrelationId=%GREEDYDATA:correlationId, %GREEDYDATArequestMediationLatency=%GREEDYDATA:requestMediationLatency, %GREEDYDATAkeyType=%GREEDYDATA:keyType, %GREEDYDATAapiId=%GREEDYDATA:apiId, %GREEDYDATAapplicationName=%GREEDYDATA:applicationName, %GREEDYDATAtargetResponseCode=%GREEDYDATA:targetResponseCode, %GREEDYDATArequestTimestamp=%GREEDYDATA:requestTimestamp, %GREEDYDATAapplicationOwner=%GREEDYDATA:applicationOwner, %GREEDYDATAuserAgent=%GREEDYDATA:userAgent, %GREEDYDATAeventType=%GREEDYDATA:eventType, %GREEDYDATAapiResourceTemplate=%GREEDYDATA:apiResourceTemplate, %GREEDYDATAresponseLatency=%GREEDYDATA:responseLatency, %GREEDYDATAregionId=%GREEDYDATA:regionId, %GREEDYDATAresponseMediationLatency=%GREEDYDATA:responseMediationLatency, %GREEDYDATAuserIp=%GREEDYDATA:userIp, %GREEDYDATAapplicationId=%GREEDYDATA:applicationId, %GREEDYDATAapiType=%GREEDYDATA:apiType

--在 grok 调试器输出中

  "timestamp": [
    [
      "2021-12-05 20:57:40,513"
    ]
  ],
  "YEAR": [
    [
      "2021"
    ]
  ],
  "MONTHNUM": [
    [
      "12"
    ]
  ],
  "MONTHDAY": [
    [
      "05"
    ]
  ],
  "HOUR": [
    [
      "20",
      null
    ]
  ],
  "MINUTE": [
    [
      "57",
      null
    ]
  ],
  "SECOND": [
    [
      "40,513"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "SPACE": [
    [
      "  "
    ]
  ],
  "level": [
    [
      "INFO"
    ]
  ],
  "GREEDYDATA": [
    [
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      ""
    ]
  ],
  "apiName": [
    [
      "ApiExp"
    ]
  ],
  "proxyResponseCode": [
    [
      "200"
    ]
  ],
  "errorType": [
    [
      "null"
    ]
  ],
  "destination": [
    [
      "http://1.1.1.1:11/service.asmx"
    ]
  ],
  "apiCreatorTenantDomain": [
    [
      "carbon.super"
    ]
  ],
  "platform": [
    [
      "Other"
    ]
  ],
  "apiMethod": [
    [
      "POST"
    ]
  ],
  "apiVersion": [
    [
      "1.0"
    ]
  ],
  "gatewayType": [
    [
      "SYNAPSE"
    ]
  ],
  "apiCreator": [
    [
      "admin"
    ]
  ],
  "responseCacheHit": [
    [
      "false"
    ]
  ],
  "backendLatency": [
    [
      "229"
    ]
  ],
  "correlationId": [
    [
      "244a4509-52a6-4f37-921d-5d2fc6e40ca3, "
    ]
  ],
  "requestMediationLatency": [
    [
      "365"
    ]
  ],
  "keyType": [
    [
      "SANDBOX"
    ]
  ],
  "apiId": [
    [
      "32154b48-70dc5-4cx5-9dgd-7er32e31bacc"
    ]
  ],
  "applicationName": [
    [
      "DefaultApplication"
    ]
  ],
  "targetResponseCode": [
    [
      "200"
    ]
  ],
  "requestTimestamp": [
    [
      "2021-12-05T20:57:39.823Z"
    ]
  ],
  "applicationOwner": [
    [
      "admin"
    ]
  ],
  "userAgent": [
    [
      "Other"
    ]
  ],
  "eventType": [
    [
      "response"
    ]
  ],
  "apiResourceTemplate": [
    [
      "/*"
    ]
  ],
  "responseLatency": [
    [
      "596"
    ]
  ],
  "regionId": [
    [
      "default"
    ]
  ],
  "responseMediationLatency": [
    [
      "2"
    ]
  ],
  "userIp": [
    [
      "1.1.1.1"
    ]
  ],
  "applicationId": [
    [
      "37d453cc-4425-fd31-gf9c-fcdfae89bbd"
    ]
  ],
  "apiType": [
    [
      "SOAP"
    ]
  ]

【问题讨论】：

尝试删除 []，你的输出看起来不像 JSON

【参考方案1】：

您可以使用 drop filter 的选项 remove_field 删除输出中不需要的特定字段。

更多细节：

https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html#plugins-filters-drop-remove_field