带有 jq 的 Bash 脚本不会从字符串中获取日期差异，并且在 i7 16GB RAM 上运行非常缓慢

Posted 2023-04-14

【中文标题】带有 jq 的 Bash 脚本不会从字符串中获取日期差异，并且在 i7 16GB RAM 上运行非常缓慢

【英文标题】：Bash script with jq wont get date difference from strings, and runs quite slowly on i7 16GB RAM

【发布时间】：2021-11-24 07:20:25

【问题描述】：

以下脚本中的 Exposure 列需要以 dd:hh:mm 格式查找 TradeCloseTime 和 TradeOpenTime 时间之间的差异。

此外，脚本运行速度非常慢（在 Core i7 16gb RAM 机器上，800 行 json 大约需要 4 分钟）

#!/bin/bash
echo "TradeNo, TradeOpenType, TradeCloseType, TradeOpenSource, TradeCloseSource, TradeOpenTime, TradeCloseTime, PNL, Exposure" > tradelist.csv
tradecount=$(jq -r '.performance.numberOfTrades|tonumber' D.json)
for ((i=0; i<$tradecount; i++))
do
tradeNo=$(jq -r '.trades['$i']|[.tradeNo][]|tonumber' D.json)
entrySide=$(jq -r '.trades['$i'].orders[0]|[.side][]' D.json)
exitSide=$(jq -r '.trades['$i'].orders[1]|[.side][]' D.json)
entrySource=$(jq -r '.trades['$i'].orders[0]|[.source][]' D.json)
exitSource=$(jq -r '.trades['$i'].orders[1]|[.source][]' D.json)
tradeEntryTime=$(jq -r '.trades['$i'].orders[0]|[.placedTime][]' D.json | tr -d 'Z' |  tr -s 'T' ' ')
tradeExitTime=$(jq -r '.trades['$i'].orders[1]|[.placedTime][]' D.json | tr -d 'Z' |  tr -s 'T' ' ')
profitPercentage=$(jq -r '(.trades['$i']|[.profitPercentage][0]|tonumber)*(100)' D.json)
echo $tradeNo","$entrySide","$exitSide","$entrySource","$exitSource","$tradeEntryTime","$tradeExitTime","$profitPercentage | tr -d '"' >> tradelist.csv
done

json 文件是这样的

"market":"exchange":"BINANCE_FUTURES","coinPair":"BTC_USDT","strategy":"name":"","type":"BACKTEST","candleSize":15,"lookbackDays":6,"leverageLong":1.00000000,"leverageShort":1.00000000,"strategyName":"ABC","strategyVersion":35,"runNo":"002","source":"Personal","strategyParameters":["name":"DurationInput","value":"87.0"],"openPositionStrategy":"actionTime":"CANDLE_CLOSE","maxPerSignal":1.00000000,"closePositionStrategy":"actionTime":"CANDLE_CLOSE","minProfit":"NaN","stopLossValue":0.07000000,"stopLossTrailing":true,"takeProfit":0.01290000,"takeProfitDeviation":"NaN","performance":"startTime":"2019-01-01T00:00:00Z","endTime":"2021-11-24T00:00:00Z","startAllocation":1000.00000000,"endAllocation":3478.58904150,"absoluteProfit":2478.58904150,"profitPerc":2.47858904,"buyHoldRatio":0.62426630,"buyHoldReturn":4.57228387,"numberOfTrades":744,"profitableTrades":0.67833109,"maxDrawdown":-0.20924885,"avgMonthlyProfit":0.05242718,"profitableMonths":0.70370370,"avgWinMonth":0.09889897,"avgLoseMonth":-0.05275563,"startPrice":null,"endPrice":57623.08000000,"trades":["tradeNo":0,"profit":-5.48836165,"profitPercentage":-0.00549085,"accumulatedBalance":994.51163835,"compoundProfitPerc":-0.00548836,"orders":["side":"Long","placedTime":"2019-09-16T21:15:00Z","placedAmount":0.09700000,"filledTime":"2019-09-16T21:15:00Z","filledAmount":0.09700000,"filledPrice":10300.49000000,"commissionPaid":0.39965901,"source":"SIGNAL","side":"CloseLong","placedTime":"2019-09-17T19:15:00Z","placedAmount":0.09700000,"filledTime":"2019-09-17T19:15:00Z","filledAmount":0.09700000,"filledPrice":10252.13000000,"commissionPaid":0.39778264,"source":"SIGNAL"],"tradeNo":1,"profit":-3.52735800,"profitPercentage":-0.00356403,"accumulatedBalance":990.98428035,"compoundProfitPerc":-0.00901572,"orders":["side":"Long","placedTime":"2019-09-19T06:00:00Z","placedAmount":0.10000000,"filledTime":"2019-09-19T06:00:00Z","filledAmount":0.10000000,"filledPrice":9893.16000000,"commissionPaid":0.39572640,"source":"SIGNAL","side":"CloseLong","placedTime":"2019-09-19T06:15:00Z","placedAmount":0.10000000,"filledTime":"2019-09-19T06:15:00Z","filledAmount":0.10000000,"filledPrice":9865.79000000,"commissionPaid":0.39463160,"source":"SIGNAL"],"tradeNo":2,"profit":-5.04965308,"profitPercentage":-0.00511770,"accumulatedBalance":985.93462727,"compoundProfitPerc":-0.01406537,"orders":["side":"Long","placedTime":"2019-09-25T10:15:00Z","placedAmount":0.11700000,"filledTime":"2019-09-25T10:15:00Z","filledAmount":0.11700000,"filledPrice":8430.00000000,"commissionPaid":0.39452400,"source":"SIGNAL","side":"CloseLong","placedTime":"2019-09-25T10:30:00Z","placedAmount":0.11700000,"filledTime":"2019-09-25T10:30:00Z","filledAmount":0.11700000,"filledPrice":8393.57000000,"commissionPaid":0.39281908,"source":"SIGNAL"]

【问题讨论】：

您正在运行 800*8 + 1 个 jq 实例。难怪它很慢。看起来您正在输出 CSV 数据？您可以通过对所有数据调用 jq 来做到这一点。在这里搜索；有很多例子。

是的，输出到 csv。我仍然需要循环 800 次。唯一的优化是避免8个内部循环？或者，考虑到 json 结构，是否可以完全避免 800 个循环？

@Shawn 您能否建议是否需要将问题分解为 2 个问题以解决日期/字符串问题和性能？

正如@Shawn 所指出的，jq 具有循环功能，这应该可以很容易地调用 jq 一次。如果您无法弄清楚其中任何一个部分，最好将 Q 分成两部分，但请注意每个 SO Q 都应遵循 minimal reproducible example 指南。

【参考方案1】：

您可以通过一个jq 呼叫完成所有操作（提取、转换和格式化）：

#!/bin/sh

echo 'TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure'

query='
  .trades[]
  | [
    .tradeNo,
    .orders[0].side,
    .orders[1].side,
    .orders[0].source,
    .orders[1].source,
    (.orders[0].placedTime | fromdate | strftime("%Y-%m-%d %H:%M:%S")),
    (.orders[1].placedTime | fromdate | strftime("%Y-%m-%d %H:%M:%S")),
    .profitPercentage * 100,
    (
      (.orders[1].placedTime | fromdate) - (.orders[0].placedTime | fromdate)
      | (. / 86400 | floor | tostring) + (. % 86400 | strftime(":%H:%M"))
    )
  ]
  |@csv
'

jq -r "$query" < D.json > tradelist.csv

JSON 示例（清除所有不相关的键）：

 
  "trades": [
    
      "tradeNo": 0,
      "profitPercentage": -0.00549085,
      "orders": [
        
          "side": "Long",
          "placedTime": "2018-12-16T21:34:46Z",
          "source": "SIGNAL"
        ,
        
          "side": "CloseLong",
          "placedTime": "2019-09-17T19:15:00Z",
          "source": "SIGNAL"
        
      ]
    
  ]

输出：

TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure
0,"Long","CloseLong","SIGNAL","SIGNAL","2018-12-16 21:34:46","2019-09-17 20:15:00",-0.549085,"274:22:40"

如果您想去掉 jq 在生成 CSV 时添加的双引号（这是完全有效的，但您需要真正的解析器来读取 CSV），那么您可以将 @csv 替换为 @tsv并使用tr '\t' ',' 对输出进行后处理，如下所示：

query='
  ...
  |@tsv
'

jq -r "$query" < D.json | tr '\t' ',' > tradelist.csv

你会得到：

TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure
0,Long,CloseLong,SIGNAL,SIGNAL,2018-12-16 21:34:46,2019-09-17 20:15:00,-0.549085,274:22:40

注意：这种去除CSV中"的方法只有在没有\n\t\r\,时才准确或" 输入数据中的字符。

【讨论】：

这非常有效，并且改进了下游更大的输出文件。谢谢！

【参考方案2】：

关于主要问题（关于计算时间差异），您很幸运，因为 jq 提供了内置函数 fromdateiso8601 用于将 ISO 时间转换为“ 自 Unix 纪元 (1970-01-01T00:00:00Z) 以来的秒数”。

使用您的 JSON 示例，

.trades[]
| [ .orders[1].placedTime, .orders[0].placedTime]
| map(fromdateiso8601)
| .[0] - .[1]

产生三个差异：

79200
900
900

这是一个将秒转换为“hh:mm:ss”格式的函数：

def hhmmss:
  def l: tostring | if length < 2 then "0\(.)" else . end;
  (. % 60) as $ss
  | ((. / 60) | floor) as $mm
  | (($mm / 60) | floor) as $hh
  | ($mm % 60) as $mm
  | [$hh, $mm, $ss] | map(l) | join(":");

【讨论】：

jq: error (at D.json:0): Cannot iterate over string ("2019-09-16...)

@SamP - 我使用 jq 1.5、jq 1.6、jq 的“主”版本和 gojq（jq 的 Go 实现）进行了检查。所以看起来你做错了什么。

在将Exposure 转换为dd:hh:mm 时，我的答案中没有想到floor

【参考方案3】：

我更喜欢使用“入口”和“出口”JSON 的中间结构。这有助于调试jq 命令。格式化是为了可读性而不是性能：

#!/usr/bin/env bash
echo "TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure" > tradelist.csv
jq -r '
  .trades[]
  |tradeNo,
    profitPercentage,
    entry:.orders[0],
    exit:.orders[1],
    entryTS:.orders[0].placedTime|fromdate,
    exitTS:.orders[1].placedTime|fromdate
  |[.tradeNo,
    .entry.side,
    .exit.side,
    .entry.source,
    .exit.source,
    (.entry.placedTime|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%Y-%m-%d %H:%M:%S")),
    (.exit.placedTime|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%Y-%m-%d %H:%M:%S")),
    (.profitPercentage*100),
    (.exitTS-.entryTS|todate|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%d:%H:%M"))]|@csv
  ' D.json | tr -d '"' >> tradelist.csv

警告：此格式假定曝光时间不到 1 个月。祝你好运！

【讨论】：

不错的把戏。缩进你的jq 查询会更有帮助;-)