jq处理JSON数据

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了jq处理JSON数据相关的知识,希望对你有一定的参考价值。

 

jq 允许你直接在命令行下对 JSON 进行操作,包括分片、过滤、转换等等。
让我们通过几个例子来说明 jq 的功能:
一、输出格式化,漂亮的打印效果
如果我们用文本编辑器打开 JSON,有时候可能看起来会一团糟,但是通过 jq 的 .(点)过滤器就可以立马让 JSON 的格式规整起来。
1.用文本编辑器打开后的样子

2.用 jq 显示的结果

复制代码 代码如下:
% jq . soundtag.json


技术分享
二、快速查询JSON数据
利用 jq 能够以 key 作为关键字来对 JSON 作出快速查询, 例如:

复制代码 代码如下:
% jq .cn soundtag.json


这将仅仅显示 cn 键对应的值。
jq 的键查询也支持链式调用,如:

复制代码 代码如下:
% jq .cn[0].pNum soundtag.json


三、jq的管道操作
熟悉命令行的朋友可能都知道 | (管道)是一个非常强大的 武器。幸运的是,jq 也提供了对管道的支持。

复制代码 代码如下:
% jq ‘.cn[] | { pNum }‘ soundTag.json


在这里,我们使用管道过滤并构造出 pNum 对象。
四、总结
如果你需要在命令行下处理 JSON,我强烈推荐 jq。jq 不仅 能够满足一般性的常见需求,更包含运算、内置函数、条件比较、 变量声明、自定函数等强大功能。对此感兴趣的朋友,不妨通过 jq 的官方手册 进行学习。

 

jq处理JSON数据

2、高亮并按属性排序显示

用 jq ‘.‘即可:

curl https://api.github.com/repos/stedolan/jq/commits?per_page=5‘ | jq .

结果:

[
  {
    "parents": [
      {
        "html_url": "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
        "url": "https://api.github.com/repos/stedolan/jq/commits/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
        "sha": "54b9c9bdb225af5d886466d72f47eafc51acb4f7"
      },
      {
        "html_url": "https://github.com/stedolan/jq/commit/8b1b503609c161fea4b003a7179b3fbb2dd4345a",
        "url": "https://api.github.com/repos/stedolan/jq/commits/8b1b503609c161fea4b003a7179b3fbb2dd4345a",
        "sha": "8b1b503609c161fea4b003a7179b3fbb2dd4345a"
      }
    ],
    "committer": {
      "type": "User",
      "received_events_url": "https://api.github.com/users/stedolan/received_events",
      "events_url": "https://api.github.com/users/stedolan/events{/privacy}",
      "repos_url": "https://api.github.com/users/stedolan/repos",
      "organizations_url": "https://api.github.com/users/stedolan/orgs",
...

这里的评论内容比较多,我们现在想拿第一个评论。

3、获取数组某一项

curl https://api.github.com/repos/stedolan/jq/commits?per_page=5‘ | jq .[0]

结果:

技术分享
{
  "parents": [
    {
      "html_url": "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
      "url": "https://api.github.com/repos/stedolan/jq/commits/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
      "sha": "54b9c9bdb225af5d886466d72f47eafc51acb4f7"
    },
    {
      "html_url": "https://github.com/stedolan/jq/commit/8b1b503609c161fea4b003a7179b3fbb2dd4345a",
      "url": "https://api.github.com/repos/stedolan/jq/commits/8b1b503609c161fea4b003a7179b3fbb2dd4345a",
      "sha": "8b1b503609c161fea4b003a7179b3fbb2dd4345a"
    }
  ],
  "committer": {
    "type": "User",
    "received_events_url": "https://api.github.com/users/stedolan/received_events",
    "events_url": "https://api.github.com/users/stedolan/events{/privacy}",
    "repos_url": "https://api.github.com/users/stedolan/repos",
    "organizations_url": "https://api.github.com/users/stedolan/orgs",
    "subscriptions_url": "https://api.github.com/users/stedolan/subscriptions",
    "starred_url": "https://api.github.com/users/stedolan/starred{/owner}{/repo}",
    "gists_url": "https://api.github.com/users/stedolan/gists{/gist_id}",
    "login": "stedolan",
    "id": 79765,
    "avatar_url": "https://1.gravatar.com/avatar/31de909d8e55dd07ed782d92ece59842?d=https%3A%2F%2Fidenticons.github.com%2Ffc5b6765b1c9cfaecea48ae71df4d279.png",
    "gravatar_id": "31de909d8e55dd07ed782d92ece59842",
    "url": "https://api.github.com/users/stedolan",
    "html_url": "https://github.com/stedolan",
    "followers_url": "https://api.github.com/users/stedolan/followers",
    "following_url": "https://api.github.com/users/stedolan/following{/other_user}"
  },
  "author": {
    "type": "User",
    "received_events_url": "https://api.github.com/users/stedolan/received_events",
    "events_url": "https://api.github.com/users/stedolan/events{/privacy}",
    "repos_url": "https://api.github.com/users/stedolan/repos",
    "organizations_url": "https://api.github.com/users/stedolan/orgs",
    "subscriptions_url": "https://api.github.com/users/stedolan/subscriptions",
    "starred_url": "https://api.github.com/users/stedolan/starred{/owner}{/repo}",
    "gists_url": "https://api.github.com/users/stedolan/gists{/gist_id}",
    "login": "stedolan",
    "id": 79765,
    "avatar_url": "https://1.gravatar.com/avatar/31de909d8e55dd07ed782d92ece59842?d=https%3A%2F%2Fidenticons.github.com%2Ffc5b6765b1c9cfaecea48ae71df4d279.png",
    "gravatar_id": "31de909d8e55dd07ed782d92ece59842",
    "url": "https://api.github.com/users/stedolan",
    "html_url": "https://github.com/stedolan",
    "followers_url": "https://api.github.com/users/stedolan/followers",
    "following_url": "https://api.github.com/users/stedolan/following{/other_user}"
  },
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f/comments",
  "html_url": "https://github.com/stedolan/jq/commit/d25341478381063d1c76e81b3a52e0592a7c997f",
  "url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f",
  "commit": {
    "comment_count": 0,
    "url": "https://api.github.com/repos/stedolan/jq/git/commits/d25341478381063d1c76e81b3a52e0592a7c997f",
    "tree": {
      "url": "https://api.github.com/repos/stedolan/jq/git/trees/6ab697a8dfb5a96e124666bf6d6213822599fb40",
      "sha": "6ab697a8dfb5a96e124666bf6d6213822599fb40"
    },
    "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161",
    "committer": {
      "date": "2013-06-22T16:30:59Z",
      "email": "[email protected]",
      "name": "Stephen Dolan"
    },
    "author": {
      "date": "2013-06-22T16:30:59Z",
      "email": "[email protected]",
      "name": "Stephen Dolan"
    }
  },
  "sha": "d25341478381063d1c76e81b3a52e0592a7c997f"
}
View Code

可以看到,我们已经拿到了一条完整的评论内容。但我们真正关心的只是评论内容和用户名,下面来获取这两项内容。

4、自定义格式输出某一项

jq .[0] | {message: .commit.message, name: .commit.committer.name}

结果:

{
  "name": "Stephen Dolan",
  "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
}

我们可以看到,已经拿到了想要的内容,并且已经按我们自己定义的格式显示了。

这里 | 后面的内容是以前面的内容为输入的, .commit  中的 . 就是指 .[0] 中的内容。

5、自定义格式输出多项

jq .[] | {message: .commit.message, name: .commit.committer.name}

结果:

{
  "name": "Stephen Dolan",
  "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
}
{
  "name": "Stephen Dolan",
  "message": "Reject all overlong UTF8 sequences."
}
{
  "name": "Stephen Dolan",
  "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences."
}
{
  "name": "Stephen Dolan",
  "message": "Fix example in manual for `floor`. See #155."
}
{
  "name": "Nicolas Williams",
  "message": "Document floor"
}

这里 .[] 获取的是数组中的所有项。

我们看到,结果是一个个独立的JSON对象,如何把结果组合成一个数组呢?

6、以数组形式自定义输出多项

jq [.[] | {message: .commit.message, name: .commit.committer.name}]

结果:

[
  {
    "name": "Stephen Dolan",
    "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
  },
  {
    "name": "Stephen Dolan",
    "message": "Reject all overlong UTF8 sequences."
  },
  {
    "name": "Stephen Dolan",
    "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences."
  },
  {
    "name": "Stephen Dolan",
    "message": "Fix example in manual for `floor`. See #155."
  },
  {
    "name": "Nicolas Williams",
    "message": "Document floor"
  }
]

我们可以看到,只要在上一步的命令中内容的两端加个中括号即可。

最后,我们如果想获取每个评论的引用评论的url(在parents节点中,有一个或多个)呢?

7、获取其他内容

jq [.[] | {message: .commit.message, name: .commit.committer.name, parents: [.parents[].html_url]}]

结果:

[
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
      "https://github.com/stedolan/jq/commit/8b1b503609c161fea4b003a7179b3fbb2dd4345a"
    ],
    "name": "Stephen Dolan",
    "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
  },
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/ff48bd6ec538b01d1057be8e93b94eef6914e9ef"
    ],
    "name": "Stephen Dolan",
    "message": "Reject all overlong UTF8 sequences."
  },
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7"
    ],
    "name": "Stephen Dolan",
    "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences."
  },
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/3dcdc582ea993afea3f5503a78a77675967ecdfa"
    ],
    "name": "Stephen Dolan",
    "message": "Fix example in manual for `floor`. See #155."
  },
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/7c4171d414f647ab08bcd20c76a4d8ed68d9c602"
    ],
    "name": "Nicolas Williams",
    "message": "Document floor"
  }
]

这里用 .parents[].html_url 获取当前项的 parents 节点中的所有项的 html_url 属性的内容,然后两边加个中括号组装成数组输出。

怎么样,经过这个例子可以看出,用jq处理JSON数据还是很方便强大的吧。

8、注意点

jq能处理的需要是严格的JSON格式数据,JSON对象和JSON字符串是不行的,如下面的两种格式数据jq是不能处理的:

json对象:

{
    a: 1,
    b: {
        c: "abc"
    }
}

json字符串:

‘{"a":1,"b":{"c":"abc"}}‘

正确的JSON格式:

{
    "a": 1,
    "b": {
        "c": "abc"
    }
}

关于什么是JSON格式数据,请参考: http://www.json.org/

















以上是关于jq处理JSON数据的主要内容,如果未能解决你的问题,请参考以下文章

jq:给力的命令行 JSON 处理工具

jq-win64.exe:从 Windows 批处理文件中的 JSON 文件解析数据

JSON代码格式化 进行查询筛选

优雅的查看json文件

[Linux] jq:命令行JSON处理工具

使用 jq 专门处理空 JSON 数组