Google BigQuery API（Python 客户端库）> 查询数据（异步）

Posted 2023-03-24

【中文标题】Google BigQuery API（Python 客户端库）> 查询数据（异步）

【英文标题】：Google BigQuery API (Python client Library) > Querying data (asynchronous)

【发布时间】：2017-07-31 01:49:39

【问题描述】：

我正在关注Python Client Libraries for the Google BigQuery API - https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery/usage.html#jobs > 查询数据（异步）

当涉及到检索结果时，执行代码：

rows, total_count, token = query.fetch_data()  # API requet

总是返回ValueError: too many values to unpack (expected 3) （顺便说一句，我认为有一个错字，应该是results.fetch_data()！）

但是，下面的代码可以正常工作

results = job.results()
rows = results.fetch_data()
tbl = [x for x in rows]

表的所有行都在 tbl 中单次返回（作为元组列表），>225K 行！

任何人都知道为什么我会收到错误，或者文档中有什么问题吗？

我如何仍然可以批量检索结果（逐页迭代）

提前非常感谢！

【参考方案1】：

不久前，我打开this issue 要求更新文档，但从答案中可以看出，它仍然需要正式发布才能更改。

请参考code base 本身以获得更好的文档字符串（在这种情况下特别是类Iterator）：

"""Iterators for paging through API responses.
These iterators simplify the process of paging through API responses
where the response is a list of results with a ``nextPageToken``.
To make an iterator work, you'll need to provide a way to convert a JSON
item returned from the API into the object of your choice (via
``item_to_value``). You also may need to specify a custom ``items_key`` so
that a given response (containing a page of results) can be parsed into an
iterable page of the actual objects you want. You then can use this to get
**all** the results from a resource::
    >>> def item_to_value(iterator, item):
    ...     my_item = MyItemClass(iterator.client, other_arg=True)
    ...     my_item._set_properties(item)
    ...     return my_item
    ...
    >>> iterator = Iterator(..., items_key='blocks',
    ...                     item_to_value=item_to_value)
    >>> list(iterator)  # Convert to a list (consumes all values).
Or you can walk your way through items and call off the search early if
you find what you're looking for (resulting in possibly fewer
requests)::
    >>> for my_item in Iterator(...):
    ...     print(my_item.name)
    ...     if not my_item.is_valid:
    ...         break
At any point, you may check the number of items consumed by referencing the
``num_results`` property of the iterator::
    >>> my_iterator = Iterator(...)
    >>> for my_item in my_iterator:
    ...     if my_iterator.num_results >= 10:
    ...         break
When iterating, not every new item will send a request to the server.
To iterate based on each page of items (where a page corresponds to
a request)::
    >>> iterator = Iterator(...)
    >>> for page in iterator.pages:
    ...     print('=' * 20)
    ...     print('    Page number: %d' % (iterator.page_number,))
    ...     print('  Items in page: %d' % (page.num_items,))
    ...     print('     First item: %r' % (next(page),))
    ...     print('Items remaining: %d' % (page.remaining,))
    ...     print('Next page token: %s' % (iterator.next_page_token,))
    ====================
        Page number: 1
      Items in page: 1
         First item: <MyItemClass at 0x7f1d3cccf690>
    Items remaining: 0
    Next page token: eav1OzQB0OM8rLdGXOEsyQWSG
    ====================
        Page number: 2
      Items in page: 19
         First item: <MyItemClass at 0x7f1d3cccffd0>
    Items remaining: 18
    Next page token: None
To consume an entire page::
    >>> list(page)
    [
        <MyItemClass at 0x7fd64a098ad0>,
        <MyItemClass at 0x7fd64a098ed0>,
        <MyItemClass at 0x7fd64a098e90>,
    ]

【讨论】：

非常感谢伙计！这真的澄清了事情:)

【参考方案2】：

是的，您对文档的看法是正确的。有错别字-

results = job.results()

rows, total_count, token = query.fetch_data() # API requet

while True:

    do_something_with(rows)

     if token is None:

          break

    rows, total_count,token=query.fetch_data(page_token=token)       # API requeste here

对于大数据集，我们每小时查询一次以获取日常工作中的数据。