Big Query 未成功执行

Posted 2023-03-25

【中文标题】Big Query 未成功执行

【英文标题】：Big Query is not executing successfully

【发布时间】：2020-07-15 11:11:12

【问题描述】：

我正在尝试在 google clod 平台上使用 pyhton 脚本执行大查询。 我正在尝试参数化查询中的值。我正在从 excel 中获取价值。

当我执行查询时，它不显示任何记录。当我在大查询中运行相同的查询时，它的执行没有任何问题。

from google.cloud import bigquery
import pandas as pd
from pandas import ExcelFile
from pandas import ExcelWriter

client=bigquery.Client()
#client.query(query_1)

df=pd.read_excel('16Junedata_03062020_80163767_action_03062020_80163767_2624_01.xls', sheet_name='Sheet4')
#newdf = df.fillna('business_day':0,'zone_id':0,'site_id':0,'device_id':0)
newdf = df.fillna(method="ffill")
z_id= newdf['zone_id']
d_id= newdf['device_id']
s_id= newdf['site_id']
vst= newdf['visit_start_time']

for n,zone_id in z_id.iteritems():
    for o, site_id in s_id.iteritems():
        for m,device_id in d_id.iteritems():
            for p,visit_start_time in vst.iteritems():
                query=("""select site_id,business_day,zone_id,device_id,last_modified,duration,visit_start_time
                from shopper_journey.visits
                where zone_id=@zid and device_id=@did and site_id=@sid and business_day="2020-05-19" 
                and CAST(visit_start_time as DATETIME)="2020-06-03 01:41:13"
                group by site_id,business_day,zone_id,device_id,last_modified,duration,visit_start_time order by last_modified desc limit 100;""")
                query_parameters= [
                        bigquery.ScalarQueryParameter("zid","ARRAY",zone_id),
                        bigquery.ScalarQueryParameter("did","ARRAY",device_id),
                        bigquery.ScalarQueryParameter("sid","ARRAY",site_id),
                        #bigquery.ScalarQueryParameter("st","DATETIME",visit_start_time)
                    ]
                job_config=bigquery.QueryJobConfig()
                job_config.query_parameters=query_parameters
                job=client.query(query,job_config=job_config)
                results=job.result()
                for row in results:
                    print("\t\t\t\t".format(row.duration,row.last_modified,row.visit_start_time))

为 "bigquery.ScalarQueryParameter("zid","ARRAY",zone_id)," 行提供的数据类型是否正确？ 即使我提供了“字符串”但它不起作用？

有人可以帮忙解决这个问题吗

【问题讨论】：

您可以从列表中删除尾随逗号query_parameters 并重试吗？

如果我删除逗号它给出错误

是同一个错误还是另一个错误？

我更正了我的脚本，现在它按预期工作了。

【参考方案1】：

更新了我的脚本，现在它可以正常工作了。

from google.cloud import bigquery
import pandas as pd

client = bigquery.Client()
df = pd.read_excel('16Junedata_03062020_80163767_action_03062020_80163767_2624_01.xlsx', sheet_name='Sheet4')

zoneid = df['zone_id']
siteid = df['site_id']
deviceid = df['device_id']
starttime = df['visit_start_time']
duration = df['Duration']

zid = zoneid.tolist()
sid = siteid.tolist()
did = deviceid.tolist()
st = starttime.tolist()
dt = duration.tolist()

query="""select site_id,business_day,zone_id,device_id,last_modified,duration,visit_start_time from 
shopper_journey.visits where zone_id=@zid and device_id=@tid and site_id=@sid and 
business_day="2020-06-03" and visit_start_time=@st group by site_id,business_day,zone_id,device_id,
last_modified,duration,visit_start_time order by last_modified desc limit 1; """

for i in range(len(zid)):
        job_config=bigquery.QueryJobConfig\
                        (query_parameters =[
        bigquery.ScalarQueryParameter("zid", "STRING", zid[i]),
        bigquery.ScalarQueryParameter("sid", "INT64", sid[i]),
        bigquery.ScalarQueryParameter("tid", "STRING", did[i]),
        bigquery.ScalarQueryParameter("st", "DATETIME", st[i])
                ])
        query_job=client.query(query, job_config=job_config)

        for row in query_job:
                print("\t\t\t\t\t".format(row.duration, row.last_modified, row.visit_start_time))