Firebase 大查询 - 如何从自定义事件表中检索数据
Posted
技术标签:
【中文标题】Firebase 大查询 - 如何从自定义事件表中检索数据【英文标题】：Firebase Big query - How to retrieve the data from Custom events table 【发布时间】：2017-07-21 12:00:26 【问题描述】：我正在开发连接 Firebase 数据库的应用程序，我的应用程序基本上是一个市场应用程序。我已将所有用户搜索历史记录到 Firebase Analytics 作为自定义事件，我已将其链接到大查询以进行分析或建议。
这是我的自定义事件日志示例：
FirebaseAnalytics mFireBaseAnalytics = FirebaseAnalytics.getInstance(context);
Bundle params = new Bundle();
params.putString("SEARCHER_ID", "xxxx_id");
params.putString("KEYWORDS", "Shoe");
params.putDouble("LATITUDE", 12.00);
params.putDouble("LONGITUDE", 80.00);
mFireBaseAnalytics.logEvent("SEARCH_PRODUCT", params);
此数据将保存在 app_events 表中的 Biq 查询中。
从这个事件中，我必须查询我当地流行的关键字搜索列表，例如 5 公里范围内。我不确定如何实现这一目标。因为请看下面的结果，它实际上是如何存储在 Big 查询中的。
"event_dim": [
"date": "20170718",
"name": "SEARCH_PRODUCT",
"params": [
"key": "LATITUDE",
"value":
"string_value": null,
"int_value": null,
"float_value": null,
"double_value": "12.9632747"
,
"key": "SEARCHER_ID",
"value":
"string_value": "-KkphUBI3_v3kdGq6qTa",
"int_value": null,
"float_value": null,
"double_value": null
,
"key": "LONGITUDE",
"value":
"string_value": null,
"int_value": null,
"float_value": null,
"double_value": "80.2131347"
,
"key": "KEYWORDS",
"value":
"string_value": "shoe cloths",
"int_value": null,
"float_value": null,
"double_value": null
,
],
]
对于这种格式，我使用标准 SQL 查询进行了如下查询
SELECT
SQRT(POW(69.1*( (
SELECT
params.value.double_value
FROM
x.params
WHERE
params.key = 'LATITUDE') - 12.00), 2) + POW(69.1*(80.098 - (SELECT
params.value.double_value
FROM
x.params
WHERE
params.key = 'LONGITUDE') ) * COS((SELECT
params.value.double_value
FROM
x.params
WHERE
params.key = 'LATITUDE') / 57.3), 2)) AS distance,
(
SELECT
params.value.string_value
FROM
x.params
WHERE
params.key = 'KEYWORDS') AS KEYWORDS,
(
SELECT
params.value.string_value
FROM
x.params
WHERE
params.key = 'SEARCHER_ID') AS SEARCHER_ID,
(
SELECT
params.value.string_value
FROM
x.params
WHERE
params.key = 'LATITUDE') AS LATITUDE,
(
SELECT
params.value.string_value
FROM
x.params
WHERE
params.key = 'LONGITUDE') AS LONGITUDE
FROM
`app_name.app_events_*`,
UNNEST(event_dim) AS x
WHERE
x.name = 'SEARCH_PRODUCT'
AND (
SELECT
params.value.string_value As SearchId
FROM
x.params
WHERE
params.key = 'SEARCHER_ID' ) = '-xx_id'
通过此查询，我面临 2 个问题。 1) 我将距离设为 NULL 2) 我不确定如何在 5 KM 范围内过滤此结果。
帮我解决这个问题。
PS： 我有另一个临时 Biqquery 表，我直接将数据插入到该表中，然后我可以使用以下查询获得预期的结果。
SELECT
SQRT(POW(111.1*( MAX(Lattitude) - 12.9632691), 2) + POW(111.1*(80.2140816 - MAX(Longitude) ) * COS( MAX(Lattitude) / 57.3), 2)) AS distance,
MAX(SearchKeyword) as SearchKeyword,
SearcherArea,
COUNT( SearcherArea) AS totalCount
FROM
`HNF.LastFewHoursData`
WHERE
SearchKeyword LIKE '%%'
GROUP BY
SearcherArea
HAVING
distance <= 5
ORDER BY
totalCount DESC
LIMIT
5
所以我只有查询由 Firebase Analytics 更新的事件表的问题。
【问题讨论】：
【参考方案1】：此查询可能对您有用：
CREATE TEMP FUNCTION distance(lat1 FLOAT64, lat2 FLOAT64, lon1 FLOAT64, lon2 FLOAT64) AS((
WITH data AS(
SELECT POW(SIN((ACOS(-1) / 180 * (lat1 -lat2)) / 2), 2) + COS(ACOS(-1) / 180 * (lat1)) * COS(ACOS(-1) / 180 * (lat2)) * POW(SIN((ACOS(-1) / 180 * (lon1 -lon2)) / 2), 2) a
)
SELECT 6371 * 2 * ATAN2(SQRT((SELECT a FROM data)), SQRT(1 - (SELECT a FROM data)))
));
WITH data AS(
SELECT ARRAY<STRUCT<name STRING, params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64> > > > > [STRUCT("SEARCH_PRODUCT" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.9632747 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 80.2131347 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-KkphUBI3_v3kdGq6qTa" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("shoes cloths" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params),
STRUCT("SEARCH_RANDOM" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.9632747 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 80.2131347 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-KkphUBI3_v3kdGq6qTa" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("shoes cloths" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params),
STRUCT("SEARCH_PRODUCT" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.9632747 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 80.2131347 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-xx_id" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("shoes cloths" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params)] event_dim UNION ALL
SELECT ARRAY<STRUCT<name STRING, params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64> > > > > [STRUCT("SEARCH_PRODUCT" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.9632747 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 80.2131347 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-KkphUBI3_v3kdGq6qTa" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("shoes cloths" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params),
STRUCT("SEARCH_RANDOM" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.9632747 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 80.2131347 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-KkphUBI3_v3kdGq6qTa" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("shoes cloths" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params),
STRUCT("SEARCH_PRODUCT" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.9632747 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 80.2131347 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-xxx_id" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("shoes cloths" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params)] event_dim UNION ALL
SELECT ARRAY<STRUCT<name STRING, params ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64> > > > > [STRUCT("SEARCH_PRODUCT" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.01 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.01 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-xx_id" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("pants" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params),
STRUCT("SEARCH_PRODUCT" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.01 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.01 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-xx_id" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("shoes cloths" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params),
STRUCT("SEARCH_PRODUCT" AS name, [STRUCT("LATITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.01 AS double_value) AS value), STRUCT("LONGITUDE" AS key, STRUCT("" as string_value, 0 AS int_value, 0.0 AS float_value, 12.01 AS double_value) AS value), STRUCT("SEARCHER_ID" AS key, STRUCT("-xx_id" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value), STRUCT("KEYWORDS" AS key, STRUCT("shoes cloths" as string_value, 0 AS int_value, 0.0 AS float_value, 0.0 AS double_value) AS value)] AS params)] event_dim
)
SELECT
keyword,
count(1) freq
FROM(
SELECT
ARRAY(SELECT AS STRUCT (SELECT value.string_value FROM UNNEST(params) WHERE key = 'KEYWORDS' AND EXISTS(SELECT 1 FROM UNNEST(params) WHERE key = 'SEARCHER_ID' and value.string_value = '-xx_id')) AS keyword, (SELECT value.double_value FROM UNNEST(params) WHERE key = 'LATITUDE' AND EXISTS(SELECT 1 FROM UNNEST(params) WHERE key = 'SEARCHER_ID' and value.string_value = '-xx_id')) latitude, (SELECT value.double_value FROM UNNEST(params) WHERE key = 'LONGITUDE' AND EXISTS(SELECT 1 FROM UNNEST(params) WHERE key = 'SEARCHER_ID' and value.string_value = '-xx_id')) longitude FROM UNNEST(event_dim) WHERE name = 'SEARCH_PRODUCT') data
FROM data
WHERE EXISTS(select 1 FROM UNNEST(event_dim) WHERE (SELECT COUNTIF(value.string_value = '-xx_id') FROM UNNEST(params) WHERE key = 'SEARCHER_ID' ) > 0)
),
UNNEST(data)
WHERE keyword IS NOT NULL AND latitude IS NOT NULL AND longitude IS NOT NULL
AND distance(latitude, X, longitude, Y) < 5 -- change X and Y to your desired location
GROUP BY keyword
ORDER BY freq DESC
我做了什么：
首先我们定义了函数distance，它计算两个空间点之间的距离（以度为单位，例如，（纬度12°，经度13°））。
data 只是对您在 Firebase 库中的相同数据的模拟。运行查询时，只需将 data 替换为您的实际 table。
最后，查询本身首先过滤掉名称不是SEARCH_PRODUCT 的事件，同时使用关键字、它们的纬度和经度构建ARRAY（还有一个过滤器可以删除不等于SEARCHER_IDs 的-xx_id )。
之后，只有一个 select 和一个 WHERE 子句，用于仅选择纬度和经度小于您要查询的位置的关键字（我在查询中将其称为 X 和 Y，您可以更改它对于您当前的位置，例如 12.09°、39.01°）。
让我知道这是否适合你。
【讨论】：
不幸的是，此查询未在 Firebase 上运行，我收到错误 X 未定义，我正在尝试更正它，但由于我无法理解查询，因此我很难编辑，而且查询看起来非常大，感谢您的投入，如果您能提供简单易行的解决方案，那就太好了。所以基本上我的主要问题是我在 AS 中分配的值（SELECT params.value.string_value FROM x.params WHERE params.key = 'KEYWORDS'）AS KEYWORDS 关键字没有直接用于 Where 条件或有条件，你知道吗原因是什么？ 您应该将X和Y的值更改为您要分析的纬度和经度。例如，如果您位于纬度 10° 和经度 20°，则替换 X=10 和 Y=20，查询将返回 X,Y (10,20) 半径 5 公里范围内的关键字频率。以上是关于Firebase 大查询 - 如何从自定义事件表中检索数据的主要内容，如果未能解决你的问题，请参考以下文章