使用 ClickHouse 提取和求和字符串内的子字段的值
Posted
技术标签:
【中文标题】使用 ClickHouse 提取和求和字符串内的子字段的值【英文标题】:Extract and sum values with subfields inside string using ClickHouse 【发布时间】:2021-02-20 12:41:18 【问题描述】:我有一个 ClickHouse 数据库,其中包含一个包含两个字段(pagePath 字符串、pageviews int)的简单表
我希望每个过滤器 + 值的总和访问量,以了解过滤器中用户的综合浏览量(最常用的过滤器)。过滤器用逗号分隔
示例数据
pagePath | pageviews |
---|---|
/url1/filter1.value1 | 1 |
/url1/filter1.value2 | 2 |
/url1/filter1.value3,filter1.value2 | 3 |
/url1/filter1.value4,filter3.value2 | 4 |
/url1/filter1.value5,filter3.value2,filter1.value2 | 5 |
/url1/filter2.value1,filter3.value2,filter1.value2 | 6 |
/url1/filter2.value2 | 7 |
/url-2/filter2.value3 | 8 |
/url-2/filter2.value4 | 9 |
/url-11/filter3.value1 | 10 |
/url-21/filter3.value1 | 11 |
/url1/filter3.value2 | 12 |
/url1/filter3.value3 | 13 |
/url1/filter3.value4 | 14 |
create table T (pagePath String , pageviews Int64) Engine=Memory;
insert into T values ('/url1/filter1.value1',1);
insert into T values ('/url1/filter1.value2',2);
insert into T values ('/url1/filter1.value3,filter1.value2',3);
insert into T values ('/url1/filter1.value4,filter3.value2',4);
insert into T values ('/url1/filter1.value5,filter3.value2,filter1.value2',5);
insert into T values ('/url1/filter2.value1,filter3.value2,filter1.value2',6);
insert into T values ('/url1/filter2.value2',7);
insert into T values ('/url-2/filter2.value3',8);
insert into T values ('/url-2/filter2.value4',9);
insert into T values ('/url-11/filter3.value1',10);
insert into T values ('/url-21/filter3.value1',11);
insert into T values ('/url1/filter3.value2',12);
insert into T values ('/url1/filter3.value3',13);
insert into T values ('/url1/filter3.value4',14);
我想获得 每个过滤器 + 值的浏览量总和
Filters | pageviews |
---|---|
filter1.value1 | 1 |
filter1.value2 | 16 (2 + 3 +5 +6) |
filter1.value3 | 3 |
filter1.value4 | 4 |
filter1.value5 | 5 |
filter2.value1 | 6 |
filter2.value2 | 7 |
filter2.value3 | 8 |
filter2.value4 | 9 |
filter3.value1 | 10 |
filter3.value1 | 11 |
filter3.value2 | 12 (4+5+6+12) |
filter3.value3 | 13 |
filter3.value4 | 14 |
我还想获得 每个过滤器的浏览量总和(没有值)
Filters | pageviews |
---|---|
filter1 | 15 (1, 2, 3, 4, 5) |
filter2 | 30 (6+ 7+ 8+ 9) |
filter1 | 60 ( 10, 11, 12, 13, 14) |
我试试
select
arrJoinFilters,
sum (PV) as totales
from
(
SELECT
arrJoinFilters,
splitByChar(',',replaceRegexpAll(pagePath,'^/url.*/(.*\..*)$','\\1')) arrFilter,
pageviews as PV
FROM
Table ARRAY JOIN arrFilter AS arrJoinFilters
GROUP by arrFilter,PV,arrJoinFilters
)
group by
arrJoinFilters
order by arrJoinFilters
但是,我认为有一些错误,我没有得到想要的第二个结果
谢谢!
【问题讨论】:
【参考方案1】:4+5+6+12 = 27
SELECT
arrayJoin(splitByChar(',',replaceRegexpAll(pagePath,'^/url.*/(.*\..*)$','\\1'))) f,
sum(pageviews) as PV
FROM T
GROUP by f
order by f
┌─f──────────────┬─PV─┐
│ filter1.value1 │ 1 │
│ filter1.value2 │ 16 │
│ filter1.value3 │ 3 │
│ filter1.value4 │ 4 │
│ filter1.value5 │ 5 │
│ filter2.value1 │ 6 │
│ filter2.value2 │ 7 │
│ filter2.value3 │ 8 │
│ filter2.value4 │ 9 │
│ filter3.value1 │ 21 │
│ filter3.value2 │ 27 │
│ filter3.value3 │ 13 │
│ filter3.value4 │ 14 │
└────────────────┴────┘
select splitByChar('.',f)[1] x, sum(PV), groupArray(PV), groupArray(f)
from (
SELECT
arrayJoin(splitByChar(',',replaceRegexpAll(pagePath,'^/url.*/(.*\..*)$','\\1'))) f,
sum(pageviews) as PV
FROM T
GROUP by f
order by f) group by x
┌─x───────┬─sum(PV)─┬─groupArray(PV)─┬─groupArray(f)──────────────────────────────────────────────────────────────────────────┐
│ filter2 │ 30 │ [6,7,8,9] │ ['filter2.value1','filter2.value2','filter2.value3','filter2.value4'] │
│ filter3 │ 75 │ [21,27,13,14] │ ['filter3.value1','filter3.value2','filter3.value3','filter3.value4'] │
│ filter1 │ 29 │ [1,16,3,4,5] │ ['filter1.value1','filter1.value2','filter1.value3','filter1.value4','filter1.value5'] │
└─────────┴─────────┴────────────────┴────────────────────────────────────────────────────────────────────────────────────────┘
【讨论】:
以上是关于使用 ClickHouse 提取和求和字符串内的子字段的值的主要内容,如果未能解决你的问题,请参考以下文章
Clickhouse实现累计求和cumulative sum