Clickhouse实现累计求和cumulative sum

Posted 2023-05-16

参考技术A 源表数据如下：

需求是按照省份和时间分类，每天累计求和，需求的结果如下：
（每天的数据都要按照省份将之前的订单数目进行累加）

需要借助clickhouse的array系列函数实现: ArrayFunction

使用 ClickHouse 提取和求和字符串内的子字段的值

【中文标题】使用 ClickHouse 提取和求和字符串内的子字段的值

【英文标题】：Extract and sum values with subfields inside string using ClickHouse

【发布时间】：2021-02-20 12:41:18

【问题描述】：

我有一个 ClickHouse 数据库，其中包含一个包含两个字段（pagePath 字符串、pageviews int）的简单表

我希望每个过滤器 + 值的总和访问量，以了解过滤器中用户的综合浏览量（最常用的过滤器）。过滤器用逗号分隔

示例数据

pagePath	pageviews
/url1/filter1.value1	1
/url1/filter1.value2	2
/url1/filter1.value3,filter1.value2	3
/url1/filter1.value4,filter3.value2	4
/url1/filter1.value5,filter3.value2,filter1.value2	5
/url1/filter2.value1,filter3.value2,filter1.value2	6
/url1/filter2.value2	7
/url-2/filter2.value3	8
/url-2/filter2.value4	9
/url-11/filter3.value1	10
/url-21/filter3.value1	11
/url1/filter3.value2	12
/url1/filter3.value3	13
/url1/filter3.value4	14

create table T (pagePath String , pageviews Int64) Engine=Memory;

insert into T values ('/url1/filter1.value1',1);
insert into T values ('/url1/filter1.value2',2);
insert into T values ('/url1/filter1.value3,filter1.value2',3);
insert into T values ('/url1/filter1.value4,filter3.value2',4);
insert into T values ('/url1/filter1.value5,filter3.value2,filter1.value2',5);
insert into T values ('/url1/filter2.value1,filter3.value2,filter1.value2',6);
insert into T values ('/url1/filter2.value2',7);
insert into T values ('/url-2/filter2.value3',8);
insert into T values ('/url-2/filter2.value4',9);
insert into T values ('/url-11/filter3.value1',10);
insert into T values ('/url-21/filter3.value1',11);
insert into T values ('/url1/filter3.value2',12);
insert into T values ('/url1/filter3.value3',13);
insert into T values ('/url1/filter3.value4',14);

我想获得 每个过滤器 + 值的浏览量总和

Filters	pageviews
filter1.value1	1
filter1.value2	16 (2 + 3 +5 +6)
filter1.value3	3
filter1.value4	4
filter1.value5	5
filter2.value1	6
filter2.value2	7
filter2.value3	8
filter2.value4	9
filter3.value1	10
filter3.value1	11
filter3.value2	12 (4+5+6+12)
filter3.value3	13
filter3.value4	14

我还想获得 每个过滤器的浏览量总和（没有值）

Filters	pageviews
filter1	15 (1, 2, 3, 4, 5)
filter2	30 (6+ 7+ 8+ 9)
filter1	60 ( 10, 11, 12, 13, 14)

我试试

select
    arrJoinFilters,
    sum (PV) as totales
from
    (
    SELECT
        arrJoinFilters,
        splitByChar(',',replaceRegexpAll(pagePath,'^/url.*/(.*\..*)$','\\1')) arrFilter,
        pageviews as PV
    FROM
        Table ARRAY JOIN arrFilter AS arrJoinFilters
    GROUP by arrFilter,PV,arrJoinFilters
        )
group by
    arrJoinFilters
    order by arrJoinFilters

但是，我认为有一些错误，我没有得到想要的第二个结果

谢谢！

【参考方案1】：

4+5+6+12 = 27

SELECT
      arrayJoin(splitByChar(',',replaceRegexpAll(pagePath,'^/url.*/(.*\..*)$','\\1'))) f,
      sum(pageviews) as PV
FROM T  
GROUP by f
order by f

┌─f──────────────┬─PV─┐
│ filter1.value1 │  1 │
│ filter1.value2 │ 16 │
│ filter1.value3 │  3 │
│ filter1.value4 │  4 │
│ filter1.value5 │  5 │
│ filter2.value1 │  6 │
│ filter2.value2 │  7 │
│ filter2.value3 │  8 │
│ filter2.value4 │  9 │
│ filter3.value1 │ 21 │
│ filter3.value2 │ 27 │
│ filter3.value3 │ 13 │
│ filter3.value4 │ 14 │
└────────────────┴────┘


select splitByChar('.',f)[1] x, sum(PV), groupArray(PV), groupArray(f)
from (
SELECT
  arrayJoin(splitByChar(',',replaceRegexpAll(pagePath,'^/url.*/(.*\..*)$','\\1'))) f,
  sum(pageviews) as PV
FROM T  
GROUP by f
order by f) group by x


┌─x───────┬─sum(PV)─┬─groupArray(PV)─┬─groupArray(f)──────────────────────────────────────────────────────────────────────────┐
│ filter2 │      30 │ [6,7,8,9]      │ ['filter2.value1','filter2.value2','filter2.value3','filter2.value4']                  │
│ filter3 │      75 │ [21,27,13,14]  │ ['filter3.value1','filter3.value2','filter3.value3','filter3.value4']                  │
│ filter1 │      29 │ [1,16,3,4,5]   │ ['filter1.value1','filter1.value2','filter1.value3','filter1.value4','filter1.value5'] │
└─────────┴─────────┴────────────────┴────────────────────────────────────────────────────────────────────────────────────────┘