PostgreSQL使用clickhousedb_fdw访问ClickHouse

Posted 2021-10-11 PostgreSQLChina

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了PostgreSQL使用clickhousedb_fdw访问ClickHouse相关的知识，希望对你有一定的参考价值。

作者：杨杰

简介

PostgreSQL FDW是一种外部访问接口，它可以被用来访问存储在外部的数据，这些数据可以是外部的PG数据库，也可以mysql、ClickHouse等数据库。

ClickHouse是一款快速的开源OLAP数据库管理系统，它是面向列的，允许使用SQL查询实时生成分析报告。

clickhouse_fdw是一个开源的外部数据包装器(FDW)用于访问ClickHouse列存数据库。

目前有以下两款clickhouse_fdw：
https://github.com/adjust/cli...

一直持续不断的有提交，目前支持PostgreSQL 11-13
https://github.com/Percona-La...

之前有一年时间没有动静，最近一段时间刚从adjust/clickhouse_fdw merge了一下，目前也支持PostgreSQL 11-13。

本文就以adjust/clickhouse_fdw为例。

安装

# libcurl >= 7.43.0
yum install libcurl-devel libuuid-devel

git clone https://github.com/adjust/clickhouse_fdw.git
cd clickhouse_fdw
mkdir build && cd build
cmake ..
make && make install

使用

CH端：
生成测试表及数据，这里我们使用CH官网提供的Star Schema Benchmark

https://clickhouse.tech/docs/...

模拟数据量：5张数据表，数据主要集中在lineorder*表，单表9000w rows左右、22G存储。

[root@vm101 ansible]# clickhouse client
ClickHouse client version 20.8.9.6.
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.8.9 revision 54438.

vm101 :) show tables;

SHOW TABLES

┌─name───────────┐
│ customer │
│ lineorder │
│ lineorder_flat │
│ part │
│ supplier │
└────────────────┘

5 rows in set. Elapsed: 0.004 sec. 

vm101 :) select count(*) from lineorder_flat;

SELECT count(*)
FROM lineorder_flat

┌──count()─┐
│ 89987373 │
└──────────┘

1 rows in set. Elapsed: 0.005 sec. 

[root@vm101 ansible]# du -sh /clickhouse/data/default/lineorder_flat/
22G /clickhouse/data/default/lineorder_flat/

PG端：
创建FDW插件

postgres=# create extension clickhouse_fdw ;
CREATE EXTENSION
postgres=# \\dew
List of foreign-data wrappers
Name | Owner | Handler | Validator 
----------------+----------+--------------------------+----------------------------
clickhouse_fdw | postgres | clickhousedb_fdw_handler | clickhousedb_fdw_validator
(1 row)

创建CH外部服务器

postgres=# CREATE SERVER clickhouse_svr FOREIGN DATA WRAPPER clickhouse_fdw 
OPTIONS(host \'10.0.0.101\', port \'9000\', dbname \'default\', driver \'binary\');
CREATE SERVER
postgres=# \\des
List of foreign servers
Name | Owner | Foreign-data wrapper 
----------------+----------+----------------------
clickhouse_svr | postgres | clickhouse_fdw
(1 row)

创建用户映射

postgres=# CREATE USER MAPPING FOR CURRENT_USER SERVER clickhouse_svr 
OPTIONS (user \'default\', password \'\');
CREATE USER MAPPING
postgres=# \\deu
List of user mappings
Server | User name 
----------------+-----------
clickhouse_svr | postgres
(1 row)

创建外部表

postgres=# IMPORT FOREIGN SCHEMA "default" FROM SERVER clickhouse_svr INTO public;
IMPORT FOREIGN SCHEMA
postgres=# \\det
List of foreign tables
Schema | Table | Server 
--------+----------------+----------------
public | customer | clickhouse_svr
public | lineorder | clickhouse_svr
public | lineorder_flat | clickhouse_svr
public | part | clickhouse_svr
public | supplier | clickhouse_svr
(5 rows)

查询

postgres=# select count(*) from lineorder_flat ;
count 
----------
89987373
(1 row)

postgres=# select "LO_ORDERKEY","C_NAME" from lineorder_flat limit 5;
LO_ORDERKEY | C_NAME 
-------------+--------------------
3271 | Customer#000099173
3271 | Customer#000099173
3271 | Customer#000099173
3271 | Customer#000099173
5607 | Customer#000273061
(5 rows)

需要注意的是CH是区分大小写的以及一些函数兼容问题，上面的示例也有展示。

测试SQL直接使用CH SSB提供的13条SQL，SQL基本类似，选一条做下测试，运行时间基本是一致的。

CH:

vm101 :) SELECT
:-] toYear(LO_ORDERDATE) AS year,
:-] C_NATION,
:-] sum(LO_REVENUE - LO_SUPPLYCOST) AS profit
:-] FROM lineorder_flat
:-] WHERE C_REGION = \'AMERICA\' AND S_REGION = \'AMERICA\' AND (P_MFGR = \'MFGR#1\' OR P_MFGR = \'MFGR#2\')
:-] GROUP BY
:-] year,
:-] C_NATION
:-] ORDER BY
:-] year ASC,
:-] C_NATION ASC;

SELECT 
toYear(LO_ORDERDATE) AS year,
C_NATION,
sum(LO_REVENUE - LO_SUPPLYCOST) AS profit
FROM lineorder_flat
WHERE (C_REGION = \'AMERICA\') AND (S_REGION = \'AMERICA\') AND ((P_MFGR = \'MFGR#1\') OR (P_MFGR = \'MFGR#2\'))
GROUP BY 
year,
C_NATION
ORDER BY 
year ASC,
C_NATION ASC

┌─year─┬─C_NATION──────┬───────profit─┐
│ 1992 │ ARGENTINA │ 157402521853 │
...
│ 1998 │ UNITED STATES │ 89854580268 │
└──────┴───────────────┴──────────────┘

35 rows in set. Elapsed: 0.195 sec. Processed 89.99 million rows, 1.26 GB (460.70 million rows/s., 6.46 GB/s.)
PG:
postgres=# SELECT
date_part(\'year\', "LO_ORDERDATE") AS year,
"C_NATION",
sum("LO_REVENUE" - "LO_SUPPLYCOST") AS profit
FROM lineorder_flat
WHERE "C_REGION" = \'AMERICA\' AND "S_REGION" = \'AMERICA\' AND ("P_MFGR" = \'MFGR#1\' OR "P_MFGR" = \'MFGR#2\')
GROUP BY
year,
"C_NATION"
ORDER BY
year ASC,
"C_NATION" ASC;
year | C_NATION | profit 
------+---------------+--------------
1992 | ARGENTINA | 157402521853
...
1998 | UNITED STATES | 89854580268
(35 rows)

Time: 195.102 ms