MySQL HeatWave Quickstart AutoPilot简介
Posted dingdingfish
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了MySQL HeatWave Quickstart AutoPilot简介相关的知识，希望对你有一定的参考价值。
本文是MySQL HeatWave Quickstart (快速入门)的续篇。
今天看了下MySQL HeatWave用户手册，已由60多页变为96页，增加了Autopilot，对象存储持久化层等内容。
基本概念：
- HeatWave是内存列式的，分布式的，无共享的
- HeatWave Cluster包括1个InnoDB和多个HeatWave节点（至少2个，最多64个）。
- HeatWave只在Oracle的公有云上提供。
先看下最新架构：
图中的OCI Object Storage是最新的功能，目的是：
HeatWave Storage Layer automatically persists the data to OCI Object Storage for fast recovery in case of a HeatWave node or cluster failure.
The time required to reload data is constant regardless of data size or HeatWave cluster size.
文中谈到了Push-Based Vectorized Query Processing的概念，相对于tuple-based，描述为：
HeatWave processes queries by pushing vector blocks (slices of columnar data) through the query execution plan from one operator to another. A push-based execution model avoids deep call stacks and saves valuable resources compared to tuple-based processing models.
还不太懂，后续再看。
一张表被分布到所有HeatWave节点，因此其需要主键。
Shape目前可以选择2种：
- mysql.HeatWave.VM.Standard.E3 （16 OCPU，512GB RAM）
- MySQL.HeatWave.BM.Standard.E3（128 OCPU，2TB RAM）
AutoPilot可参见Oracle Announces MySQL Autopilot for MySQL HeatWave Service。
以及MySQL Autopilot - Machine Learning Automation for MySQL HeatWave。
AutoPilot是基于机器学习的自动化服务。
来看一下AutoPilot的4大功能：
System Setup
Auto provisioning
帮助见这里。
可以自动估算HeatWave Cluster的节点数量。见菜单“Estimate Node Count”。
此步骤执行时，Schema里必须有数据。输出中表示估算节点数是1，但最小应为2。
Data Load
Auto Parallel Load，Auto Encoding，Auto Data Placement 3项子功能。
Auto Parallel Load
帮助见这里。可以指定表，为简化，下例指定整个Schema。
先来看Auto Parallel Load，其实就是执行在节点估算时给出的命令，这比之前方便太多了！：
MySQL 10.0.1.28:3306 ssl tpch SQL > CALL sys.heatwave_load(JSON_ARRAY("tpch"), NULL);
+------------------------------------------+
| INITIALIZING HEATWAVE AUTO LOAD |
+------------------------------------------+
| Version: 1.15 |
| |
| Load Mode: normal |
| Load Policy: disable_unsupported_columns |
| Output Mode: normal |
| |
+------------------------------------------+
6 rows in set (0.0080 sec)
+------------------------------------------------------------------------+
| OFFLOAD ANALYSIS |
+------------------------------------------------------------------------+
| Verifying input schemas: 1 |
| User excluded items: 0 |
| |
| SCHEMA OFFLOADABLE OFFLOADABLE SUMMARY OF |
| NAME TABLES COLUMNS ISSUES |
| ------ ----------- ----------- ---------- |
| `tpch` 8 61 |
| |
| Total offloadable schemas: 1 |
| |
+------------------------------------------------------------------------+
10 rows in set (0.0080 sec)
+-----------------------------------------------------------------------------------------------------------------------------+
| CAPACITY ESTIMATION |
+-----------------------------------------------------------------------------------------------------------------------------+
| Default load pool for tables: TRANSACTIONAL |
| Default encoding for string columns: VARLEN (unless specified in the schema) |
| Estimating memory footprint for 1 schema(s) |
| |
| TOTAL ESTIMATED ESTIMATED TOTAL DICTIONARY VARLEN ESTIMATED |
| SCHEMA OFFLOADABLE HEATWAVE NODE MYSQL NODE STRING ENCODED ENCODED LOAD |
| NAME TABLES FOOTPRINT FOOTPRINT COLUMNS COLUMNS COLUMNS TIME |
| ------ ----------- --------- --------- ------- ---------- ------- --------- |
| `tpch` 8 2.22 GiB 32.00 MiB 29 0 29 10.00 s |
| |
| Sufficient MySQL host memory available to load all tables. |
| Sufficient HeatWave cluster memory available to load all tables. |
| |
+-----------------------------------------------------------------------------------------------------------------------------+
13 rows in set (0.0080 sec)
+---------------------------------------------------------------------------------------------------------------------------------------+
| EXECUTING LOAD |
+---------------------------------------------------------------------------------------------------------------------------------------+
| HeatWave Load script generated |
| Retrieve load script containing 24 generated DDL command(s) using the query below: |
| SELECT log->>"$.sql" AS "Load Script" FROM sys.heatwave_load_report WHERE type = "sql" ORDER BY id; |
| |
| Adjusting load parallelism dynamically per table |
| Using current parallelism of 32 thread(s) as maximum |
| |
| Using SQL_MODE: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION |
| |
| Proceeding to load 8 tables into HeatWave |
| |
| Applying changes will take approximately 9.74 s |
| |
+---------------------------------------------------------------------------------------------------------------------------------------+
13 rows in set (0.0080 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (1 of 8): `tpch`.`customer` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 8 |
| Table loaded using 3 thread(s) |
| |
+----------------------------------------+
7 rows in set (0.0080 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (2 of 8): `tpch`.`lineitem` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 16 |
| Table loaded using 32 thread(s) |
| |
+----------------------------------------+
7 rows in set (0.0080 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (3 of 8): `tpch`.`nation` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 4 |
| Table loaded using 1 thread(s) |
| |
+----------------------------------------+
7 rows in set (0.0080 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (4 of 8): `tpch`.`orders` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 9 |
| Table loaded using 16 thread(s) |
| |
+----------------------------------------+
7 rows in set (0.0080 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (5 of 8): `tpch`.`part` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 9 |
| Table loaded using 4 thread(s) |
| |
+----------------------------------------+
7 rows in set (0.0080 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (6 of 8): `tpch`.`partsupp` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 5 |
| Table loaded using 15 thread(s) |
| |
+----------------------------------------+
7 rows in set (0.0080 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (7 of 8): `tpch`.`region` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 3 |
| Table loaded using 1 thread(s) |
| |
+----------------------------------------+
7 rows in set (0.0080 sec)
+----------------------------------------+
| LOADING TABLE |
+----------------------------------------+
| TABLE (8 of 8): `tpch`.`supplier` |
| Commands executed successfully: 3 of 3 |
| Warnings encountered: 0 |
| Table loaded successfully! |
| Total columns loaded: 7 |
| Table loaded using 1 thread(s) |
| |
+----------------------------------------+
7 rows in set (0.0080 sec)
+-------------------------------------------------------------------------------+
| LOAD SUMMARY |
+-------------------------------------------------------------------------------+
| |
| SCHEMA TABLES TABLES COLUMNS LOAD |
| NAME LOADED FAILED LOADED DURATION |
| ------ ------ ------ ------- -------- |
| `tpch` 8 0 61 2.55 s |
| |
+-------------------------------------------------------------------------------+
6 rows in set (0.0080 sec)
Query OK, 0 rows affected (0.0080 sec)
相关查看命令：
MySQL 10.0.1.28:3306 ssl tpch SQL > SELECT log FROM sys.heatwave_load_report WHERE type="error";
Empty set (0.0007 sec)
MySQL 10.0.1.28:3306 ssl tpch SQL > SELECT log FROM sys.heatwave_load_report WHERE type="warn";
Empty set (0.0006 sec)
MySQL 10.0.1.28:3306 ssl tpch SQL > SELECT log->>"$.sql" AS "Load Script" FROM sys.heatwave_load_report
-> WHERE type = "sql" ORDER BY id;
+-------------------------------------------------------+
| Load Script |
+-------------------------------------------------------+
| SET SESSION innodb_parallel_read_threads = 3; |
| ALTER TABLE `tpch`.`customer` SECONDARY_ENGINE=RAPID; |
| ALTER TABLE `tpch`.`customer` SECONDARY_LOAD; |
| SET SESSION innodb_parallel_read_threads = 32; |
| ALTER TABLE `tpch`.`lineitem` SECONDARY_ENGINE=RAPID; |
| ALTER TABLE `tpch`.`lineitem` SECONDARY_LOAD; |
| SET SESSION innodb_parallel_read_threads = 1; |
| ALTER TABLE `tpch`.`nation` SECONDARY_ENGINE=RAPID; |
| ALTER TABLE `tpch`.`nation` SECONDARY_LOAD; |
| SET SESSION innodb_parallel_read_threads = 16; |
| ALTER TABLE `tpch`.`orders` SECONDARY_ENGINE=RAPID; |
| ALTER TABLE `tpch`.`orders` SECONDARY_LOAD; |
| SET SESSION innodb_parallel_read_threads = 4; |
| ALTER TABLE `tpch`.`part` SECONDARY_ENGINE=RAPID; |
| ALTER TABLE `tpch`.`part` SECONDARY_LOAD; |
| SET SESSION innodb_parallel_read_threads = 15; |
| ALTER TABLE `tpch`.`partsupp` SECONDARY_ENGINE=RAPID; |
| ALTER TABLE `tpch`.`partsupp` SECONDARY_LOAD; |
| SET SESSION innodb_parallel_read_threads = 1; |
| ALTER TABLE `tpch`.`region` SECONDARY_ENGINE=RAPID; |
| ALTER TABLE `tpch`.`region` SECONDARY_LOAD; |
| SET SESSION innodb_parallel_read_threads = 1; |
| ALTER TABLE `tpch`.`supplier` SECONDARY_ENGINE=RAPID; |
| ALTER TABLE `tpch`.`supplier` SECONDARY_LOAD; |
+-------------------------------------------------------+
24 rows in set (0.0007 sec)
Auto Encoding
帮助见这里。
实际就是建议器，决定是VARLEN还是DICTIONARY编码.
以下输出中，建议部分列的编码从VARLEN改为DICTIONARY，以提升性能。而且还给出命令，以生成修改脚本。
MySQL 10.0.1.28:3306 ssl tpch SQL > CALL sys.heatwave_advisor(JSON_OBJECT("auto_enc",JSON_OBJECT("mode","recommend")));
+-------------------------------+
| INITIALIZING HEATWAVE ADVISOR |
+-------------------------------+
| Version: 1.15 |
| |
| Output Mode: normal |
| Excluded Queries: 0 |
| Target Schemas: All |
| |
+-------------------------------+
6 rows in set (0.0098 sec)
+---------------------------------------------------------+
| ANALYZING LOADED DATA |
+---------------------------------------------------------+
| Total 8 tables loaded in HeatWave for 1 schemas |
| Tables excluded by user: 0 (within target schemas) |
| |
| SCHEMA TABLES COLUMNS |
| NAME LOADED LOADED |
| ------ ------ ------ |
| `tpch` 8 61 |
| |
+---------------------------------------------------------+
8 rows in set (0.0098 sec)
+--------------------------------------------------------------------------------------------+
| ENCODING SUGGESTIONS |
+--------------------------------------------------------------------------------------------+
| Total Auto Encoding suggestions produced for 25 columns |
| Queries executed: 1 |
| Total query execution time: 0 ps |
| Most recent query executed on: Thursday 1st January 1970 00:00:00 |
| Oldest query executed on: Thursday 1st January 1970 00:00:00 |
| |
| CURRENT SUGGESTED |
| COLUMN COLUMN COLUMN |
| NAME ENCODING ENCODING |
| ------ -------- --------- |
| `tpch`.`customer`.`C_ADDRESS` VARLEN DICTIONARY |
| `tpch`.`customer`.`C_COMMENT` VARLEN DICTIONARY |
| `tpch`.`customer`.`C_MKTSEGMENT` VARLEN DICTIONARY |
| `tpch`.`customer`.`C_NAME` VARLEN DICTIONARY |
| `tpch`.`customer`.`C_PHONE` VARLEN DICTIONARY |
| `tpch`.`lineitem`.`L_COMMENT` VARLEN DICTIONARY |
| `tpch`.`lineitem`.`L_SHIPINSTRUCT` VARLEN DICTIONARY |
| `tpch`.`nation`.`N_COMMENT` VARLEN DICTIONARY |
| `tpch`.`nation`.`N_NAME` VARLEN DICTIONARY |
| `tpch`.`orders`.`O_CLERK` VARLEN DICTIONARY |
| `tpch`.`orders`.`O_COMMENT` VARLEN DICTIONARY |
| `tpch`.`orders`.`O_ORDERPRIORITY` VARLEN DICTIONARY |
| `tpch`.`part`.`P_BRAND` VARLEN DICTIONARY |
| `tpch`.`part`.`P_COMMENT` VARLEN DICTIONARY |
| `tpch`.`part`.`P_CONTAINER` VARLEN DICTIONARY |
| `tpch`.`part`.`P_MFGR` VARLEN DICTIONARY |
| `tpch`.`part`.`P_NAME` 以上是关于MySQL HeatWave Quickstart AutoPilot简介的主要内容，如果未能解决你的问题，请参考以下文章
MySQL HeatWave Quickstart AutoPilot简介
Oracle CloudWorld 2022：重磅推出MySQL湖仓产品-MySQL HeatWave Lakehouse