MySQL HeatWave airportdb Quickstart 实验过程
Posted dingdingfish
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了MySQL HeatWave airportdb Quickstart 实验过程相关的知识，希望对你有一定的参考价值。
本实验参考此文档。
这个实验可以了解以下知识点：
- 如何从对象存储倒入数据库到mysql
- 如何使用Autopilot中的Auto Parallel Load特性加载数据到HeatWave
在这里提供了很多MySQL的示例数据库。我们需要的airport DB从这里下载，约640MB。
下载并解压：
$ wget https://downloads.mysql.com/docs/airport-db.zip
$ ls -l airport-db.zip
-rw-rw-r--. 1 opc opc 655695008 Sep 10 17:26 airport-db.zip
$ unzip airport-db.zip
$ ls -l airport-db
total 641612
-rwxr-xr-x. 1 opc opc 1589 Sep 10 17:25 airportdb@airline@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airline@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 627 Sep 10 17:25 airportdb@airline.json
-rwxr-xr-x. 1 opc opc 1259 Sep 10 17:25 airportdb@airline.sql
-rwxr-xr-x. 1 opc opc 24022 Sep 10 17:25 airportdb@airplane@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airplane@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 628 Sep 10 17:25 airportdb@airplane.json
-rwxr-xr-x. 1 opc opc 1105 Sep 10 17:25 airportdb@airplane.sql
-rwxr-xr-x. 1 opc opc 181412 Sep 10 17:25 airportdb@airplane_type@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airplane_type@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 605 Sep 10 17:25 airportdb@airplane_type.json
-rwxr-xr-x. 1 opc opc 1115 Sep 10 17:25 airportdb@airplane_type.sql
-rwxr-xr-x. 1 opc opc 142410 Sep 10 17:25 airportdb@airport@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airport@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 408522 Sep 10 17:25 airportdb@airport_geo@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airport_geo@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 774 Sep 10 17:25 airportdb@airport_geo.json
-rwxr-xr-x. 1 opc opc 1366 Sep 10 17:25 airportdb@airport_geo.sql
-rwxr-xr-x. 1 opc opc 612 Sep 10 17:25 airportdb@airport.json
-rwxr-xr-x. 1 opc opc 582 Sep 10 17:25 airportdb@airport_reachable.json
-rwxr-xr-x. 1 opc opc 1010 Sep 10 17:25 airportdb@airport_reachable.sql
-rwxr-xr-x. 1 opc opc 9 Sep 10 17:25 airportdb@airport_reachable.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@airport_reachable.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 1209 Sep 10 17:25 airportdb@airport.sql
-rwxr-xr-x. 1 opc opc 23990482 Sep 10 17:25 airportdb@booking@0.tsv.zst
-rwxr-xr-x. 1 opc opc 512 Sep 10 17:25 airportdb@booking@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24787816 Sep 10 17:25 airportdb@booking@10.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@10.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24605829 Sep 10 17:25 airportdb@booking@11.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@11.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24662300 Sep 10 17:25 airportdb@booking@12.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@12.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24598565 Sep 10 17:25 airportdb@booking@13.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@13.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24474339 Sep 10 17:25 airportdb@booking@14.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@14.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24659075 Sep 10 17:25 airportdb@booking@15.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@15.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24460743 Sep 10 17:25 airportdb@booking@16.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@16.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24872813 Sep 10 17:25 airportdb@booking@17.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@17.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24730510 Sep 10 17:25 airportdb@booking@18.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@18.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24664897 Sep 10 17:25 airportdb@booking@19.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@19.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 23821261 Sep 10 17:25 airportdb@booking@1.tsv.zst
-rwxr-xr-x. 1 opc opc 520 Sep 10 17:25 airportdb@booking@1.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24477378 Sep 10 17:25 airportdb@booking@20.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@20.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24603821 Sep 10 17:25 airportdb@booking@21.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@21.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24510521 Sep 10 17:25 airportdb@booking@22.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@22.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24492613 Sep 10 17:25 airportdb@booking@23.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@23.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 7089904 Sep 10 17:25 airportdb@booking@@24.tsv.zst
-rwxr-xr-x. 1 opc opc 160 Sep 10 17:25 airportdb@booking@@24.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 29859055 Sep 10 17:25 airportdb@booking@2.tsv.zst
-rwxr-xr-x. 1 opc opc 648 Sep 10 17:25 airportdb@booking@2.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 22284735 Sep 10 17:25 airportdb@booking@3.tsv.zst
-rwxr-xr-x. 1 opc opc 488 Sep 10 17:25 airportdb@booking@3.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 28336025 Sep 10 17:25 airportdb@booking@4.tsv.zst
-rwxr-xr-x. 1 opc opc 616 Sep 10 17:25 airportdb@booking@4.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24829241 Sep 10 17:25 airportdb@booking@5.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@5.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24574052 Sep 10 17:25 airportdb@booking@6.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@6.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24435530 Sep 10 17:25 airportdb@booking@7.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@7.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24483962 Sep 10 17:25 airportdb@booking@8.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@8.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24687308 Sep 10 17:25 airportdb@booking@9.tsv.zst
-rwxr-xr-x. 1 opc opc 544 Sep 10 17:25 airportdb@booking@9.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 646 Sep 10 17:25 airportdb@booking.json
-rwxr-xr-x. 1 opc opc 1371 Sep 10 17:25 airportdb@booking.sql
-rwxr-xr-x. 1 opc opc 89932 Sep 10 17:25 airportdb@employee@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@employee@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 883 Sep 10 17:25 airportdb@employee.json
-rwxr-xr-x. 1 opc opc 2016 Sep 10 17:25 airportdb@employee.sql
-rwxr-xr-x. 1 opc opc 10133356 Sep 10 17:25 airportdb@flight@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 256 Sep 10 17:25 airportdb@flight@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 712 Sep 10 17:25 airportdb@flight.json
-rwxr-xr-x. 1 opc opc 1035 Sep 10 17:25 airportdb@flight_log.json
-rwxr-xr-x. 1 opc opc 1827 Sep 10 17:25 airportdb@flight_log.sql
-rwxr-xr-x. 1 opc opc 9 Sep 10 17:25 airportdb@flight_log.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@flight_log.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 172016 Sep 10 17:25 airportdb@flightschedule@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@flightschedule@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 829 Sep 10 17:25 airportdb@flightschedule.json
-rwxr-xr-x. 1 opc opc 1686 Sep 10 17:25 airportdb@flightschedule.sql
-rwxr-xr-x. 1 opc opc 1821 Sep 10 17:25 airportdb@flight.sql
-rwxr-xr-x. 1 opc opc 1199 Sep 10 17:25 airportdb.json
-rwxr-xr-x. 1 opc opc 494930 Sep 10 17:25 airportdb@passenger@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 8 Sep 10 17:25 airportdb@passenger@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 1672340 Sep 10 17:25 airportdb@passengerdetails@@0.tsv.zst
-rwxr-xr-x. 1 opc opc 32 Sep 10 17:25 airportdb@passengerdetails@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 748 Sep 10 17:25 airportdb@passengerdetails.json
-rwxr-xr-x. 1 opc opc 1569 Sep 10 17:25 airportdb@passengerdetails.sql
-rwxr-xr-x. 1 opc opc 633 Sep 10 17:25 airportdb@passenger.json
-rwxr-xr-x. 1 opc opc 1183 Sep 10 17:25 airportdb@passenger.sql
-rwxr-xr-x. 1 opc opc 583 Sep 10 17:25 airportdb.sql
-rwxr-xr-x. 1 opc opc 35538282 Sep 10 17:25 airportdb@weatherdata@0.tsv.zst
-rwxr-xr-x. 1 opc opc 1752 Sep 10 17:25 airportdb@weatherdata@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 4733492 Sep 10 17:25 airportdb@weatherdata@@1.tsv.zst
-rwxr-xr-x. 1 opc opc 240 Sep 10 17:25 airportdb@weatherdata@@1.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 738 Sep 10 17:25 airportdb@weatherdata.json
-rwxr-xr-x. 1 opc opc 1310 Sep 10 17:25 airportdb@weatherdata.sql
-rwxr-xr-x. 1 opc opc 2586 Sep 10 17:25 @.done.json
-rwxr-xr-x. 1 opc opc 859 Sep 10 17:25 @.json
-rwxr-xr-x. 1 opc opc 46349 Sep 10 17:25 @.manifest.json
-rwxr-xr-x. 1 opc opc 240 Sep 10 17:25 @.post.sql
-rwxr-xr-x. 1 opc opc 4884 Sep 10 17:25 README.txt
-rwxr-xr-x. 1 opc opc 240 Sep 10 17:25 @.sql
安装OCI CLI，OCI Linux 7中居然已经安装好了，还是最新版本。
使用OCI CLI将目录中的数据库文件批量上传到对象存储：
$ oci os object bulk-upload -ns ocichina001 -bn airport --src-dir ~/airport-db
然后将数据库从对象存储导入，并不需要事先建库，会自动建库：
$ mysqlsh root@10.0.1.233
MySQL 10.0.1.233:33060+ ssl JS > util.loadDump("", {threads: 16, loadIndexes: "false", osBucketName: "airport", osNamespace: "ocichina001", ignoreVersion: true})
Loading DDL and Data from OCI ObjectStorage bucket=airport, prefix='' using 16 threads.
Opening dump...
Target is MySQL 8.0.26-u1-cloud (MySQL Database Service). Dump was produced from MySQL 8.0.26-cloud
Fetching dump data from remote location...
Fetching 14 table metadata files for schema `airportdb`...
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL script for schema `airportdb`
[Worker005] Executing DDL script for `airportdb`.`flight`
[Worker003] Executing DDL script for `airportdb`.`employee`
[Worker011] Executing DDL script for `airportdb`.`weatherdata`
[Worker012] Executing DDL script for `airportdb`.`booking`
[Worker014] Executing DDL script for `airportdb`.`flight_log`
[Worker006] Executing DDL script for `airportdb`.`passengerdetails`
[Worker010] Executing DDL script for `airportdb`.`flightschedule`
[Worker002] Executing DDL script for `airportdb`.`airport_reachable`
[Worker013] Executing DDL script for `airportdb`.`airplane_type` (indexes removed for deferred creation)
[Worker009] Executing DDL script for `airportdb`.`airplane`
[Worker015] Executing DDL script for `airportdb`.`airline`
[Worker008] Executing DDL script for `airportdb`.`passenger`
[Worker007] Executing DDL script for `airportdb`.`airport`
[Worker000] Executing DDL script for `airportdb`.`airport_geo`
NOTE: [Worker007] Error processing table `airportdb`.`airport`, will retry after delay: MySQL Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
[Worker013] airportdb@airplane_type@@0.tsv.zst: Records: 342 Deleted: 0 Skipped: 0 Warnings: 0
[Worker008] airportdb@airline@@0.tsv.zst: Records: 113 Deleted: 0 Skipped: 0 Warnings: 0
[Worker000] airportdb@employee@@0.tsv.zst: Records: 1000 Deleted: 0 Skipped: 0 Warnings: 0
[Worker014] airportdb@airplane@@0.tsv.zst: Records: 5583 Deleted: 0 Skipped: 0 Warnings: 0
[Worker012] airportdb@airport@@0.tsv.zst: Records: 9854 Deleted: 0 Skipped: 0 Warnings: 0
[Worker002] airportdb@flightschedule@@0.tsv.zst: Records: 9881 Deleted: 0 Skipped: 0 Warnings: 0
[Worker006] airportdb@passenger@@0.tsv.zst: Records: 36095 Deleted: 0 Skipped: 0 Warnings: 0
[Worker003] airportdb@airport_geo@@0.tsv.zst: Records: 9854 Deleted: 0 Skipped: 0 Warnings: 0
[Worker005] airportdb@passengerdetails@@0.tsv.zst: Records: 36095 Deleted: 0 Skipped: 0 Warnings: 0
[Worker011] airportdb@flight@@0.tsv.zst: Records: 462553 Deleted: 0 Skipped: 0 Warnings: 0
[Worker004] airportdb@weatherdata@0.tsv.zst: Records: 4082688 Deleted: 0 Skipped: 0 Warnings: 0
[Worker004] airportdb@weatherdata@@1.tsv.zst: Records: 543744 Deleted: 0 Skipped: 0 Warnings: 0
[Worker010] airportdb@booking@3.tsv.zst: Records: 2024926 Deleted: 0 Skipped: 0 Warnings: 0
[Worker001] airportdb@booking@0.tsv.zst: Records: 2160442 Deleted: 0 Skipped: 0 Warnings: 0
[Worker009] airportdb@booking@1.tsv.zst: Records: 2159969 Deleted: 0 Skipped: 0 Warnings: 0
[Worker008] airportdb@booking@6.tsv.zst: Records: 2214973 Deleted: 0 Skipped: 0 Warnings: 0
[Worker002] airportdb@booking@10.tsv.zst: Records: 2214991 Deleted: 0 Skipped: 0 Warnings: 0
[Worker003] airportdb@booking@12.tsv.zst: Records: 2214478 Deleted: 0 Skipped: 0 Warnings: 0
[Worker013] airportdb@booking@5.tsv.zst: Records: 2215073 Deleted: 0 Skipped: 0 Warnings: 0
[Worker012] airportdb@booking@9.tsv.zst: Records: 2214959 Deleted: 0 Skipped: 0 Warnings: 0
[Worker000] airportdb@booking@7.tsv.zst: Records: 2214707 Deleted: 0 Skipped: 0 Warnings: 0
[Worker014] airportdb@booking@8.tsv.zst: Records: 2214781 Deleted: 0 Skipped: 0 Warnings: 0
[Worker005] airportdb@booking@13.tsv.zst: Records: 2215250 Deleted: 0 Skipped: 0 Warnings: 0
[Worker006] airportdb@booking@11.tsv.zst: Records: 2215122 Deleted: 0 Skipped: 0 Warnings: 0
[Worker007] airportdb@booking@4.tsv.zst: Records: 2531382 Deleted: 0 Skipped: 0 Warnings: 0
[Worker015] airportdb@booking@2.tsv.zst: Records: 2699650 Deleted: 0 Skipped: 0 Warnings: 0
[Worker011] airportdb@booking@14.tsv.zst: Records: 2215310 Deleted: 0 Skipped: 0 Warnings: 0
[Worker000] airportdb@booking@@24.tsv.zst: Records: 644485 Deleted: 0 Skipped: 0 Warnings: 0
[Worker004] airportdb@booking@15.tsv.zst: Records: 2215229 Deleted: 0 Skipped: 0 Warnings: 0
[Worker010] airportdb@booking@16.tsv.zst: Records: 2215038 Deleted: 0 Skipped: 0 Warnings: 0
[Worker001] airportdb@booking@17.tsv.zst: Records: 2214973 Deleted: 0 Skipped: 0 Warnings: 0
[Worker009] airportdb@booking@18.tsv.zst: Records: 2215177 Deleted: 0 Skipped: 0 Warnings: 0
[Worker002] airportdb@booking@20.tsv.zst: Records: 2214883 Deleted: 0 Skipped: 0 Warnings: 0
[Worker012] airportdb@booking@23.tsv.zst: Records: 2215175 Deleted: 0 Skipped: 0 Warnings: 0
[Worker008] airportdb@booking@19.tsv.zst: Records: 2214884 Deleted: 0 Skipped: 0 Warnings: 0
[Worker003] airportdb@booking@21.tsv.zst: Records: 2214070 Deleted: 0 Skipped: 0 Warnings: 0
[Worker013] airportdb@booking@22.tsv.zst: Records: 2214692 Deleted: 0 Skipped: 0 Warnings: 0
Executing common postamble SQL
37 chunks (59.50M rows, 2.03 GB) for 12 tables in 1 schemas were loaded in 2 min 6 sec (avg throughput 16.15 MB/s)
0 warnings were reported during the load.
验证数据库已创建，表已导入：
MySQL 10.0.1.233:33060+ ssl JS > \\sql
Switching to SQL mode... Commands end with ;
MySQL 10.0.1.233:33060+ ssl SQL > use airportdb;
Default schema set to `airportdb`.
Fetching table and column names from `airportdb` for auto-completion... Press ^C to stop.
MySQL 10.0.1.233:33060+ ssl airportdb SQL > select count(*) from flight;
+----------+
| count(*) |
+----------+
| 462553 |
+----------+
1 row in set (0.0134 sec)
添加HeatWave Cluster，默认的2个节点即可。
加载数据到HeatWave，这是Autopilot的特性：
MySQL 10.0.1.233:33060+ ssl airportdb SQL > CALL sys.heatwave_load(JSON_ARRAY('airportdb'), NULL);
+------------------------------------------+
| INITIALIZING HEATWAVE AUTO LOAD |
+------------------------------------------+
| Version: 1.15 |
| |
| Load Mode: normal |
| Load Policy: disable_unsupported_columns |
| Output Mode: normal |
| |
+------------------------------------------+
6 rows in set (5.2867 sec)
+-----------------------------------------------------------------------------------------------+
| OFFLOAD ANALYSIS |
+-----------------------------------------------------------------------------------------------+
| Verifying input schemas: 1 |
| User excluded items: 0 |
| |
| SCHEMA OFFLOADABLE OFFLOADABLE SUMMARY OF |
| NAME TABLES COLUMNS ISSUES |
| ------ ----------- ----------- ---------- |
| `airportdb` 14 104 2 table(s) are partially loadable |
| |
| Total offloadable schemas: 1 |
| |
+-----------------------------------------------------------------------------------------------+
10 rows in set (5.2867 sec)
+-----------------------------------------------------------------------------------------------------------------------------+
| CAPACITY ESTIMATION |
+-----------------------------------------------------------------------------------------------------------------------------+
| Default load pool for tables: TRANSACTIONAL |
| Default encoding for string columns: VARLEN (unless specified in the schema) |
| Estimating memory footprint for 1 schema(s) |
| |
| TOTAL ESTIMATED ESTIMATED TOTAL DICTIONARY VARLEN ESTIMATED |
| SCHEMA OFFLOADABLE HEATWAVE NODE MYSQL NODE STRING ENCODED ENCODED LOAD |
| NAME TABLES FOOTPRINT FOOTPRINT COLUMNS COLUMNS COLUMNS TIME |
| ------ 以上是关于MySQL HeatWave airportdb Quickstart 实验过程的主要内容，如果未能解决你的问题，请参考以下文章
MySQL HeatWave Quickstart AutoPilot简介
MySQL HeatWave Quickstart (快速入门)
Oracle CloudWorld 2022：重磅推出MySQL湖仓产品-MySQL HeatWave Lakehouse