MySQL HeatWave airportdb Quickstart 实验过程

Posted dingdingfish

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了MySQL HeatWave airportdb Quickstart 实验过程相关的知识,希望对你有一定的参考价值。

本实验参考此文档

这个实验可以了解以下知识点:

  1. 如何从对象存储倒入数据库到mysql
  2. 如何使用Autopilot中的Auto Parallel Load特性加载数据到HeatWave

这里提供了很多MySQL的示例数据库。我们需要的airport DB从这里下载,约640MB。

下载并解压:

$ wget https://downloads.mysql.com/docs/airport-db.zip

$ ls -l airport-db.zip
-rw-rw-r--. 1 opc opc 655695008 Sep 10 17:26 airport-db.zip

$ unzip airport-db.zip

$ ls -l airport-db
total 641612
-rwxr-xr-x. 1 opc opc     1589 Sep 10 17:25 airportdb@airline@@0.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@airline@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      627 Sep 10 17:25 airportdb@airline.json
-rwxr-xr-x. 1 opc opc     1259 Sep 10 17:25 airportdb@airline.sql
-rwxr-xr-x. 1 opc opc    24022 Sep 10 17:25 airportdb@airplane@@0.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@airplane@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      628 Sep 10 17:25 airportdb@airplane.json
-rwxr-xr-x. 1 opc opc     1105 Sep 10 17:25 airportdb@airplane.sql
-rwxr-xr-x. 1 opc opc   181412 Sep 10 17:25 airportdb@airplane_type@@0.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@airplane_type@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      605 Sep 10 17:25 airportdb@airplane_type.json
-rwxr-xr-x. 1 opc opc     1115 Sep 10 17:25 airportdb@airplane_type.sql
-rwxr-xr-x. 1 opc opc   142410 Sep 10 17:25 airportdb@airport@@0.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@airport@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc   408522 Sep 10 17:25 airportdb@airport_geo@@0.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@airport_geo@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      774 Sep 10 17:25 airportdb@airport_geo.json
-rwxr-xr-x. 1 opc opc     1366 Sep 10 17:25 airportdb@airport_geo.sql
-rwxr-xr-x. 1 opc opc      612 Sep 10 17:25 airportdb@airport.json
-rwxr-xr-x. 1 opc opc      582 Sep 10 17:25 airportdb@airport_reachable.json
-rwxr-xr-x. 1 opc opc     1010 Sep 10 17:25 airportdb@airport_reachable.sql
-rwxr-xr-x. 1 opc opc        9 Sep 10 17:25 airportdb@airport_reachable.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@airport_reachable.tsv.zst.idx
-rwxr-xr-x. 1 opc opc     1209 Sep 10 17:25 airportdb@airport.sql
-rwxr-xr-x. 1 opc opc 23990482 Sep 10 17:25 airportdb@booking@0.tsv.zst
-rwxr-xr-x. 1 opc opc      512 Sep 10 17:25 airportdb@booking@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24787816 Sep 10 17:25 airportdb@booking@10.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@10.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24605829 Sep 10 17:25 airportdb@booking@11.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@11.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24662300 Sep 10 17:25 airportdb@booking@12.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@12.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24598565 Sep 10 17:25 airportdb@booking@13.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@13.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24474339 Sep 10 17:25 airportdb@booking@14.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@14.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24659075 Sep 10 17:25 airportdb@booking@15.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@15.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24460743 Sep 10 17:25 airportdb@booking@16.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@16.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24872813 Sep 10 17:25 airportdb@booking@17.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@17.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24730510 Sep 10 17:25 airportdb@booking@18.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@18.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24664897 Sep 10 17:25 airportdb@booking@19.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@19.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 23821261 Sep 10 17:25 airportdb@booking@1.tsv.zst
-rwxr-xr-x. 1 opc opc      520 Sep 10 17:25 airportdb@booking@1.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24477378 Sep 10 17:25 airportdb@booking@20.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@20.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24603821 Sep 10 17:25 airportdb@booking@21.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@21.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24510521 Sep 10 17:25 airportdb@booking@22.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@22.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24492613 Sep 10 17:25 airportdb@booking@23.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@23.tsv.zst.idx
-rwxr-xr-x. 1 opc opc  7089904 Sep 10 17:25 airportdb@booking@@24.tsv.zst
-rwxr-xr-x. 1 opc opc      160 Sep 10 17:25 airportdb@booking@@24.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 29859055 Sep 10 17:25 airportdb@booking@2.tsv.zst
-rwxr-xr-x. 1 opc opc      648 Sep 10 17:25 airportdb@booking@2.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 22284735 Sep 10 17:25 airportdb@booking@3.tsv.zst
-rwxr-xr-x. 1 opc opc      488 Sep 10 17:25 airportdb@booking@3.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 28336025 Sep 10 17:25 airportdb@booking@4.tsv.zst
-rwxr-xr-x. 1 opc opc      616 Sep 10 17:25 airportdb@booking@4.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24829241 Sep 10 17:25 airportdb@booking@5.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@5.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24574052 Sep 10 17:25 airportdb@booking@6.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@6.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24435530 Sep 10 17:25 airportdb@booking@7.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@7.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24483962 Sep 10 17:25 airportdb@booking@8.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@8.tsv.zst.idx
-rwxr-xr-x. 1 opc opc 24687308 Sep 10 17:25 airportdb@booking@9.tsv.zst
-rwxr-xr-x. 1 opc opc      544 Sep 10 17:25 airportdb@booking@9.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      646 Sep 10 17:25 airportdb@booking.json
-rwxr-xr-x. 1 opc opc     1371 Sep 10 17:25 airportdb@booking.sql
-rwxr-xr-x. 1 opc opc    89932 Sep 10 17:25 airportdb@employee@@0.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@employee@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      883 Sep 10 17:25 airportdb@employee.json
-rwxr-xr-x. 1 opc opc     2016 Sep 10 17:25 airportdb@employee.sql
-rwxr-xr-x. 1 opc opc 10133356 Sep 10 17:25 airportdb@flight@@0.tsv.zst
-rwxr-xr-x. 1 opc opc      256 Sep 10 17:25 airportdb@flight@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      712 Sep 10 17:25 airportdb@flight.json
-rwxr-xr-x. 1 opc opc     1035 Sep 10 17:25 airportdb@flight_log.json
-rwxr-xr-x. 1 opc opc     1827 Sep 10 17:25 airportdb@flight_log.sql
-rwxr-xr-x. 1 opc opc        9 Sep 10 17:25 airportdb@flight_log.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@flight_log.tsv.zst.idx
-rwxr-xr-x. 1 opc opc   172016 Sep 10 17:25 airportdb@flightschedule@@0.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@flightschedule@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      829 Sep 10 17:25 airportdb@flightschedule.json
-rwxr-xr-x. 1 opc opc     1686 Sep 10 17:25 airportdb@flightschedule.sql
-rwxr-xr-x. 1 opc opc     1821 Sep 10 17:25 airportdb@flight.sql
-rwxr-xr-x. 1 opc opc     1199 Sep 10 17:25 airportdb.json
-rwxr-xr-x. 1 opc opc   494930 Sep 10 17:25 airportdb@passenger@@0.tsv.zst
-rwxr-xr-x. 1 opc opc        8 Sep 10 17:25 airportdb@passenger@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc  1672340 Sep 10 17:25 airportdb@passengerdetails@@0.tsv.zst
-rwxr-xr-x. 1 opc opc       32 Sep 10 17:25 airportdb@passengerdetails@@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      748 Sep 10 17:25 airportdb@passengerdetails.json
-rwxr-xr-x. 1 opc opc     1569 Sep 10 17:25 airportdb@passengerdetails.sql
-rwxr-xr-x. 1 opc opc      633 Sep 10 17:25 airportdb@passenger.json
-rwxr-xr-x. 1 opc opc     1183 Sep 10 17:25 airportdb@passenger.sql
-rwxr-xr-x. 1 opc opc      583 Sep 10 17:25 airportdb.sql
-rwxr-xr-x. 1 opc opc 35538282 Sep 10 17:25 airportdb@weatherdata@0.tsv.zst
-rwxr-xr-x. 1 opc opc     1752 Sep 10 17:25 airportdb@weatherdata@0.tsv.zst.idx
-rwxr-xr-x. 1 opc opc  4733492 Sep 10 17:25 airportdb@weatherdata@@1.tsv.zst
-rwxr-xr-x. 1 opc opc      240 Sep 10 17:25 airportdb@weatherdata@@1.tsv.zst.idx
-rwxr-xr-x. 1 opc opc      738 Sep 10 17:25 airportdb@weatherdata.json
-rwxr-xr-x. 1 opc opc     1310 Sep 10 17:25 airportdb@weatherdata.sql
-rwxr-xr-x. 1 opc opc     2586 Sep 10 17:25 @.done.json
-rwxr-xr-x. 1 opc opc      859 Sep 10 17:25 @.json
-rwxr-xr-x. 1 opc opc    46349 Sep 10 17:25 @.manifest.json
-rwxr-xr-x. 1 opc opc      240 Sep 10 17:25 @.post.sql
-rwxr-xr-x. 1 opc opc     4884 Sep 10 17:25 README.txt
-rwxr-xr-x. 1 opc opc      240 Sep 10 17:25 @.sql

安装OCI CLI,OCI Linux 7中居然已经安装好了,还是最新版本。

使用OCI CLI将目录中的数据库文件批量上传到对象存储:

$ oci os object bulk-upload -ns ocichina001 -bn airport --src-dir ~/airport-db

然后将数据库从对象存储导入,并不需要事先建库,会自动建库:

$ mysqlsh root@10.0.1.233
MySQL  10.0.1.233:33060+ ssl  JS > util.loadDump("", {threads: 16, loadIndexes: "false", osBucketName: "airport", osNamespace: "ocichina001", ignoreVersion: true})
Loading DDL and Data from OCI ObjectStorage bucket=airport, prefix='' using 16 threads.
Opening dump...
Target is MySQL 8.0.26-u1-cloud (MySQL Database Service). Dump was produced from MySQL 8.0.26-cloud
Fetching dump data from remote location...
Fetching 14 table metadata files for schema `airportdb`...
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL script for schema `airportdb`
[Worker005] Executing DDL script for `airportdb`.`flight`
[Worker003] Executing DDL script for `airportdb`.`employee`
[Worker011] Executing DDL script for `airportdb`.`weatherdata`
[Worker012] Executing DDL script for `airportdb`.`booking`
[Worker014] Executing DDL script for `airportdb`.`flight_log`
[Worker006] Executing DDL script for `airportdb`.`passengerdetails`
[Worker010] Executing DDL script for `airportdb`.`flightschedule`
[Worker002] Executing DDL script for `airportdb`.`airport_reachable`
[Worker013] Executing DDL script for `airportdb`.`airplane_type` (indexes removed for deferred creation)
[Worker009] Executing DDL script for `airportdb`.`airplane`
[Worker015] Executing DDL script for `airportdb`.`airline`
[Worker008] Executing DDL script for `airportdb`.`passenger`
[Worker007] Executing DDL script for `airportdb`.`airport`
[Worker000] Executing DDL script for `airportdb`.`airport_geo`
NOTE: [Worker007] Error processing table `airportdb`.`airport`, will retry after delay: MySQL Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
[Worker013] airportdb@airplane_type@@0.tsv.zst: Records: 342  Deleted: 0  Skipped: 0  Warnings: 0
[Worker008] airportdb@airline@@0.tsv.zst: Records: 113  Deleted: 0  Skipped: 0  Warnings: 0
[Worker000] airportdb@employee@@0.tsv.zst: Records: 1000  Deleted: 0  Skipped: 0  Warnings: 0
[Worker014] airportdb@airplane@@0.tsv.zst: Records: 5583  Deleted: 0  Skipped: 0  Warnings: 0
[Worker012] airportdb@airport@@0.tsv.zst: Records: 9854  Deleted: 0  Skipped: 0  Warnings: 0
[Worker002] airportdb@flightschedule@@0.tsv.zst: Records: 9881  Deleted: 0  Skipped: 0  Warnings: 0
[Worker006] airportdb@passenger@@0.tsv.zst: Records: 36095  Deleted: 0  Skipped: 0  Warnings: 0
[Worker003] airportdb@airport_geo@@0.tsv.zst: Records: 9854  Deleted: 0  Skipped: 0  Warnings: 0
[Worker005] airportdb@passengerdetails@@0.tsv.zst: Records: 36095  Deleted: 0  Skipped: 0  Warnings: 0
[Worker011] airportdb@flight@@0.tsv.zst: Records: 462553  Deleted: 0  Skipped: 0  Warnings: 0
[Worker004] airportdb@weatherdata@0.tsv.zst: Records: 4082688  Deleted: 0  Skipped: 0  Warnings: 0
[Worker004] airportdb@weatherdata@@1.tsv.zst: Records: 543744  Deleted: 0  Skipped: 0  Warnings: 0
[Worker010] airportdb@booking@3.tsv.zst: Records: 2024926  Deleted: 0  Skipped: 0  Warnings: 0
[Worker001] airportdb@booking@0.tsv.zst: Records: 2160442  Deleted: 0  Skipped: 0  Warnings: 0
[Worker009] airportdb@booking@1.tsv.zst: Records: 2159969  Deleted: 0  Skipped: 0  Warnings: 0
[Worker008] airportdb@booking@6.tsv.zst: Records: 2214973  Deleted: 0  Skipped: 0  Warnings: 0
[Worker002] airportdb@booking@10.tsv.zst: Records: 2214991  Deleted: 0  Skipped: 0  Warnings: 0
[Worker003] airportdb@booking@12.tsv.zst: Records: 2214478  Deleted: 0  Skipped: 0  Warnings: 0
[Worker013] airportdb@booking@5.tsv.zst: Records: 2215073  Deleted: 0  Skipped: 0  Warnings: 0
[Worker012] airportdb@booking@9.tsv.zst: Records: 2214959  Deleted: 0  Skipped: 0  Warnings: 0
[Worker000] airportdb@booking@7.tsv.zst: Records: 2214707  Deleted: 0  Skipped: 0  Warnings: 0
[Worker014] airportdb@booking@8.tsv.zst: Records: 2214781  Deleted: 0  Skipped: 0  Warnings: 0
[Worker005] airportdb@booking@13.tsv.zst: Records: 2215250  Deleted: 0  Skipped: 0  Warnings: 0
[Worker006] airportdb@booking@11.tsv.zst: Records: 2215122  Deleted: 0  Skipped: 0  Warnings: 0
[Worker007] airportdb@booking@4.tsv.zst: Records: 2531382  Deleted: 0  Skipped: 0  Warnings: 0
[Worker015] airportdb@booking@2.tsv.zst: Records: 2699650  Deleted: 0  Skipped: 0  Warnings: 0
[Worker011] airportdb@booking@14.tsv.zst: Records: 2215310  Deleted: 0  Skipped: 0  Warnings: 0
[Worker000] airportdb@booking@@24.tsv.zst: Records: 644485  Deleted: 0  Skipped: 0  Warnings: 0
[Worker004] airportdb@booking@15.tsv.zst: Records: 2215229  Deleted: 0  Skipped: 0  Warnings: 0
[Worker010] airportdb@booking@16.tsv.zst: Records: 2215038  Deleted: 0  Skipped: 0  Warnings: 0
[Worker001] airportdb@booking@17.tsv.zst: Records: 2214973  Deleted: 0  Skipped: 0  Warnings: 0
[Worker009] airportdb@booking@18.tsv.zst: Records: 2215177  Deleted: 0  Skipped: 0  Warnings: 0
[Worker002] airportdb@booking@20.tsv.zst: Records: 2214883  Deleted: 0  Skipped: 0  Warnings: 0
[Worker012] airportdb@booking@23.tsv.zst: Records: 2215175  Deleted: 0  Skipped: 0  Warnings: 0
[Worker008] airportdb@booking@19.tsv.zst: Records: 2214884  Deleted: 0  Skipped: 0  Warnings: 0
[Worker003] airportdb@booking@21.tsv.zst: Records: 2214070  Deleted: 0  Skipped: 0  Warnings: 0
[Worker013] airportdb@booking@22.tsv.zst: Records: 2214692  Deleted: 0  Skipped: 0  Warnings: 0
Executing common postamble SQL

37 chunks (59.50M rows, 2.03 GB) for 12 tables in 1 schemas were loaded in 2 min 6 sec (avg throughput 16.15 MB/s)
0 warnings were reported during the load.

验证数据库已创建,表已导入:

 MySQL  10.0.1.233:33060+ ssl  JS > \\sql
Switching to SQL mode... Commands end with ;
 MySQL  10.0.1.233:33060+ ssl  SQL > use airportdb;
Default schema set to `airportdb`.
Fetching table and column names from `airportdb` for auto-completion... Press ^C to stop.

 MySQL  10.0.1.233:33060+ ssl  airportdb  SQL > select count(*) from flight;
+----------+
| count(*) |
+----------+
|   462553 |
+----------+
1 row in set (0.0134 sec)

添加HeatWave Cluster,默认的2个节点即可。

加载数据到HeatWave,这是Autopilot的特性:

 MySQL  10.0.1.233:33060+ ssl  airportdb  SQL > CALL sys.heatwave_load(JSON_ARRAY('airportdb'), NULL);
+------------------------------------------+
| INITIALIZING HEATWAVE AUTO LOAD          |
+------------------------------------------+
| Version: 1.15                            |
|                                          |
| Load Mode: normal                        |
| Load Policy: disable_unsupported_columns |
| Output Mode: normal                      |
|                                          |
+------------------------------------------+
6 rows in set (5.2867 sec)

+-----------------------------------------------------------------------------------------------+
| OFFLOAD ANALYSIS                                                                              |
+-----------------------------------------------------------------------------------------------+
| Verifying input schemas: 1                                                                    |
| User excluded items: 0                                                                        |
|                                                                                               |
| SCHEMA                       OFFLOADABLE    OFFLOADABLE     SUMMARY OF                        |
| NAME                              TABLES        COLUMNS     ISSUES                            |
| ------                       -----------    -----------     ----------                        |
| `airportdb`                           14            104     2 table(s) are partially loadable |
|                                                                                               |
| Total offloadable schemas: 1                                                                  |
|                                                                                               |
+-----------------------------------------------------------------------------------------------+
10 rows in set (5.2867 sec)

+-----------------------------------------------------------------------------------------------------------------------------+
| CAPACITY ESTIMATION                                                                                                         |
+-----------------------------------------------------------------------------------------------------------------------------+
| Default load pool for tables: TRANSACTIONAL                                                                                 |
| Default encoding for string columns: VARLEN (unless specified in the schema)                                                |
| Estimating memory footprint for 1 schema(s)                                                                                 |
|                                                                                                                             |
|                                TOTAL       ESTIMATED       ESTIMATED       TOTAL     DICTIONARY      VARLEN       ESTIMATED |
| SCHEMA                   OFFLOADABLE   HEATWAVE NODE      MYSQL NODE      STRING        ENCODED     ENCODED            LOAD |
| NAME                          TABLES       FOOTPRINT       FOOTPRINT     COLUMNS        COLUMNS     COLUMNS            TIME |
| ------        

以上是关于MySQL HeatWave airportdb Quickstart 实验过程的主要内容,如果未能解决你的问题,请参考以下文章

MySQL HeatWave Quickstart AutoPilot简介

MySQL HeatWave Quickstart (快速入门)

Oracle CloudWorld 2022:重磅推出MySQL湖仓产品-MySQL HeatWave Lakehouse

P1339 [USACO09OCT]热浪Heat Wave

洛谷—— P1339 [USACO09OCT]热浪Heat Wave

MySQL