Oracle LiveLabs实验:Get started with Oracle GoldenGate for Big Data

Posted dingdingfish

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Oracle LiveLabs实验:Get started with Oracle GoldenGate for Big Data相关的知识,希望对你有一定的参考价值。

概述

此实验申请地址在这里,时间为5小时。

实验帮助在这里

此实验的作者是Brian Elliott, Madhu Kumar S,实验整体不错,感谢作者。

就HBASE和Cassandra两个实验有些不同,以后再做一遍。

简介

It easily extends customers’ real-time data integration architectures to big data systems without impacting the performance of the source systems and enables timely business insight for better decision making.

好处是实时的,对源端无影响,与大数据系统的集成。

Our source data changes are generated by Classic GoldenGate and trail file changes will be received into the GoldenGate Microservices Target.

实验环境如下,源就是mysql,目标是MySQL或大数据平台(重点),大数据平台又分为HDFS,Hive,Kafka等。

此实验中的每一个Lab都可以单独运行,前提是需要首先运行Lab 2,即先安装GoldenGate for Big Data。

Lab 1: 环境设置确认

Task 1: Gather compute instance details

找到OCI中实例的IP 地址:129.146.176.171

Task 2: Choose a path

选择连接方法,我选的putty。

Task 2A: Upload Key to Cloud Shell and Connect

略。

Task 2B: Connect via MAC or Windows CYGWIN Emulator

略。

Task 2C: Connect via Windows using Putty

是一个2 OCPU的AMD虚拟机,默认用户是opc:

[opc@quickstart ~]$ hostname
quickstart.cloudera

[opc@quickstart ~]$ free -g
             total       used       free     shared    buffers     cached
Mem:            31          6         24          0          0          1
-/+ buffers/cache:          5         25
Swap:            7          0          7

[opc@quickstart ~]$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 25
model           : 1
model name      : AMD EPYC 7J13 64-Core Processor
stepping        : 1
cpu MHz         : 2545.218
cache size      : 512 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 16
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good extd_apicid unfair_spinlock eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core arat xsaveopt invpcid_single ssbd retpoline ibrs ibpb stibp npt nrip_save fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx clzero arch_capabilities
bogomips        : 5090.43
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

...

processor       : 3
vendor_id       : AuthenticAMD
cpu family      : 25
model           : 1
model name      : AMD EPYC 7J13 64-Core Processor
stepping        : 1
cpu MHz         : 2545.218
cache size      : 512 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 2
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 16
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good extd_apicid unfair_spinlock eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core arat xsaveopt invpcid_single ssbd retpoline ibrs ibpb stibp npt nrip_save fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx clzero arch_capabilities
bogomips        : 5090.43
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Lab 2: 部署GoldenGate for Big Data

Task 0: Running your Lab

先切换到ggadmin用户,显示菜单,按Q退出:

$ sudo su - ggadmin
************************************************************************
*             Oracle GoldenGate for Big Data - Lab Menu                *
************************************************************************
*                                                                      *
* [1] Lab : Deploy GoldenGate for Big Data                             *
* [2] Lab : MySQL --> MySQL one-way replication                        *
* [3] Lab : MySQL --> HDFS (delimited text format )                    *
* [4] Lab : MySQL --> Hive (Avro format)                               *
* [5] Lab : MySQL --> HBase                                            *
* [6] Lab : MySQL --> Kafka (Json format)                              *
* [7] Lab : MySQL --> Cassandra                                        *
* [8] Lab : MySQL --> Kafka (via GGMA for BigData)                     *
* [I] Auto-install OGG for Big Data                                    *
* [R] Lab Reset (Cleanup all procs & files)                            *
*                                                                      *
* [Q] Exit                                                             *
*                                                                      *
************************************************************************
Enter your menu choice [1-7, I, R, Q]: Q

Task 1: Performing the Install

先解压到目录gg4hadoop123010

cd /u01/gg4hadoop123010
tar -xvf  /u01/gg_binaries/gg4hadoop123010/ggs_Adapters_Linux_x64.tar

测试OGG命令行可用:

$ ./ggsci

Oracle GoldenGate Command Interpreter
Version 12.2.0.1.160823 OGGCORE_OGGADP.12.2.0.1.0_PLATFORMS_161019.1437
Linux, x64, 64bit (optimized), Generic on Oct 19 2016 16:01:40
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2016, Oracle and/or its affiliates. All rights reserved.



GGSCI (quickstart.cloudera) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED


GGSCI (quickstart.cloudera) 2> exit

自动安装OGG for BigData,运行labmenu,选I:

$ labmenu
************************************************************************
*             Oracle GoldenGate for Big Data - Lab Menu                *
************************************************************************
*                                                                      *
* [1] Lab : Deploy GoldenGate for Big Data                             *
* [2] Lab : MySQL --> MySQL one-way replication                        *
* [3] Lab : MySQL --> HDFS (delimited text format )                    *
* [4] Lab : MySQL --> Hive (Avro format)                               *
* [5] Lab : MySQL --> HBase                                            *
* [6] Lab : MySQL --> Kafka (Json format)                              *
* [7] Lab : MySQL --> Cassandra                                        *
* [8] Lab : MySQL --> Kafka (via GGMA for BigData)                     *
* [I] Auto-install OGG for Big Data                                    *
* [R] Lab Reset (Cleanup all procs & files)                            *
*                                                                      *
* [Q] Exit                                                             *
*                                                                      *
************************************************************************
Enter your menu choice [1-7, I, R, Q]: I
Auto-installing OGG for Big Data...
Auto-install Complete...
Press any key to continue. . .

About Oracle GoldenGate for Big Data

关键特性:

  • Non-invasive, real-time transactional data streaming
  • Secured, reliable and fault-tolerant data delivery
  • Easy to install, configure and maintain
  • Streams real-time changed data
  • Easily extensible and flexible to stream changed data to other big data targets and message queues

关键好处:

  • Improve IT productivity in integrating with big data systems
  • Use real-time data in big data analytics for more timely and reliable insight
  • Improve operations and customer experience with enhanced business insight
  • Minimize overhead on source systems to maintain high performance

Oracle GoldenGate for Big Data 为 Flume、HDFS、Hive、HBase、Kafka 和 Cassandra 提供优化的高性能交付,以支持客户的实时大数据分析计划。

Oracle GoldenGate for Big Data 包括 Oracle GoldenGate for Java,使客户能够轻松集成到其他大数据系统,例如 Apache Storm、Apache Spark、Oracle NoSQL、MongoDB、SAP HANA、IBM PureData System for Analytics 等。

Oracle GoldenGate for Big Data 的实时数据流平台还允许客户将他们的大数据存储库或大数据湖与他们的生产系统保持同步。

Lab 3: MySQL to MySQL

Introduction

实验架构:

源和目标都是MySQL,OGG抽取的是MySQL的binary log。

Task 0: Running your Lab

从opc用户切换到ggadmin用户:

sudo su - ggadmin

Task 1: Explore GoldenGate Configuration

进入菜单,选择2,看说明,然后选择Q退出:

$ labmenu
************************************************************************
*             Oracle GoldenGate for Big Data - Lab Menu                *
************************************************************************
*                                                                      *
* [1] Lab : Deploy GoldenGate for Big Data                             *
* [2] Lab : MySQL --> MySQL one-way replication                        *
* [3] Lab : MySQL --> HDFS (delimited text format )                    *
* [4] Lab : MySQL --> Hive (Avro format)                               *
* [5] Lab : MySQL --> HBase                                            *
* [6] Lab : MySQL --> Kafka (Json format)                              *
* [7] Lab : MySQL --> Cassandra                                        *
* [8] Lab : MySQL --> Kafka (via GGMA for BigData)                     *
* [I] Auto-install OGG for Big Data                                    *
* [R] Lab Reset (Cleanup all procs & files)                            *
*                                                                      *
* [Q] Exit                                                             *
*                                                                      *
************************************************************************
Enter your menu choice [1-7, I, R, Q]: 2

***************************************************************************************************
*                     MySQL --> MySQL uni-directional replication Lab                             *
***************************************************************************************************
*                                                                                                 *
* - In this lab, you'll configure GG to capture from MySQL (database: ggsource) and               *
*   deliver to MySQL (database: ggtarget)                                                         *
*                                                                                                 *
* - Review the following files before running through the lab                                     *
*   1) cat /u01/gg4mysql/dirprm/create_mysql_gg_procs.oby                                         *
*   2) cat /u01/gg4mysql/dirprm/mgr.prm                                                           *
*   3) cat /u01/gg4mysql/dirprm/extmysql.prm                                                      *
*   4) cat /u01/gg4mysql/dirprm/pmpmysql.prm                                                      *
*   5) cat /u01/gg4mysql/dirprm/repmysql.prm                                                      *
*                                                                                                 *
* - Follow the steps in the Student Handbook to complete this lab                                 *
*                                                                                                 *
***************************************************************************************************

Press ENTER to return to the Lab Menu or Q to Quit:  Q

按照菜单上所列的命令,查看5个文件:

[ggadmin@quickstart ~]$ cd /u01/gg4mysql/dirprm

[ggadmin@quickstart dirprm]$ cat /u01/gg4mysql/dirprm/create_mysql_gg_procs.oby
add extract extmysql, tranlog, begin now
add exttrail ./dirdat/et, extract extmysql, megabytes 10

add extract pmpmysql, EXTTRAILSOURCE ./dirdat/et
add rmttrail ./dirdat/rt, extract pmpmysql, megabytes 10

add replicat repmysql, exttrail ./dirdat/rt nodbcheckpoint

[ggadmin@quickstart dirprm]$ cat /u01/gg4mysql/dirprm/mgr.prm
port 8000

[ggadmin@quickstart dirprm]$ cat /u01/gg4mysql/dirprm/extmysql.prm
extract extmysql

SETENV (MYSQL_UNIX_PORT="/var/lib/mysql/mysql.sock")

sourcedb ggsource, userid ggdemo, password oracle
exttrail ./dirdat/et

TRANLOGOPTIONS ALTLOGDEST "/var/lib/mysql/mysql-bin.index"

NOCOMPRESSUPDATES
NOCOMPRESSDELETES
GETUPDATEBEFORES

table ggsource.*;

[ggadmin@quickstart dirprm]$ cat /u01/gg4mysql/dirprm/pmpmysql.prm
extract pmpmysql

RMTHOST localhost, MGRPORT 8000

exttrail ./dirdat/rt

passthru

REPORTCOUNT EVERY 60 SECONDS, RATE

table ggsource.*;
[ggadmin@quickstart dirprm]$ cat /u01/gg4mysql/dirprm/repmysql.prm
replicat repmysql

SETENV (MYSQL_UNIX_PORT="/var/lib/mysql/mysql.sock")

targetdb ggtarget, userid ggdemo, password oracle
discardfile ./dirrpt/repmysql.dsc, purge

map ggsource.*, target ggtarget.*;

Task 2: Start GoldenGate Processes

切换到OGG Home目录:

$ echo $GGMYSQL
/u01/gg4mysql

$ cd $GGMYSQL

$ ls -l ggsci
-rwxr-x--- 1 ggadmin ggadmin 41542196 Dec 11  2015 ggsci

在OGG命令行中运行命令:

$ ./ggsci

Oracle GoldenGate Command Interpreter for MySQL
Version 12.2.0.1.1 OGGCORE_12.2.0.1.0_PLATFORMS_151211.1401
Linux, x64, 64bit (optimized), MySQL Enterprise on Dec 11 2015 16:23:51
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2015, Oracle and/or its affiliates. All rights reserved.



GGSCI (quickstart.cloudera) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED


GGSCI (quickstart.cloudera) 2> start mgr
Manager started.


GGSCI (quickstart.cloudera) 3> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

GGSCI (quickstart.cloudera) 4> obey ./dirprm/create_mysql_gg_procs.oby

GGSCI (quickstart.cloudera) 5> add extract extmysql, tranlog, begin now

EXTRACT added.


GGSCI (quickstart.cloudera) 6> add exttrail ./dirdat/et, extract extmysql, megabytes 10

EXTTRAIL added.

GGSCI (quickstart.cloudera) 7>

GGSCI (quickstart.cloudera) 7> add extract pmpmysql, EXTTRAILSOURCE ./dirdat/et

EXTRACT added.


GGSCI (quickstart.cloudera) 8> add rmttrail ./dirdat/rt, extract pmpmysql, megabytes 10

RMTTRAIL added.

GGSCI (quickstart.cloudera) 9>

GGSCI (quickstart.cloudera) 9> add replicat repmysql, exttrail ./dirdat/rt nodbcheckpoint

REPLICAT added.

GGSCI (quickstart.cloudera) 10> start extmysql

Sending START request to MANAGER ...
EXTRACT EXTMYSQL starting


GGSCI (quickstart.cloudera) 11> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     EXTMYSQL    00:00:39      00:00:08
EXTRACT     STOPPED     PMPMYSQL    00:00:00      00:00:45
REPLICAT    STOPPED     REPMYSQL    00:00:00      00:00:43

GGSCI (quickstart.cloudera) 12> start pmpmysql

Sending START request to MANAGER ...
EXTRACT PMPMYSQL starting


GGSCI (quickstart.cloudera) 13> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     EXTMYSQL    00:00:00      00:00:05
EXTRACT     RUNNING     PMPMYSQL    00:00:00      00:00:04
REPLICAT    STOPPED     REPMYSQL    00:00:00      00:01:30


GGSCI (quickstart.cloudera) 14> start repmysql

Sending START request to MANAGER ...
REPLICAT REPMYSQL starting


GGSCI (quickstart.cloudera) 15> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     EXTMYSQL    00:00:00      00:00:05
EXTRACT     RUNNING     PMPMYSQL    00:00:00      00:00:03
REPLICAT    RUNNING     REPMYSQL    00:00:00      00:00:04

至此,Manager,Extract,Pump,Replicate进程都启动了。

Task 3: Load Data into Source Database

第一次运行mysqlselect,源与目标的3张表均无数据,然后加载数据,源与目标都有数据,而且一致。说明OGG生效了:

$ mysqlselect

$ loadsource
-------------------------
MySQL Source - loading data ...
-------------------------

... DONE.

$ mysqlselect
-------------------------
MySQL Source
-------------------------
+-----+
|     |
+-----+
| EMP |
+-----+
+-------+--------+-----------+------+------------+---------+---------+--------+
| empno | ename  | job       | mgr  | hiredate   | sal     | comm    | deptno |
+-------+--------+-----------+------+------------+---------+---------+--------+
|  7369 | SMITH  | CLERK     | 7902 | 1993-06-13 |  800.00 |    0.00 |     20 |
|  7499 | ALLEN  | SALESMAN  | 7698 | 1998-08-15 | 1600.00 |  300.00 |     30 |
|  7521 | WARD   | SALESMAN  | 7698 | 1996-03-26 | 1250.00 |  500.00 |     30 |
|  7566 | JONES  | MANAGER   | 7839 | 1995-10-31 | 2975.00 |    NULL |     20 |
|  7654 | MARTIN | SALESMAN  | 7698 | 1998-12-05 | 1250.00 | 1400.00 |     30 |
|  7698 | BLAKE  | MANAGER   | 7839 | 1992-06-11 | 2850.00 |    NULL |     30 |
|  7782 | CLARK  | MANAGER   | 7839 | 1993-05-14 | 2450.00 |    NULL |     10 |
|  7788 | SCOTT  | ANALYST   | 7566 | 1996-03-05 | 3000.00 |    NULL |     20 |
|  7839 | KING   | PRESIDENT | NULL | 1990-06-09 | 5000.00 |    0.00 |     10 |
|  7844 | TURNER | SALESMAN  | 7698 | 1995-06-04 | 1500.00 |    0.00 |     30 |
|  7876 | ADAMS  | CLERK     | 7788 | 1999-06-04 | 1100.00 |    NULL |     20 |
|  7900 | JAMES  | CLERK     | 7698 | 2000-06-23 |  950.00 |    NULL |     30 |
|  7902 | FORD   | ANALYST   | 7566 | 1997-12-05 | 3000.00 |    NULL |     20 |
|  7934 | MILLER | CLERK     | 7782 | 2000-01-21 | 1300.00 |    NULL |     10 |
+-------+--------+-----------+------+------------+---------+---------+--------+
+------+
|      |
+------+
| DEPT |
+------+
+--------+------------+----------+
| deptno | dname      | location |
+--------+------------+----------+
|     10 | Accounting | New York |
|     20 | Research   | Dallas   |
|     30 | Sales      | Chicago  |
|     40 | Operations | Boston   |
+--------+------------+----------+
+----------+
|          |
+----------+
| SALGRADE |
+----------+
+-------+---------+----------+
| grade | losal   | hisal    |
+-------+---------+----------+
|     1 |  700.00 |  1200.00 |
|     2 | 1201.00 |  1400.00 |
|     3 | 1401.00 |  2000.00 |
|     4 | 2001.00 |  3000.00 |
|     5 | 3001.00 | 99999.00 |
+-------+---------+----------+

-------------------------
MySQL Target
-------------------------
+-----+
|     |
+-----+
| EMP |
+-----+
+-------+--------+-----------+------+------------+---------+---------+--------+
| empno | ename  | job       | mgr  | hiredate   | sal     | comm    | deptno |
+-------+--------+-----------+------+------------+---------+---------+--------+
|  7369 | SMITH  | CLERK     | 7902 | 1993-06-13 |  800.00 |    0.00 |     20 |
|  7499 | ALLEN  | SALESMAN  | 7698 | 1998-08-15 | 1600.00 |  300.00 |     30 |
|  7521 | WARD   | SALESMAN  | 7698 | 1996-03-26 | 1250.00 |  500.00 |     30 |
|  7566 | JONES  | MANAGER   | 7839 | 1995-10-31 | 2975.00 |    NULL |     20 |
|  7654 | MARTIN | SALESMAN  | 7698 | 1998-12-05 | 1250.00 | 1400.00 |     30 |
|  7698 | BLAKE  | MANAGER   | 7839 | 1992-06-11 | 2850.00 |    NULL |     30 |
|  7782 | CLARK  | MANAGER   | 7839 | 1993-05-14 | 2450.00 |    NULL |     10 |
|  7788 | SCOTT  | ANALYST   | 7566 | 1996-03-05 | 3000.00 |    NULL |     20 |
|  7839 | KING   | PRESIDENT | NULL | 1990-06-09 | 5000.00 |    0.00 |     10 |
|  7844 | TURNER | SALESMAN  | 7698 | 1995-06-04 | 1500.00 |    0.00 |     30 |
|  7876 | ADAMS  | CLERK     | 7788 | 1999-06-04 | 1100.00 |    NULL |     20 |
|  7900 | JAMES  | CLERK     | 7698 | 2000-06-23 |  950.00 |    NULL |     30 |
|  7902 | FORD   | ANALYST   | 7566 | 1997-12-05 | 3000.00 |    NULL |     20 |
|  7934 | MILLER | CLERK     | 7782 | 2000-01-21 | 1300.00 |    NULL |     10 |
+-------+--------+-----------+------+------------+---------+---------+--------+
+------+
|      |
+------+
| DEPT |
+------+
+--------+------------+----------+
| deptno | dname      | location |
+--------+------------+----------+
|     10 | Accounting | New York |
|     20 | Research   | Dallas   |
|     30 | Sales      | Chicago  |
|     40 | Operations | Boston   |
+--------+------------+----------+
+----------+
|          |
+----------+
| SALGRADE |
+----------+
+-------+---------+----------+
| grade | losal   | hisal    |
+-------+---------+----------+
|     1 |  700.00 |  1200.00 |
|     2 | 1201.00 |  1400.00 |
|     3 | 1401.00 |  2000.00 |
|     4 | 2001.00 |  3000.00 |
|     5 | 3001.00 | 99999.00 |
+-------+---------+----------+

从OGG命令行中确认:

$ ./ggsci

Oracle GoldenGate Command Interpreter for MySQL
Version 12.2.0.1.1 OGGCORE_12.2.0.1.0_PLATFORMS_151211.1401
Linux, x64, 64bit (optimized), MySQL Enterprise on Dec 11 2015 16:23:51
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2015, Oracle and/or its affiliates. All rights reserved.



GGSCI (quickstart.cloudera) 1> stats extmysql total

Sending STATS request to EXTRACT EXTMYSQL ...

Start of Statistics at 2022-03-21 05:51:48.

Output to ./dirdat/et:

Extracting from ggsource.dept to ggsource.dept:

*** Total statistics since 2022-03-21 05:50:10 ***
        Total inserts                                      4.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                   4.00

Extracting from ggsource.emp to ggsource.emp:

*** Total statistics since 2022-03-21 05:50:10 ***
        Total inserts                                     14.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                  14.00

Extracting from ggsource.salgrade to ggsource.salgrade:

*** Total statistics since 2022-03-21 05:50:10 ***
        Total inserts                                      5.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                   5.00

End of Statistics.


GGSCI (quickstart.cloudera) 2> stats repmysql total

Sending STATS request to REPLICAT REPMYSQL ...

Start of Statistics at 2022-03-21 05:52:39.

Replicating from ggsource.dept to ggtarget.dept:

*** Total statistics since 2022-03-21 05:50:13 ***
        Total inserts                                      4.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                   4.00

Replicating from ggsource.emp to ggtarget.emp:

*** Total statistics since 2022-03-21 05:50:13 ***
        Total inserts                                     14.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                  14.00

Replicating from ggsource.salgrade to ggtarget.salgrade:

*** Total statistics since 2022-03-21 05:50:13 ***
        Total inserts                                      5.00
        Total updates                                      0.00
        Total deletes                                      0.00
        Total discards                                     0.00
        Total operations                                   5.00

End of Statistics.

Summary

综上所述,我们将数据加载到 MySQL 数据库 ggsource 中,GG 提取过程 extmysql 捕获了 MySQL binary log中的更改并将它们写入本地 OGG trail文件。 pump进程 pmpmysql 将数据从本地trail(在源上)路由到远程trail(在目标上)。 复制进程 repmysql 读取远程trail文件,并将更改应用到 MySQL 数据库 ggtarget。

Lab 4: MySQL to HDFS

Introduction

实验架构,从MySQL 到 HDFS:

和上一个实验的区别在于replicat进程:

The replicat process rhdfs will read the remote trail files, and write the data to the HDFS target directory /user/ggtarget/hdfs/

也就是将CSV文件写入到HDFS。

Task 0: Running your Lab

切换到ggadmin用户:

sudo su - ggadmin

Task 1: Explore GoldenGate Configuration

进入菜单,输入R重置环境(指清理所有OGG进程和清空MySQL中表的数据),然后输入3查看:

$ labmenu
************************************************************************
*             Oracle GoldenGate for Big Data - Lab Menu                *
************************************************************************
*                                                                      *
* [1] Lab : Deploy GoldenGate for Big Data                             *
* [2] Lab : MySQL --> MySQL one-way replication                        *
* [3] Lab : MySQL --> HDFS (delimited text format )                    *
* [4] Lab : MySQL --> Hive (Avro format)                               *
* [5] Lab : MySQL --> HBase                                            *
* [6] Lab : MySQL --> Kafka (Json format)                              *
* [7] Lab : MySQL --> Cassandra                                        *
* [8] Lab : MySQL --> Kafka (via GGMA for BigData)                     *
* [I] Auto-install OGG for Big Data                                    *
* [R] Lab Reset (Cleanup all procs & files)                            *
*                                                                      *
* [Q] Exit                                                             *
*                                                                      *
************************************************************************
Enter your menu choice [1-7, I, R, Q]: R

Lab Reset: Cleaning up all GG processes and recreating MySQL schemas...
------  THIS MAY TAKE 3-5 MINUTES TO RUN - PLEASE BE PATIENT :)  ------
Lab Reset Complete...
Press any key to continue. . .

************************************************************************
*             Oracle GoldenGate for Big Data - Lab Menu                *
************************************************************************
*                                                                      *
* [1] Lab : Deploy GoldenGate for Big Data                             *
* [2] Lab : MySQL --> MySQL one-way replication                        *
* [3] Lab : MySQL --> HDFS (delimited text format )                    *
* [4] Lab : MySQL --> Hive (Avro format)                               *
* [5] Lab : MySQL --> HBase                                            *
* [6] Lab : MySQL --> Kafka (Json format)                              *
* [7] Lab : MySQL --> Cassandra                                        *
* [8] Lab : MySQL --> Kafka (via GGMA for BigData)                     *
* [I] Auto-install OGG for Big Data                                    *
* [R] Lab Reset (Cleanup all procs & files)                            *
*                                                                      *
* [Q] Exit                                                             *
*                                                                      *
************************************************************************
Enter your menu choice [1-7, I, R, Q]: 3

************************************************************************************

以上是关于Oracle LiveLabs实验:Get started with Oracle GoldenGate for Big Data的主要内容,如果未能解决你的问题,请参考以下文章

Oracle LiveLabs实验: Oracle多租户基础

Oracle LiveLabs实验:Oracle RAC Fundamentals

Oracle LiveLabs实验:Install Oracle Database 21c

Oracle LiveLabs实验:DB Security - Oracle Label Security (OLS)

Oracle LiveLabs实验:Oracle Label Security (OLS)

Oracle LiveLabs实验:Introduction to Oracle Spatial