数据大挪移之ora2pg使用指南
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了数据大挪移之ora2pg使用指南相关的知识，希望对你有一定的参考价值。
ora2pg是一款免费的小工具，用于将Oracle数据库迁移到PostgreSQL。ora2pg通过自动扫描oracle数据库获取表结构以及数据，然后生成PostgreSQL数据库的SQL脚本或者直接在PostgreSQL上创建表并插入数据。而且它还支持视图、序列以及分区表，其功能非常强大。这里有一个120G左右的oracle 11g测试库，因业务需求将其迁移至PostgreSQL 10.6。
一、安装ora2pg
1.1 安装OS依赖软件包
[[email protected] ~]# yum install perl-DBD-Pg perl perl-devel perl-DBI perl-CPAN bzip2 perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker1.2 安装Oracle瘦客户端
从Oracle官方网站下载oracle 12c的瘦客户端，然后进行安装并配置。
[[email protected] u02]# la /u02/software/oracle/
total 55M
drwxr-xr-x 2 root root 4.0K Dec 3 15:04 .
drwxr-xr-x 7 root root 76 Dec 3 15:04 ..
-rw-r--r-- 1 root root 51M Nov 28 14:34 oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 593K Nov 28 14:32 oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 1.5M Nov 28 14:32 oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 692K Nov 28 14:32 oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm
-rw-r--r-- 1 root root 922K Nov 28 14:35 oracle-instantclient12.2-tools-12.2.0.1.0-1.x86_64.rpm
[[email protected] ~]# yum -y localinstall /u02/software/oracle/oracle-instantclient12.2-*.rpm
[[email protected] ~]# mkdir -p /usr/lib/oracle/12.2/client64/network/admin编辑root环境变量，加入以下内容：
[[email protected] ~]# vi .bash_profile
export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib
export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK
export TNS_ADMIN=/usr/lib/oracle/12.2/client64/network/admin
export ORACLE_HOME=/usr/lib/oracle/12.2/client64/
export PATH=$PATH:$ORACLE_HOME/bin:$HOME/bin
[[email protected] ~]# source .bash_profile1.3 安装ora2pg
[[email protected] ~]# wget https://sourceforge.net/projects/ora2pg/files/19.1/ora2pg-19.1.tar.bz2
[[email protected] ~]# tar -xjf ora2pg-19.1.tar.bz2
[[email protected] ~]# cd ora2pg-19.1
[[email protected] ora2pg-19.1]# perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for Ora2Pg
Done...
------------------------------------------------------------------------------
Please read documentation at http://ora2pg.darold.net/ before asking for help
------------------------------------------------------------------------------
Now type: make && make install
[[email protected] ora2pg-19.1]# make && make install
cp lib/Ora2Pg.pm blib/lib/Ora2Pg.pm
cp lib/Ora2Pg/GEOM.pm blib/lib/Ora2Pg/GEOM.pm
cp lib/Ora2Pg/PLSQL.pm blib/lib/Ora2Pg/PLSQL.pm
cp lib/Ora2Pg/mysql.pm blib/lib/Ora2Pg/MySQL.pm
cp scripts/ora2pg blib/script/ora2pg
/usr/bin/perl -MExtUtils::MY -e ‘MY->fixin(shift)‘ -- blib/script/ora2pg
cp scripts/ora2pg_scanner blib/script/ora2pg_scanner
/usr/bin/perl -MExtUtils::MY -e ‘MY->fixin(shift)‘ -- blib/script/ora2pg_scanner
Manifying blib/man3/ora2pg.3
Installing /usr/local/share/perl5/Ora2Pg.pm
Installing /usr/local/share/perl5/Ora2Pg/GEOM.pm
Installing /usr/local/share/perl5/Ora2Pg/PLSQL.pm
Installing /usr/local/share/perl5/Ora2Pg/MySQL.pm
Installing /usr/local/share/man/man3/ora2pg.3
Installing /usr/local/bin/ora2pg
Installing /usr/local/bin/ora2pg_scanner
Installing default configuration file (ora2pg.conf.dist) to /etc/ora2pg
Appending installation info to /usr/lib64/perl5/perllocal.pod1.4 安装DBD::Oracle
[[email protected] ~]# perl -MCPAN -e shell
Terminal does not support AddHistory.
cpan shell -- CPAN exploration and modules installation (v1.9800)
Enter ‘h‘ for help.
cpan[1]> get DBD::Oracle
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/01mailrc.txt.gz
Reading ‘/root/.cpan/sources/authors/01mailrc.txt.gz‘
............................................................................DONE
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/modules/02packages.details.txt.gz
Reading ‘/root/.cpan/sources/modules/02packages.details.txt.gz‘
Database was generated on Wed, 28 Nov 2018 23:54:25 GMT
HTTP::Date not available
.............
New CPAN.pm version (v2.16) available.
[Currently running version is v1.9800]
You might want to try
install CPAN
reload cpan
to both upgrade CPAN.pm and run the new version without leaving
the current session.
...............................................................DONE
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/modules/03modlist.data.gz
Reading ‘/root/.cpan/sources/modules/03modlist.data.gz‘
DONE
Writing /root/.cpan/Metadata
Running get for module ‘DBD::Oracle‘
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz
Fetching with HTTP::Tiny:
http://mirror.truenetwork.ru/CPAN/authors/id/Z/ZA/ZARQUON/CHECKSUMS
Checksum for /root/.cpan/sources/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz ok
Scanning cache /root/.cpan/build for sizes
DONE
cpan[2]> quit
Terminal does not support GetHistory.
Lockfile removed.
[[email protected] ~]# cd .cpan/build/DBD-Oracle-1.76-sU5yu5/
[[email protected] DBD-Oracle-1.76-sU5yu5]# perl Makefile.PL
[[email protected] DBD-Oracle-1.76-sU5yu5]# make && make install1.5 创建ora2pg配置文件
首先从示例文件复制一份，然后根据实际情况更改配置文件相关参数：
[[email protected] ~]# cp /etc/ora2pg/ora2pg.conf.dist /etc/ora2pg/ora2pg.conf
[[email protected] ~]# vi /etc/ora2pg/ora2pg.conf
--Oracle连接参数
ORACLE_HOME /usr/lib/oracle/12.2/client64
ORACLE_DSN dbi:Oracle:host=172.16.100.19;sid=kkdb;port=1521
ORACLE_USER system --如果这里的用户是sys、system，则user_grants选项必须设置为1
ORACLE_PWD abcABC12
USER_GRANTS 1
ORA_INITIAL_COMMAND commit
--schema参数
EXPORT_SCHEMA 1
SCHEMA XXKIN
CREATE_SCHEMA 0
COMPILE_SCHEMA 0
PG_SCHEMA public
--导出对象(根据实际情况选择)
TYPE TABLE COPY VIEW TYPE
DISABLE_COMMENT 0
EXTERNAL_TO_FDW 1
TRUNCATE_TABLE 0
USE_TABLESPACE 0
REORDERING_COLUMNS 0
--约束参数
KEEP_PKEY_NAMES 0
PKEY_IN_CREATE 0
FKEY_ADD_UPDATE never
FKEY_DEFERRABLE 0
DEFER_FKEY 0
DROP_FKEY 1
--触发器参数
DISABLE_SEQUENCE 1
DISABLE_TRIGGERS 1
--对象修改参数
PRESERVE_CASE 0
INDEXES_RENAMING 0
USE_INDEX_OPCLASS 0
PREFIX_PARTITION 0
DISABLE_PARTITION 1
WITH_OID 0
ORA_RESERVED_WORDS audit,comment
USE_RESERVED_WORDS 1
DISABLE_UNLOGGED 0
--PostgreSQL参数
PG_DSN dbi:Pg:dbname=kkdb;host=192.168.120.149;port=5432
PG_USER xxkin
PG_PWD redhat
OUTPUT xxkin.sql
OUTPUT_DIR /tmp
BZIP2
FILE_PER_CONSTRAINT 0
FILE_PER_INDEX 0
FILE_PER_FKEYS 0
FILE_PER_TABLE 0
FILE_PER_FUNCTION 0
STOP_ON_ERROR 1
COPY_FREEZE 0
CREATE_OR_REPLACE 1
--TYPE参数
PG_NUMERIC_TYPE 1
PG_INTEGER_TYPE 1
DEFAULT_NUMERIC bigint
ENABLE_MICROSECOND 1
TO_NUMBER_CONVERSION numeric
GEN_USER_PWD 0
FORCE_OWNER 0
FORCE_SECURITY_INVOKER 0
--数据参数
DATA_LIMIT 10000
NOESCAPE 0
TRANSACTION serializable
STANDARD_CONFORMING_STRINGS 1
NO_LOB_LOCATOR 1
XML_PRETTY 0
LOG_ON_ERROR 0
TRIM_TYPE BOTH
INTERNAL_DATE_MAX 49
FUNCTION_CHECK 1
--Export/Import性能参数
JOBS 1
ORACLE_COPIES 1
PARALLEL_TABLES 1
DEFAULT_PARALLELISM_DEGREE 0
PARALLEL_MIN_ROWS 100000
DROP_INDEXES 0
SYNCHRONOUS_COMMIT 0
--PLSQL参数
EXPORT_INVALID 0
PLSQL_PGSQL 1
NULL_EQUAL_EMPTY 0
EMPTY_LOB_NULL 0
PACKAGE_AS_SCHEMA 1
REWRITE_OUTER_JOIN 1
FUNCTION_STABLE 1
COMMENT_COMMIT_ROLLBACK 0
COMMENT_SAVEPOINT 0
USE_ORAFCE 0
AUTONOMOUS_TRANSACTION 1
--ASSESSMENT参数
ESTIMATE_COST 0
COST_UNIT_VALUE 5
DUMP_AS_html 0
TOP_MAX 10
HUMAN_DAYS_LIMIT 5
--PostgreSQL特性参数
PG_SUPPORTS_WHEN 1
PG_SUPPORTS_INSTEADOF 1
PG_SUPPORTS_MVIEW 1
PG_SUPPORTS_CHECKOPTION 1
PG_SUPPORTS_IFEXISTS 1
PG_SUPPORTS_LATERAL 1
PG_SUPPORTS_PARTITION 1
BITMAP_AS_GIN 1
PG_BACKGROUND 0
PG_SUPPORTS_SUBSTR 1
PG_SUPPORTS_NAMED_OPERATOR 1
PG_SUPPORTS_IDENTITY 1
PG_SUPPORTS_PROCEDURE 0
--SPATIAL参数
AUTODETECT_SPATIAL_TYPE 1
CONVERT_SRID 1
DEFAULT_SRID 4326
GEOMETRY_EXTRACT_TYPE INTERNAL
--FDW参数
FDW_SERVER orcl
--MySQL参数
MYSQL_PIPES_AS_CONCAT 0
MYSQL_INTERNAL_EXTRACT_FORMAT 0到此，整个ora2pg的安装配置已完成。
二、使用ora2pg
2.1 导出全部表结构
如果在配置文件里指定的type包含了COPY，导完表结构后，表的数据也会随之导出，加-t参数即可只导出表结构不导出数据。
[[email protected] ~]# ora2pg -t table view
[========================>] 777/777 tables (100.0%) end of scanning.
[========================>] 777/777 tables (100.0%) end of table export.
[========================>] 0/0 views (100.0%) end of output. 导出完成后，在PostgreSQL中创建数据库以及对应的表。
-bash-4.2$ psql
psql (10.6)
Type "help" for help.
postgres=# CREATE USER xxkin WITH password ‘redhat‘ CREATEDB SUPERUSER replication bypa***ls createrole login;
postgres=# create database kkdb;
postgres=# alter database kkdb owner to xxkin;
postgres=# c kkdb
You are now connected to database "kkdb" as user "postgres".
kkdb=# i /tmp/output.sqlora2pg有很多参数可供使用，具体请参考其帮助文件。
2.2 通过ora2pg迁移数据
之前已创建好数据库以及相关表结构，现在只需通过ora2pg将数据同步到PostgreSQL数据库上即可。
[[email protected] ~]# ora2pg -d -t copy
[2018-11-30 11:38:51] Ora2Pg version: 19.1
[2018-11-30 11:38:51] Trying to connect to database: dbi:Oracle:host=172.16.100.19;sid=kkdb;port=1521
[2018-11-30 11:38:51] Isolation level: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
[2018-11-30 11:38:51] DEBUG: executing initial command to Oracle: commit
[2018-11-30 11:38:51] Retrieving table information...
[2018-11-30 11:40:55] [1] Scanning table ABC_DEL_CLASS (1 rows)...
[2018-11-30 11:40:55] [2] Scanning table ACTIVITY_DATA (41 rows)...
[2018-11-30 11:40:55] [3] Scanning table ACTIVITY_GBEAN_STAT (243897 rows)...
......如果在迁移过程中，需要排除某些表，加-e参数，如下:
[[email protected] ~]# ora2pg -d -t copy -e usr_dx_tbs,hr_info_pg,op_log总结：
以下是在使用过程遇到的问题，提出的建议：
1、先导出表结构，然后在同步数据；
2、在导出表结构的时候禁用分区表；
3、注意PostgreSQL的保留的关键字是否为oracle表字段名；
4、删除所有外键约束。
5、直接通过ora2pg迁移数据，不要将oracle表数据导出为csv格式，然后再导入PostgreSQL数据库。
参考文献：ora2pg DOCUMENTATION
以上是关于数据大挪移之ora2pg使用指南的主要内容，如果未能解决你的问题，请参考以下文章