数据泵导入数据,完成测试数据的迁移

Posted lvcha001

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了数据泵导入数据,完成测试数据的迁移相关的知识,希望对你有一定的参考价值。

需求:

客户环境linux oracle 11.2.0.4单实例

新环境linux redhat oracle 11.2.0.1 单实例

客户需要准备什么一套新的测试库,当前测试库有软件,但是无DB.整个库都需要迁移过去。

 

流程及思路:

1.搭建测试库DB,字符集与生产环境相同

2.创建一个测试表,模拟导出导入之后,观察中文能否读写!

3.正式迁移导入测试数据!

  a)提前创建用户表空间及授予DBA权限;

  b)导入测试数据

  c)检测确认

 

 

一、测试库DB创建

先看看物理内存多大,去判断测试库选择多大内存! 
$ free -m total used free shared buffers cached Mem: 16079 14978 1100 0 137 13930 -/+ buffers/cache: 910 15168 Swap: 8039 0 8039 $ echo 3 > /proc/sys/vm/drop_caches # free -g total used free shared buffers cached Mem: 15 0 15 0 0 0 -/+ buffers/cache: 0 15 Swap: 7 0 7
发现测试库有15G内存大小,但是内存都已使用,和客户沟通是新环境随便搞之后,直接清理cache内存

软件已经安装,检查后,查询oralce_home,及操作系统空间是否充足,查询后是500G,足够。

$echo ORACLE_HOME
# su - psoft
--配置数据库响应文件参数
echo ‘[GENERAL]
RESPONSEFILE_VERSION = "11.2.0"
OPERATION_TYPE = "createDatabase"
[CREATEDATABASE]
GDBNAME = "tt1"
SID = "tt1"
TEMPLATENAME = "General_Purpose.dbc"
STORAGETYPE=FS
DATAFILEDESTINATION =/xx/datafile
CHARACTERSET = "UTF8"
NATIONALCHARACTERSET= "AL16UTF16"
TOTALMEMORY = "8192"
SYSPASSWORD = "oracle"
SYSTEMPASSWORD = "oracle" ‘>>/home/xx/dbca_install.rsp

上述静默安装参数简要说明,字符集UTF8,内存8G,sys密码oracle,数据库名称tt1


--静默安装数据库
$ORACLE_HOME/bin/dbca -silent -responseFile /home/xx/dbca_install.rsp

--静默安装监听配置
$ netca -silent -responsefile $ORACLE_HOME/assistants/netca/netca.rsp

安装完毕!

日志格式化,具体操作可以找以前写的博客日志格式化内容,操作啥的不细说了。

alter database add logfile thread 1 group 4(‘/home/xx/hcm/oracledb/datafile/tt1/redo04.log‘) size 200m;
alter database add logfile thread 1 group 5(‘/home/xx/hcm/oracledb/datafile/tt1/redo05.log‘) size 200m;
alter database add logfile thread 1 group 6(‘/home/xx/hcm/oracledb/datafile/tt1/redo06.log‘) size 200m;
alter database add logfile thread 1 group 7(‘/home/xx/hcm/oracledb/datafile/tt1/redo07.log‘) size 200m;
alter database add logfile thread 1 group 8(‘/home/xx/hcm/oracledb/datafile/tt1/redo08.log‘) size 200m;
alter database add logfile thread 1 group 9(‘/home/xx/hcm/oracledb/datafile/tt1/redo09.log‘) size 200m;
alter database drop logfile group 1;
alter database drop logfile group 2;
alter database drop logfile group 3;

 

二、测试导入

生产库创建对象(得到客户认可)模拟后进行删除,测试库进行导入。
生产库
> env|grep LANG LANG=en_US.UTF-8 NLS_LANG=AMERICAN_AMERICA.UTF8

SQL> create table system.a_yz_test(id int,cc varchar2(20));
SQL> insert into system.a_yz_test values(1,‘呵呵‘);
1 row created.
SQL> commit;
Commit complete.
SQL> select * from system.a_yz_test;
ID CC
---------- --------------------
1 呵呵

SQL>select * from dba_directories;
SQL>create directory dump as ‘/bak‘;

> expdp ‘/ as sysdba‘ directory=dump dumpfile=yang%u.dmp logfile=yang.log TABLES=SYSTEM.A_YZ_TEST
> scp /bak/yang01.dmp root@xx:/home/xx/.

测试库导入

SQL>select * from dba_directories;
SQL>create directory dump as ‘/home/xx‘;
$ env|grep LANG
NLS_LANG=simplified chinese_china.al32utf8
LANG=zh_US.utf-8
$ export ORACLE_SID=tt1
$ impdp ‘/ as sysdba‘ directory=dump dumpfile=yang%u.dmp logfile=yang.log
ORA-39083: 对象类型 PRE_TABLE_ACTION 创建失败, 出现错误:
ORA-23327: 导入的延迟 rpc 数据与导入 db 的GLOBAL NAME不匹配
ORA-39083: 对象类型 PRE_TABLE_ACTION 创建失败, 出现错误:
ORA-23327: 导入的延迟 rpc 数据与导入 db 的GLOBAL NAME不匹配
SQL> select * from system.a_yz_test;
ID CC
---------- ------------------------------------------------------------
1 呵呵   --说明导入正常,可以进行全库导出导入了。

三、全库导出导入

流程比较细
1.生产库查询需要导出的用户名称
SQL> select username ,DEFAULT_TABLESPACE,to_char(CREATED,‘yyyy-mm-dd hh24:mi:ss‘) as "c_date" from dba_users
where ACCOUNT_STATUS=‘OPEN‘ order by 3;
排除SYS,SYSTEM均需要迁移
2.需要考虑迁移的数据量,大小200g以内,生产库及测试库操作系统本地均大于200g【小于需要考虑使用压缩导出】

SQL> select sum(bytes)/1024/1024/1024 from dba_segments;

SQL> select sum(bytes)/1024/1024/1024 from dba_data_files;

 3.查询需要迁移用户使用了什么表空间

SQL> select owner,sum(bytes)/1024/1024/1024,tablespace_name from dba_segments where owner in(‘A‘,‘H‘)
group by owner,tablespace_name;

SQL>select tablespace_name,sum(bytes)/1024/1024/1024 from dba_segments where owner in(‘PS‘,‘SYSADM‘,‘PEOPLE‘,‘RBDB‘,‘G3DB‘,‘KAOQIN_DB‘,‘DBMONITOR‘,‘RBHRDB‘,‘HRFANRAN‘)
group by tablespace_name order by 2;

本次客户比较奇葩,足足有50多个表空间,实际操作,使用了remap_tablespace进行转换后,导入还是报错,create table xx tablespace ccx 不存在,导入报错!
因此这次迁移,是同步所有生产环境的数据库进行操作!!!
那么也就是需要迁移同步表空间! 我们发现表空间使用最大<30g,因此只需要对每个表空间创建一个数据文件即可!

select distinct ‘create tablespace ‘||tablespace_name||‘ datafile ‘‘/home/psoft/hcm/oracledb/datafile/tt1/‘||tablespace_name||‘.dbf‘‘ size 1m autoextend on next 50m;‘ as "sql_text" from dba_data_files
group by tablespace_name;

生产环境执行,输出脚本内容,测试库执行,创建表空间!

4.测试库导入的用户密码呢?

  通过数据泵impdp 参数sqlfile  以及include=user  可以得到创建用户的信息,编辑脚本后,可以创建用户,客户也不需要让测试人员修改密码,直接用生产环境的用户密码操作即可!

impdp ‘/ as sysdba‘ dumpfile=tt%u.dmp directory=dump logfile=tt_user.log include=user sqlfile=impdp_user.sql



真正的操作
1.导出

> expdp / as sysdba‘ directory=dump dumpfile=tt%u.dmp logfile=tt.log SCHEMAS=A,B,C PARALLEL=4 . . exported "SYSCC"."TBCC" 0 KB 0 rows Processing object type SCHEMA_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT Processing object type SCHEMA_EXPORT/TABLE/COMMENT Processing object type SCHEMA_EXPORT/PACKAGE/PACKAGE_SPEC Processing object type SCHEMA_EXPORT/FUNCTION/FUNCTION Processing object type SCHEMA_EXPORT/PROCEDURE/PROCEDURE Processing object type SCHEMA_EXPORT/PACKAGE/COMPILE_PACKAGE/PACKAGE_SPEC/ALTER_PACKAGE_SPEC Processing object type SCHEMA_EXPORT/FUNCTION/ALTER_FUNCTION Processing object type SCHEMA_EXPORT/PROCEDURE/ALTER_PROCEDURE Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX Processing object type SCHEMA_EXPORT/TABLE/INDEX/FUNCTIONAL_INDEX/INDEX Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/FUNCTIONAL_INDEX/INDEX_STATISTICS Processing object type SCHEMA_EXPORT/VIEW/VIEW Processing object type SCHEMA_EXPORT/VIEW/GRANT/OWNER_GRANT/OBJECT_GRANT Processing object type SCHEMA_EXPORT/PACKAGE/PACKAGE_BODY Processing object type SCHEMA_EXPORT/TABLE/TRIGGER Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS Processing object type SCHEMA_EXPORT/MATERIALIZED_VIEW Master table "SYS"."SYS_EXPORT_SCHEMA_01" successfully loaded/unloaded ****************************************************************************** Dump file set for SYS.SYS_EXPORT_SCHEMA_01 is: /bak/tt01.dmp /bak/tt02.dmp /bak/tt03.dmp /bak/tt04.dmp Job "SYS"."SYS_EXPORT_SCHEMA_01" successfully completed at Sat Apr 4 20:08:35 2020 elapsed 0 00:12:20
2.SCP !省略
3.测试库
3.1 创建表空间
3.2 准备创建用户

impdp ‘/ as sysdba‘ dumpfile=tt%u.dmp directory=dump logfile=tt_user.log include=user sqlfile=impdp_user.sql

编辑create_user.sql脚本

修改默认表空间名称!

CREATE USER "A" IDENTIFIED BY VALUES ‘XXXX40F0056DAB96165BC21989CC23;0AE52ADF439D30BD‘ default tablespace users
TEMPORARY TABLESPACE "TEMP";

随后创建脚本
3.3 授权
有的用户没权限,导入会报错,最简单粗暴的就是直接grant dba

SQL> select ‘grant dba to ‘||USERNAME||‘;‘ from dba_users where ACCOUNT_STATUS=‘OPEN‘;
SQL> grant dba to a;
授权成功。

3.4 导入
本次导入很多次失败
failed 1
使用多次转换,但是有些表在dba_segments 不存在,并未真正创建segments
但是呢,表的元数据空表,有其他的表空间,当时没有创建dba_segments不存在的表空间,最终导致导入失败。

nohup time impdp ‘/ as sysdba‘ dumpfile=tt%u.dmp directory=dump logfile=tt.log parallel=8 EXCLUDE=STATISTICS REMAP_TABLESPACE=BNAPP:USERS,PSIMAGE2:USERS,HRLARGE:USERS,COAPP:USERS,EOAPP:USERS,PTTLRG:USERS,HRAPP2:USERS,PTTBL:USERS,PSIMAGE:USERS,PTAMSG:USERS,PTWORK:USERS,PTPRJWK:USERS,CULARG1:USERS,SYSAUX:USERS,PTPRC:USERS,CULARGE:USERS,EPAPP:USERS,PAAPP:USERS,PIAPP:USERS,HRAPP3:USERS,HRAPP6:USERS,SAAPP:USERS,HRSWORK:USERS,BNLARGE:USERS,TLAPP:USERS,HPAPP:USERS,GIAPP:USERS,HRSAPP:USERS,AAAPP:USERS,PTLOCK:USERS,CULARG2:USERS,PSDEFAULT:USERS,PSINDEX:USERS,HRAPP:USERS,PTAPPE:USERS,PTAPP:USERS,GPAPP:USERS,EOEWAPP:USERS,ERAPP:USERS,CCAPP:USERS,HRWORK:USERS,ADAPP:USERS,TLLARGE:USERS,PYAPP:USERS,PTRPTS:USERS,HRAPP7:USERS,EOECAPP:USERS,EOECLRG:USERS,EOLARGE:USERS,FGAPP:USERS,HRAPP4:USERS,HRAPP1:USERS,AALARGE:USERS,HRAPP5:USERS,FAAPP:USERS,STAPP:USERS,CULARG3:USERS,EOIUAPP:USERS,PTTREE:USERS,PY0LRG:USERS metrics=yes &

结束导出进程

$ impdp ‘/ as sysdba‘ attach=SYS_IMPORT_FULL_01
Import> kill
是否确实要停止此作业 ([Y]/N): y


select ‘drop table ‘||owner||‘.‘||table_name||‘ purge;‘ from dba_tables where owner in(select USERNAME from dba_users where ACCOUNT_STATUS=‘OPEN‘ and username not in(‘SYS‘,‘SYSTEM‘))
group by owner,table_name ;

删除! drop user xx cascade;即可,但是导入对象太多,删除太慢!!!

  failed 2 

nohup time impdp ‘/ as sysdba‘ dumpfile=tt%u.dmp directory=dump logfile=tt.log parallel=8 EXCLUDE=STATISTICS

  并行太多!!! 导入20分钟???  session event???   还在分配通道!!!

SQL_ID EVENT
-- -------------------- ------------------------------ --
afcz9s4uazbpk wait for unread message on bro adcast channel

afcz9s4uazbpk wait for unread message on bro adcast channel

 停止导出进程,删除表,删除用户及数据,重新导入

 成功! 导出多少进程,导入多少! 排除统计信息!

nohup time impdp ‘/ as sysdba‘ dumpfile=tt%u.dmp directory=dump logfile=tt.log parallel=4 EXCLUDE=STATISTICS &

 确认没啥问题,OK,DBA权限是否回收,客户说测试库,那就不需要咯。

 

 

以上是关于数据泵导入数据,完成测试数据的迁移的主要内容,如果未能解决你的问题,请参考以下文章

数据泵导出导入

Oracle 数据泵迁移用户创建 SQL语句

Oracle 数据泵迁移用户创建 SQL语句

expdp/impdp 数据泵导入导出

浅谈oracle逻辑备份数据泵备份及冷备份

ORACLE 数据泵导入导出数据