sqlldr批量生成控制文件

Posted 2020-08-02

参考文档：http://askjoey.blog.51cto.com/7594056/1671852

环境：oracle11g，redhat linux

1.建立基础表

create table TAB_COLUMN_UNION
(
  TAB_COLUMN_ORA    VARCHAR2(4000),
  TAB_COLUMN_SOURCE VARCHAR2(4000),
  TABLE_NAME        VARCHAR2(500)
)
tablespace SDATA
  pctfree 10
  initrans 1
  maxtrans 255
  storage
  (
    initial 16
    next 8
    minextents 1
    maxextents unlimited
  );

table_name 中插入需要生成控制文件的表名。

TAB_COLUMN_ORA 通过存储过程生成表对应的每列

TAB_COLUMN_SOURCE 通过存储过程生成导出成数据文件的语句，这里假设源库和目标库都是oracle

存储过程如下：

CREATE OR REPLACE PROCEDURE ETL_DOWNLOAD as



n_count integer;

v_column_ora varchar2(4000);


begin

update tab_column_union set tab_column_ora=null;

update tab_column_union set TAB_COLUMN_SOURCE=null;


for c1 in (select column_name,a.table_name,column_id,data_type from user_tables a,user_tab_columns b
  where a.table_name=b.TABLE_NAME and b.DATA_TYPE not in (‘CLOB‘,‘BLOB‘) order by a.table_name,column_id)

loop

n_count:=0;

v_column_ora:=null;

select ‘‘‘"‘‘||‘||c1.column_name||‘||‘‘"‘‘‘ into v_column_ora from dual;

update tab_column_union set TAB_COLUMN_SOURCE=TAB_COLUMN_SOURCE||v_column_ora||‘||‘‘,‘‘||‘ where table_name=c1.table_name;

--select count(*) into n_count from tab_column_ex where table_name=c1.table_name and tab_column=c1.column_name;

case when  c1.data_type=‘DATE‘ then

   select c1.column_name||‘ DATE "YYYY-MM-DD HH24:MI:SS"‘ into v_column_ora from dual;

   update tab_column_union set tab_column_ora=tab_column_ora||v_column_ora||‘,‘||chr(10) where table_name=c1.table_name;

when c1.data_type=‘NUMBER‘ then

   select c1.column_name||‘ INTEGER EXTERNAL‘ into v_column_ora from dual;

   update tab_column_union set tab_column_ora=tab_column_ora||v_column_ora||‘,‘||chr(10) where table_name=c1.table_name;

else


 update tab_column_union set tab_column_ora=tab_column_ora||c1.column_name||‘,‘||chr(10) where table_name=c1.table_name;


end case;


END LOOP;

--update tab_column_union t set tab_column_db2=substr(t.tab_column_db2,1,length(t.tab_column_db2)-1);

update tab_column_union t set tab_column_ora=substr(t.tab_column_ora,1,length(t.tab_column_ora)-2);

update tab_column_union t set TAB_COLUMN_SOURCE=substr(t.TAB_COLUMN_SOURCE,1,length(t.TAB_COLUMN_SOURCE)-7);

update tab_column_union t set TAB_COLUMN_SOURCE=‘select ‘||TAB_COLUMN_SOURCE||‘from ‘||table_name||‘;‘;

    COMMIT;

end ETL_DOWNLOAD;

步骤：1.先把需要从源库导入到目标库的表 在 目标库中建立起来，表结构保持一致。

         2.建立基础表TAB_COLUMN_UNION

         插入需要生成控制文件的表名，注意表名要大写，否则执行存储过程会找不到表。

         3.建立存储过程ETL_DOWNLOAD

执行存储过程后会写入TAB_COLUMN_ORA和TAB_COLUMN_SOURCE

再在sqlplus 下执行：

set lines 1000

set pages 0

select ctl_name from (
select ‘spool /home/oracle/‘||lower(table_name)||‘.ctl‘ ctl_name,table_name, 0 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select ‘select ‘‘LOAD DATA‘,table_name,0.1 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select ‘INFILE  ‘‘‘‘$ZM_PATH/‘||table_name||‘_$rq.CSV‘‘‘‘‘,table_name,0.11 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select ‘TRUNCATE into table ‘||table_name,table_name,0.12 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select ‘FIELDS TERMINATED BY  ‘||‘‘‘‘||‘‘‘‘||‘,‘||‘‘‘‘||‘‘‘‘||‘ OPTIONALLY ENCLOSED BY ‘||‘‘‘‘||‘‘‘‘||‘"‘||‘‘‘‘||‘‘‘‘,table_name, 0.2 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select ‘TRAILING NULLCOLS‘,table_name,0.3 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select ‘(‘ ,table_name,0.4 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select b.tab_column_ora,a.table_name,0.5 cid from user_tables a ,tab_column_union b where a.table_name=b.TABLE_NAME
union all
select ‘) ‘‘ from dual;‘ ,table_name,0.6 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select ‘spool off‘,table_name,0.7 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
) aa order by table_name,cid;

把执行结果复制出来写入到一个文件里，文件名ctlout1.sql

通过linux命令

sed  ‘/^$/d‘ ctlout1.sql> ctlout2.sql

去掉ctlout1.sql中的空行。

然后在通过sqlplus连接上目标库，

先执行：

set lines 1000
set pages 0
set echo off
set feedback off
set heading off
set termout off
set trims on

然后再

@ctlout2.sql，会在/home/oracle下生成控制文件。

 

另外如果源库也是oracle，那么可以在sqlplus下执行下列语句生成导出文件。

----生成导出文件

sqlplus  -s 下执行

set lines 1000
set pages 0
set echo off
set feedback off
set heading off
set termout off
set trims on

spool /home/oracle/dataout.sql
select txt_name from (
select ‘spool /datafile/zmcf/ZMCF_CSV/‘||upper(table_name)||‘.CSV‘ txt_name,table_name, 0 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
union all
select t.tab_column_source,t.table_name,0.1 from tab_column_union t
union all
select ‘spool off‘,table_name,0.7 cid from user_tables a where a.TABLE_NAME in (select table_name from tab_column_union)
) aa order by table_name,cid;

spool off

连接到源库，

set lines 1000
set pages 0
set echo off
set feedback off
set heading off
set termout off
set trims on

@dataout.sql

生成数据文件，

需要先建立目录：/datafile/zmcf/ZMCF_CSV

 

另外在我的环境源库字符集gbk，目标库字符集utf8

导出的时候设置export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK

导出txt文件再导入到目标库，

同样在导入时先设置字符集

export NLS_LANG=AMERICAN_AMERICA.AL32UTF8

发现导入时总是报分隔符不正确，怀疑是字符集问题。

file -i 查看导出文件的字符编码，发现内含中文的文件字符集并不是gbk，而是其他的字符集，可能问题就出现在这里。

那么就换种思路，再导出时，设置客户端字符集为：

export NLS_LANG=AMERICAN_AMERICA.AL32UTF8

然后再导出，查看导出文件，含中文的文件字符集已经是utf-8

再把文件正常导入到目标库，发现导入正常。

问题解决