ORA-00600 [kpdbLoadCbk-bad-obj#] ORA-65106 手动修复PDB container$信息

Posted 2022-11-28 renfengjun

环境：

Oracle Linux7 x64

./opatch lspatches 
27335416;OCW JAN 2018 RELEASE UPDATE 12.2.0.1.180116 (27335416) 
27105253;Database Release Update : 12.2.0.1.180116 (27105253) 

OPatch succeeded.

问题描述：

在我们创建PDB的时候遇到了存储故障， 导致PDB没有创建完成。
PDB 处于invalid 状态。

Pluggable Database DBXXXX with pdb id - 41 is created as UNUSABLE. 
If any errors are encountered before the pdb is marked as NEW, 
then the pdb must be dropped 
local undo-1, localundoscn-0x00000000000000e4 

因为创建PDB是由自动化脚本完成，目录固定，并且应用程序段的连接DSN都是固定的，如果无法使用原有的PDB名称会带来较大的麻烦，所以我们必须要删除后使用API重建PDB。
但是在我们尝试drop的时候报错：

DROP PLUGGABLE DATABASE DBXXXX INCLUDING DATAFILES 

ORA-00600: internal error code, arguments: [kpdbLoadCbk-bad-obj#], [158968], [], [], [], [], [], [], [], [], [], [] 

删除数据文件的时候

alter database datafile '/data/oradata/DBXXXX/system.dbf' offline drop; 
* 
ERROR at line 1: 
ORA-65106: Pluggable database #41 (DBXXXX) is in an invalid state. 

我们尝试过

drop keep datafiles ;
drop including datafiles ;
open ;
open restricted ;

这些命令都会报错 ORA-00600 error .
ORA-00600 [kpdbLoadCbk-bad-obj#]

尝试recover

run 
SET UNTIL TIME "TO_DATE('14-JAN-2019 20:00:00','DD-MON-YYYY HH24:MI:SS')"; 
RECOVER PLUGGABLE DATABASE DBXXXX preview; 
 
RMAN-11003: failure during parse/execution of SQL statement: alter database datafile 4813 online 
ORA-65106: Pluggable database #41 (DBXXXX) is in an invalid state. 

原因：

数据字典信息不一致

select object_id from dba_objects where object_name='DBXXXX'
158968

select obj# from sys.container$ where con_id#=41 ;
4294950953

这里，container$ 里面的obj#应该与dba_objects中的object_id一致。

解决方法：

尝试直接update container$ set obj#=158968 ;没有解决问题，看来还有其他不一致内容。
只能通过pdb$seed的container信息来修复此PDB，以达到删除PDB的目的。

下面是执行步骤
先查询出来container$表中的关于pdb41的con_id#,dbid,con_uid

select con_id# , dbid,con_uid from container$ where con_id#=41 ;

得到结果后

obj#=dba_objects.object_id=158968
con_id#=41
dbid=422445898
con_uid=422445898

下面来修复

备份原有的41号PDB的container$防止后面误操作

SYS@>create table container_41 as select * from container$ where con_id#=41 ; 

Table created. 

copy pdb$seed的container信息

SYS@>create table container_seed as select * from sys.container$ where con_id#=2; 

Table created. 

修改copy出来的pdb$seed的信息,只需要修改以下4项即可

SYS@>update container_seed set obj#=158968,con_id#=41,dbid=422445898,con_uid=422445898; 

1 row updated. 

删除原来的41号PDB container信息

SYS@>delete from sys.container$ where con_id#=41 ; 

1 row deleted. 

SYS@>commit; 

Commit complete. 

插入修复后的41号PDB container信息

SYS@>insert into sys.container$ select * from container_seed ; 

1 row created. 

SYS@>commit ; 

Commit complete. 

最后成功修复并且删除PDB

SYS@>DROP PLUGGABLE DATABASE DBXXXX INCLUDING DATAFILES ; 

Pluggable database dropped. 

删除用来备份的container信息表

drop table container_seed purge ;
drop table container_41 purge ;

这是极端情况，大家遇到问题可以先参考MOS：
Do NOT Delete Data Files at the OS Level in a Multitenant Database before Dropping a PDB (Doc ID 2406001.1)
如果有备份的话，也可以尝试restore
若都无法解决问题，可以参考本文的步骤，修复损坏的PDB container信息。