记一次数据库异常恢复

Posted dayu-liu

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了记一次数据库异常恢复相关的知识,希望对你有一定的参考价值。

 

Last login: Fri Jul 20 16:54:26 2018 from 9.1.173.100
[[email protected] ~]# su - oracle
[ctcdb:oracle]:/home/oracle>sqlplus "/ as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jul 20 16:55:38 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1.3683E+11 bytes
Fixed Size                  2270360 bytes
Variable Size            1.2616E+10 bytes
Database Buffers         1.2402E+11 bytes
Redo Buffers              189456384 bytes
Database mounted.
ORA-00742: Log read detects lost write in thread %d sequence %d block %d
ORA-00312: online log 1 thread 1: ‘+DATADG/orcl/redo01.log‘


SQL> shutdown abort
ORACLE instance shut down.
SQL> startup 
ORACLE instance started.

Total System Global Area 1.3683E+11 bytes
Fixed Size                  2270360 bytes
Variable Size            1.2616E+10 bytes
Database Buffers         1.2402E+11 bytes
Redo Buffers              189456384 bytes
Database mounted.
ORA-00742: Log read detects lost write in thread %d sequence %d block %d
ORA-00312: online log 1 thread 1: ‘+DATADG/orcl/redo01.log‘


SQL>  RECOVER DATABASE USING BACKUP CONTROLFILE UNTIL CANCEL
ORA-00279: change 361503907875 generated at 07/20/2018 14:00:39 needed for
thread 1
ORA-00289: suggestion : +ARCHDG
ORA-15173: entry ‘ORCL‘ does not exist in directory ‘/‘
ORA-00280: change 361503907875 for thread 1 is in sequence #9922


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: cannot open archived log ‘+ARCHDG‘
ORA-17503: ksfdopn:2 Failed to open file +ARCHDG
ORA-15045: ASM file name ‘+ARCHDG‘ is not in reference form


ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: ‘+DATADG/orcl/datafile/system.259.978924803‘


SQL> ALTER DATABASE OPEN RESETLOGS;
ALTER DATABASE OPEN RESETLOGS
*
ERROR at line 1:
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: ‘+DATADG/orcl/datafile/system.259.978924803‘


SQL> shutdown abort;
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 1.3683E+11 bytes
Fixed Size                  2270360 bytes
Variable Size            1.2616E+10 bytes
Database Buffers         1.2402E+11 bytes
Redo Buffers              189456384 bytes
Database mounted.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
[ctcdb:oracle]:/home/oracle>lsnctl status
-bash: lsnctl: command not found
[ctcdb:oracle]:/home/oracle>lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 20-JUL-2018 17:06:20

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                20-JUL-2018 16:53:40
Uptime                    0 days 0 hr. 12 min. 39 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0.4/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/ctcdb/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ctcdb)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM", status READY, has 1 handler(s) for this service...
Service "orcl" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully
[ctcdb:oracle]:/home/oracle>sqlplus "/ as sysdba"

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jul 20 17:07:00 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> 
SQL> 
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL> recover database using backup controlfile;     
ORA-00279: change 361503907875 generated at 07/20/2018 14:00:39 needed for
thread 1
ORA-00289: suggestion : +ARCHDG
ORA-15173: entry ‘ORCL‘ does not exist in directory ‘/‘
ORA-00280: change 361503907875 for thread 1 is in sequence #9922


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
AUTO
ORA-00308: cannot open archived log ‘+ARCHDG‘
ORA-17503: ksfdopn:2 Failed to open file +ARCHDG
ORA-15045: ASM file name ‘+ARCHDG‘ is not in reference form


ORA-00308: cannot open archived log ‘+ARCHDG‘
ORA-17503: ksfdopn:2 Failed to open file +ARCHDG
ORA-15045: ASM file name ‘+ARCHDG‘ is not in reference form


SQL> alter database open RESETLOGS;
alter database open RESETLOGS
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: ‘+DATADG/orcl/datafile/system.259.978924803‘


SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
[ctcdb:oracle]:/home/oracle>ps -ef | grep ckpt
grid      11493      1  0 16:54 ?        00:00:00 asm_ckpt_+ASM
oracle    36151      1  0 17:05 ?        00:00:00 ora_ckpt_orcl
oracle   102216  11677  0 17:21 pts/2    00:00:00 grep ckpt
[ctcdb:oracle]:/home/oracle>cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.71.73 ctcdb
[ctcdb:oracle]:/home/oracle>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jul 20 17:22:12 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> select open_mode from v$database;

OPEN_MODE
----------------------------------------
MOUNTED

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL> 
SQL> 
SQL> 
SQL> 
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done


SQL> show parameter background

NAME                                 TYPE
------------------------------------ ----------------------
VALUE
------------------------------
background_core_dump                 string
partial
background_dump_dest                 string
/u01/app/oracle/diag/rdbms/orc
l/orcl/trace
SQL> set line 300
SQL> /
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL> recover database using backup controlfile;
ORA-00279: change 361503907875 generated at 07/20/2018 14:00:39 needed for thread 1
ORA-00289: suggestion : +ARCHDG
ORA-15173: entry ‘ORCL‘ does not exist in directory ‘/‘
ORA-00280: change 361503907875 for thread 1 is in sequence #9922


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel 
Media recovery cancelled.
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done


SQL> shutdown immediate
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
[ctcdb:oracle]:/home/oracle>exit
logout
[[email protected] ~]# su - grid
[ctcdb:grid]:/home/grid>which crsctl
/u01/app/11.2.0.4/grid/bin/crsctl
[ctcdb:grid]:/home/grid>exit
logout
[[email protected] ~]#  /u01/app/11.2.0.4/grid/bin/crsctl stop crs
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.
[[email protected] ~]#  /u01/app/11.2.0.4/grid/bin/crsctl stop crs -f
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.
[[email protected] ~]#  /u01/app/11.2.0.4/grid/bin/crsctl start crs
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.
[[email protected] ~]#  /u01/app/11.2.0.4/grid/bin/crsctl stop has 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘ctcdb‘
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr‘ on ‘ctcdb‘
CRS-2673: Attempting to stop ‘ora.ARCHDG.dg‘ on ‘ctcdb‘
CRS-2673: Attempting to stop ‘ora.DATADG.dg‘ on ‘ctcdb‘
CRS-2677: Stop of ‘ora.ARCHDG.dg‘ on ‘ctcdb‘ succeeded
CRS-2677: Stop of ‘ora.LISTENER.lsnr‘ on ‘ctcdb‘ succeeded
CRS-5017: The resource action "ora.DATADG.dg stop" encountered the following error: 
ORA-03113: end-of-file on communication channel
Process ID: 11558
Session ID: 1099 Serial number: 3
. For details refer to "(:CLSN00108:)" in "/u01/app/11.2.0.4/grid/log/ctcdb/agent/ohasd/oraagent_grid//oraagent_grid.log".

CRS-2677: Stop of ‘ora.DATADG.dg‘ on ‘ctcdb‘ succeeded
CRS-2679: Attempting to clean ‘ora.DATADG.dg‘ on ‘ctcdb‘
CRS-2681: Clean of ‘ora.DATADG.dg‘ on ‘ctcdb‘ succeeded
CRS-2673: Attempting to stop ‘ora.asm‘ on ‘ctcdb‘
CRS-2677: Stop of ‘ora.asm‘ on ‘ctcdb‘ succeeded
CRS-2673: Attempting to stop ‘ora.cssd‘ on ‘ctcdb‘
CRS-2677: Stop of ‘ora.cssd‘ on ‘ctcdb‘ succeeded
CRS-2673: Attempting to stop ‘ora.evmd‘ on ‘ctcdb‘
CRS-2677: Stop of ‘ora.evmd‘ on ‘ctcdb‘ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘ctcdb‘ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[[email protected] ~]#  /u01/app/11.2.0.4/grid/bin/crsctl start crs
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.
[[email protected] ~]#  /u01/app/11.2.0.4/grid/bin/crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[[email protected] ~]# su - oracle
[ctcdb:oracle]:/home/oracle>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jul 20 17:44:04 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 1.3683E+11 bytes
Fixed Size                  2270360 bytes
Variable Size            1.2616E+10 bytes
Database Buffers         1.2402E+11 bytes
Redo Buffers              189456384 bytes
SQL> alter database mount;

Database altered.

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL> alter database recover;
alter database recover
*
ERROR at line 1:
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done


SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: ‘+DATADG/orcl/datafile/system.259.978924803‘


SQL> 
SQL> 
SQL> recover datafile 1;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done


SQL> recover database using backup controlfile;
ORA-00279: change 361503907875 generated at 07/20/2018 14:00:39 needed for
thread 1
ORA-00289: suggestion : +ARCHDG
ORA-15173: entry ‘ORCL‘ does not exist in directory ‘/‘
ORA-00280: change 361503907875 for thread 1 is in sequence #9922


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
auto
ORA-00308: cannot open archived log ‘+ARCHDG‘
ORA-17503: ksfdopn:2 Failed to open file +ARCHDG
ORA-15045: ASM file name ‘+ARCHDG‘ is not in reference form


ORA-00308: cannot open archived log ‘+ARCHDG‘
ORA-17503: ksfdopn:2 Failed to open file +ARCHDG
ORA-15045: ASM file name ‘+ARCHDG‘ is not in reference form


SQL> archive log list;                       
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            +ARCHDG
Oldest online log sequence     9920
Current log sequence           9922




SQL> select file#,CHECKPOINT_CHANGE# from v$datafile_header;

     FILE# CHECKPOINT_CHANGE#
---------- ------------------
         1       361503907875
         2       361503907875
         3       361503907875
         4       361503907875
         5       361503907875
         6       361503907875
         7       361503907875
         8       361503907875
         9       361503907875
        10       361503907875
        11       361503907875

     FILE# CHECKPOINT_CHANGE#
---------- ------------------
        12       361503907875
        13       361503907875
        14       361503907875
        15       361503907875
        16       361503907875
        17       361503907875
        18       361503907875
        19       361503907875
        20       361503907875
        21       361503907875
        22       361503907875

     FILE# CHECKPOINT_CHANGE#
---------- ------------------
        23       361503907875
        24       361503907875
        25       361503907875
        26       361503907875
        27       361503907875
        28       361503907875

SQL> RECOVER DATABASE USING BACKUP CONTROLFILE UNTIL CANCEL;
ORA-00279: change 361503907875 generated at 07/20/2018 14:00:39 needed for
thread 1
ORA-00289: suggestion : +ARCHDG
ORA-00280: change 361503907875 for thread 1 is in sequence #9922


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+DATADG/orcl/redo01.log
Log applied.
Media recovery complete.


SQL> alter database open RESETLOGS;


SQL> select file#,CHECKPOINT_CHANGE# from v$datafile_header;

     FILE# CHECKPOINT_CHANGE#
---------- ------------------
         1       361503908937
         2       361503908937
         3       361503908937
         4       361503908937
         5       361503908937
         6       361503908937
         7       361503908937
         8       361503908937
         9       361503908937
        10       361503908937
        11       361503908937

     FILE# CHECKPOINT_CHANGE#
---------- ------------------
        12       361503908937
        13       361503908937
        14       361503908937
        15       361503908937
        16       361503908937
        17       361503908937
        18       361503908937
        19       361503908937
        20       361503908937
        21       361503908937
        22       361503908937

     FILE# CHECKPOINT_CHANGE#
---------- ------------------
        23       361503908937
        24       361503908937
        25       361503908937
        26       361503908937
        27       361503908937
        28       361503908937
        
        
        
        
        

 

以上是关于记一次数据库异常恢复的主要内容,如果未能解决你的问题,请参考以下文章

记一次线上mysql主从架构异常的恢复经历

记一次数据库不小心批量更新恢复的骚操作

记一次重装系统 恢复原来MongoDB数据库的经历

记一次故障处理----主机异常关闭后mongodb二进制文件损坏

记一次因硬盘故障导致的docker服务无法启动

记一次mysql的数据恢复