ORA-00445: 后台进程 "J001" 在 120 秒之后仍没有启动

Posted 2020-12-12 love-vce

今天早上系统突然挂掉，查看数据库发现数据库内存使用完了，然后看trace日志发现:

os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 08:58:05 ]
NOTE: scheduling delay has not been sampled for 0.577206 secs 0.000000 secs from [ 08:58:01 - 08:58:06 ], 5 sec avg
0.000000 secs from [ 08:57:06 - 08:58:06 ], 1 min avg
0.000000 secs from [ 08:53:06 - 08:58:06 ], 5 min avg
loadavg : 0.24 0.59 0.49
Memory (Avail / Total) = 209.00M / 64314.86M
Swap (Avail / Total) = 23657.57M / 31996.64M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S oracle 19976 1 0 78 0 - 6663032 semtim 08:56 ? 00:00:00 ora_j001_oatest
Stack:
skgpgpstack: failed to access /usr/bin/gdb with errno=2

-------------------------------------------------------------------------------
Process diagnostic dump actual duration=0.090000 sec
(max dump time=30.000000 sec)

*** 2020-06-24 08:58:06.266

*** 2020-06-24 08:58:20.793
Killing process (ospid 19976): (reason=KSOREQ_WAIT_CANCELLED error=0)
... and the process is still alive after kill!
Incident 76360 created, dump file: /oracle/app/diag/rdbms/oatest/oatest/incident/incdir_76360/oatest_cjq0_5024_i76360.trc
ORA-00445: 后台进程 "J001" 在 120 秒之后仍没有启动

使用top查看内存占用情况oracle进程占用最多

关掉数据库，释放内存资源

修复后查看资料需要把ASLR关掉，系统使用的是RHEL5

ORA-00445: background process "m001" did not start after 120 seconds
Incident details in: /opt/u01/app/oracle/diag/rdbms/incident/incdir_3721/db1_mmon_7417_i3721.trc
ERROR: Unable to normalize symbol name for the following short stack (at offset 2):
Tue Jun 21 03:03:06 2011
ORA-00445: background process "J003" did not start after 120 seconds
or
Waited for process W002 to initialize for 60 seconds
The system appears to be running very slowly and defunct processes can appear.
Changes
REDHAT 5 kernel 2.6.18-194.el5 #1 SMP Tue Mar 16
Oracle 11.2.0.2 Single Instance
IBM: Linux on System z
Cause
Recent linux kernels have a feature called Address Space Layout Randomization (ASLR).
ASLR is a feature that is activated by default on some of the newer linux distributions.
It is designed to load shared memory objects in random addresses.
In Oracle, multiple processes map a shared memory object at the same address across the processes.

With ASLR turned on Oracle cannot guarantee the availability of this shared memory address.
This conflict in the address space means that a process trying to attach a shared memory object to a specific address may not be able to do so, resulting in a failure in shmat subroutine.

However, on subsequent retry (using a new process) the shared memory attachment may work.
The result is a random set of failures
Solution
It should be noted that this problem has only been positively diagnosed in Redhat 5 and Oracle 11.2.0.2.
It is also likely, as per unpublished BUG:8527473, that this issue will reproduce running on Generic Linux platforms running any Oracle 11.2.0.x. on Redhat/OEL kernels which have ASLR.

ASLR also exists in SLES10 and SLES 11 kernels and by default ASLR is turned on, however, to date not problem has been seen on SuSE.

You can verify whether ASLR is being used as follows:

# /sbin/sysctl -a | grep randomize
kernel.randomize_va_space = 1

If the parameter is set to any value other than 0 then ASLR is in use.
On Redhat 5 to permanently disable ASLR.
add/modify this parameter in /etc/sysctl.conf
kernel.randomize_va_space=0
kernel.exec-shield=0

You need to reboot for kernel.exec-shield parameter to take effect.
Note that both kernel parameters are required for ASLR to be switched off.

更改前是:

/sbin/sysctl -a | grep randomize
kernel.randomize_va_space = 1

更改后是:

/sbin/sysctl -a | grep randomize
kernel.randomize_va_space = 0