CRS-2674

Posted 2021-02-06 wwjd365

[email protected]:~> $ORACLE_HOME/bin/srvctl start database -d ndscdb
PRCR-1079 : Failed to start resource ora.ndscdb.db
CRS-5017: The resource action "ora.ndscdb.db start" encountered the following error: 
ORA-00443: background process "LMD1" did not start
. For details refer to "(:CLSN00107:)" in "/oracle/app/grid/diag/crs/WWJD-db1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.ndscdb.db' on 'WWJD-db1' failed
CRS-2632: There are no more servers to try to place resource 'ora.ndscdb.db' on that would satisfy its placement policy

--alert.log
Starting background process LMD1
2019-01-28T01:29:19.181299-08:00
Process startup failed, error stack:
2019-01-28T01:29:19.181569-08:00
Errors in file /oracle/app/oracle/diag/rdbms/ndscdb/NDSCDB1/trace/NDSCDB1_psp0_148992.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
2019-01-28T01:29:19.181989-08:00
* Load Monitor used for high load check 
* New Low - High Load Threshold Range = [61440 - 81920] 
2019-01-28T01:29:19.184921-08:00
LMS0 started with pid=24, OS id=149071 at elevated (RT) priority
2019-01-28T01:29:19.186634-08:00
LMS2 started with pid=26, OS id=149075 at elevated (RT) priority
2019-01-28T01:29:19.187214-08:00
LMS1 started with pid=25, OS id=149073 at elevated (RT) priority
2019-01-28T01:29:19.187425-08:00
LMS3 started with pid=27, OS id=149077 at elevated (RT) priority
2019-01-28T01:29:20.180278-08:00
Process LMD1 died, see its trace file
2019-01-28T01:29:20.182144-08:00
USER (ospid: 148922): terminating the instance due to error 443
2019-01-28T01:29:21.228555-08:00
Instance terminated by USER, pid = 148922

--trace
[email protected]:~> more /oracle/app/oracle/diag/rdbms/ndscdb/NDSCDB1/trace/NDSCDB1_psp0_148992.trc
Trace file /oracle/app/oracle/diag/rdbms/ndscdb/NDSCDB1/trace/NDSCDB1_psp0_148992.trc
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Build label:    RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME:    /oracle/app/oracle/product/12.2.0/db_1
System name:    Linux
Node name:      WWJD-DB1
Release:        4.4.162-94.69-default
Version:        #1 SMP Mon Nov 5 18:58:52 UTC 2018 (9e06c56)
Machine:        x86_64
Instance name: NDSCDB1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 4
Unix process pid: 148992, image: [email protected] (PSP0)

*** 2019-01-28T01:29:17.870458-08:00
*** CLIENT ID:() 2019-01-28T01:29:17.870482-08:00
*** SERVICE NAME:() 2019-01-28T01:29:17.870488-08:00
*** MODULE NAME:() 2019-01-28T01:29:17.870493-08:00
*** ACTION NAME:() 2019-01-28T01:29:17.870501-08:00
*** CLIENT DRIVER:() 2019-01-28T01:29:17.870506-08:00
 
[kji_attach][LMON !running, !call kjual]
*** SESSION ID:(2065.45615) 2019-01-28T01:29:19.181339-08:00
*** SERVICE NAME:() 2019-01-28T01:29:19.181353-08:00
 

*** 2019-01-28T01:29:19.181318-08:00
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
OS - DIAGNOSTICS
----------------
loadavg : 0.32 0.51 0.63
Memory (Avail / Total) = 497616.35M / 515669.59M
Swap (Avail / Total) = 65536.00M /  65536.00M
Max user processes limits(s / h) =  65536 / 65536
----------------
[email protected]:~> 

• 诊断 ORA-27300 ORA-27301 ORA-27302 错误 (文档 ID 2179478.1)
• ORA-27300 OS system dependent operation:fork failed with status: 11 (文档 ID 392006.1)
• 在 Unix AIX，HP-UX，Linux，Solaris 和 MS Windows 操作系统上安装和配置 Oracle 数据库（RDBMS）的要求的快速参考(12.1/12.2) (文档 ID 2226599.1)

--Oracle要求
Shell limits

Soft File Descriptors     最少 1024 
Hard File Descriptors     最少 65536 
Soft FILE size            -1 (Unlimited)
Soft CPU time            -1 (Unlimited)
Soft DATA segment   -1 (Unlimited)
Soft STACK size        -1 (Unlimited)
Number of processes (Soft) 最少 2047
Number of processes (Hard) 最少 16384
maxuprocs       16384


[email protected]:~> cat /etc/security/limits.conf
grid  soft  nproc  2047
grid  hard  nproc  16384
grid  soft   nofile  1024
grid  hard  nofile  65536
grid  soft   stack  10240
grid  hard  stack  32768

oracle  soft  nproc  2047
oracle  hard  nproc  16384
oracle  soft  nofile  1024
oracle  hard  nofile  65536
oracle  soft  stack  10240
oracle  hard  stack  32768

# End of file
* - nofile 102400
* - nproc 148270
#Oracle recommended value for memlock is set to 3145728 for user oracle
oracle  hard  memlock  3145728
#Oracle recommended value for memlock is set to 3145728 for user oracle
oracle  soft  memlock  3145728
[email protected]:~> ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2062592
max locked memory       (kbytes, -l) 3145728
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2047
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[email protected]:~> 

重启集群,好了...
下班了，先回家...