Table of Contents
What to do first ?
- Check your disk space using: # df
- Check whether your are a firewall: # service iptables status ( <— this command is very important )
- Use Nslookup and ping to verify you Cluster Interconnect
CRS does not start GIPC error: [29] msg [gipcretConnectionRefused]
Posted chendian0
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了CRS does not start GIPC error: [29] msg [gipcretConnectionRefused]相关的知识，希望对你有一定的参考价值。
CRS does not start GIPC error: [29] msg [gipcretConnectionRefused]
CRS does not start GIPC error: [29] msg [gipcretConnectionRefused] Table of Contents What to do first ? Scenario 1: Wrong IP Address Scenario 2: Filesystem full ( 12c ) Scenario 3 : Firwall ON References What to do first ? Check your disk space using: # df Check whether your are a firewall: # service iptables status ( <— this command is very important ) Use Nslookup and ping to verify you Cluster Interconnect Scenario 1: Wrong IP Address Errors: GIPC repot error [29] msg [gipcretConnectionRefused] CHM report clsu_get_private_ip failed Check CRS status [[email protected] Desktop]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4529: Cluster Synchronization Services is online CRS-4534: Cannot communicate with Event Manager [[email protected] network-scripts]# my_crs_stat_init NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.asm ONLINE OFFLINE Instance Shutdown ora.cluster_interconnect.haip ONLINE OFFLINE ora.crf ONLINE ONLINE grac41 ora.crsd ONLINE OFFLINE ora.cssd ONLINE UNKNOWN grac41 ora.cssdmonitor ONLINE ONLINE grac41 ora.ctssd ONLINE OFFLINE ora.diskmon OFFLINE OFFLINE ora.drivers.acfs ONLINE ONLINE grac41 ora.evmd ONLINE OFFLINE ora.gipcd ONLINE ONLINE grac41 ora.gpnpd ONLINE ONLINE grac41 ora.mdnsd ONLINE ONLINE grac41 --> ASM, HAIP, CRSD, CTSSD, DISKMON, EVMD resource are OFFLINE ! Check traces - ohasd trace file [[email protected] ohasd]# cat ohasd.log | grep -i failed 2014-04-22 15:09:17.966: [ AGFW][2735122176]{0:0:2} ora.cluster_interconnect.haip 1 1 received state from probe request. Old state = UNKNOWN, New state = FAILED 2014-04-22 15:09:30.292: [ GPNP][2745628416]clsgpnp_getCachedProfileEx: [at clsgpnp.c:623] Result: (26) CLSGPNP_NO_PROFILE. Failed to get offline GPnP service profile. 2014-04-22 15:09:30.602: [ GPNP][2717640448]clsgpnp_getCachedProfileEx: [at clsgpnp.c:623] Result: (26) CLSGPNP_NO_PROFILE. Failed to get offline GPnP service profile. --> HAIP goes to FAILED status Try to find any repeating updated tracefiles - maybe some RAC process tries to fix the network problem [[email protected] grac41]$ date; find . -type f -printf "%CY-%Cm-%Cd %CH:%CM:%CS %h/%f\n" | sort -n | tail -5 Tue Apr 22 13:24:40 CEST 2014 2014-04-22 13:24:30.0571859790 ./gpnpd/gpnpd.log 2014-04-22 13:24:33.0756944610 ./agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log 2014-04-22 13:24:38.0881994320 ./ohasd/ohasd.log 2014-04-22 13:24:38.3523314350 ./gipcd/gipcd.log 2014-04-22 13:24:39.0876989250 ./crfmond/crfmond.log [[email protected] grac41]$ date; find . -type f -printf "%CY-%Cm-%Cd %CH:%CM:%CS %h/%f\n" | sort -n | tail -5 Tue Apr 22 13:24:43 CEST 2014 2014-04-22 13:24:30.0571859790 ./gpnpd/gpnpd.log 2014-04-22 13:24:33.0756944610 ./agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log 2014-04-22 13:24:43.1007044060 ./ohasd/ohasd.log 2014-04-22 13:24:43.3668374000 ./gipcd/gipcd.log 2014-04-22 13:24:43.7580328990 ./crfmond/crfmond.log [[email protected] grac41]$ date; find . -type f -printf "%CY-%Cm-%Cd %CH:%CM:%CS %h/%f\n" | sort -n | tail -5 Tue Apr 22 13:24:47 CEST 2014 2014-04-22 13:24:30.0571859790 ./gpnpd/gpnpd.log 2014-04-22 13:24:33.0756944610 ./agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log 2014-04-22 13:24:43.1007044060 ./ohasd/ohasd.log 2014-04-22 13:24:44.0972023860 ./crfmond/crfmond.log 2014-04-22 13:24:46.4033548850 ./gipcd/gipcd.log --> Here we cans see that ./ohasd/ohasd.log ./gipcd/gipcd.log ./crfmond/crfmond.log Use tail to see what‘s going : [[email protected] grac41]$ tail -f ./gpnpd/gpnpd.log 2014-04-22 13:19:59.175: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:21:29.469: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:22:59.792: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:24:30.057: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:26:00.383: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:27:30.622: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:29:00.869: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:30:31.203: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:32:01.459: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] 2014-04-22 13:33:31.770: [ OCRMSG][4002494208]GIPC error [29] msg [gipcretConnectionRefused] [[email protected] grac41]$ tail -f ./ohasd/ohasd.log 2014-04-22 13:33:42.806: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:33:47.817: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:33:52.839: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:33:57.848: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:34:03.859: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:34:09.874: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:34:15.881: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:34:20.900: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:34:25.920: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd 2014-04-22 13:34:30.934: [GIPCHDEM][2222126848]gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x2d03370 [0000000000000010] { gipchaContext : host ‘grac41‘, name ‘CLSFRAME_grac4‘, luid ‘57127705-00000000‘, numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd [[email protected] grac41]$ tail -f ./crfmond/crfmond.log [ CLWAL][467654400]clsw_Initialize: OLR initlevel [70000] 2014-04-22 13:34:49.349: [ CRFM][467654400]crfm_connstr: clsu_get_private_ip failed(7). 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_connect_to: send fail(gipcret: 13) 2014-04-22 13:34:49.458: [ CRFM][467654400]crfmctx dump follows 2014-04-22 13:34:49.458: [ CRFM][467654400]**************************** 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: connection local name: tcp://0.0.0.0:45871 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: connection peer name: tcp://192.168.1.101:61021 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: connaddr: tcp://grac41:61021 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: ctype: 2 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: mytype: 0 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: hostname grac41 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: myport: 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: rhostname 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: rport: 2014-04-22 13:34:49.458: [ CRFM][467654400]crfm_dumpctx: flags: 1 2014-04-22 13:34:49.458: [ CRFM][467654400]**************************** According to above traces we can see that clsu_get_private_ip failed getting private IP tcp://192.168.1.101 Check Network status and DNS [[email protected] Desktop]# ifconfig eth1 Link encap:Ethernet HWaddr 08:00:27:89:E9:A2 inet addr:192.168.2.101 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe89:e9a2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:17148 errors:0 dropped:0 overruns:0 frame:0 TX packets:13307 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:22041591 (21.0 MiB) TX bytes:1211055 (1.1 MiB) Interrupt:9 Base address:0xd240 eth2 Link encap:Ethernet HWaddr 08:00:27:6B:E2:BD inet addr:192.168.1.101 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe6b:e2bd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:17517 errors:0 dropped:0 overruns:0 frame:0 TX packets:13475 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:22191772 (21.1 MiB) TX bytes:1230703 (1.1 MiB) Interrupt:5 Base address:0xd260 --> Check public and private interface for errors / Looks good [[email protected]ac41 Desktop]# nslookup grac41 Name: grac41.example.com Address: 192.168.1.101 [[email protected] Desktop]# nslookup grac41int Name: grac41int.example.com Address: 192.168.2.101 [[email protected] Desktop]# nslookup 192.168.1.101 101.1.168.192.in-addr.arpa name = grac41.example.com. [[email protected] Desktop]# nslookup 192.168.2.101 101.2.168.192.in-addr.arpa name = grac41int.example.com. --> DNS and Network seems to be ok Restart CRS [email protected] Desktop]# crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘grac41‘ CRS-2673: Attempting to stop ‘ora.crf‘ on ‘grac41‘ CRS-2673: Attempting to stop ‘ora.ctssd‘ on ‘grac41‘ CRS-2673: Attempting to stop ‘ora.evmd‘ on ‘grac41‘ ... CRS-2673: Attempting to stop ‘ora.gpnpd‘ on ‘grac41‘ CRS-2677: Stop of ‘ora.gpnpd‘ on ‘grac41‘ succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘grac41‘ has completed CRS-4133: Oracle High Availability Services has been stopped. Cleanup /var/tmp/.oracle # rm /var/tmp/.oracle/* [[email protected] Desktop]# crsctl start crs [[email protected] Desktop]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4529: Cluster Synchronization Services is online CRS-4534: Cannot communicate with Event Manager --> Problem persists Check OS logfile # cat /var/log/messages --> Nothing related Run orcheck ( and orcdump ) to check whether we can access our OCR repostory [[email protected] Desktop]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 4076 Available space (kbytes) : 258044 ID : 630679368 Device/File Name : +OCR Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded Query voting disk : [[email protected] grac41]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE b0e94e5d83054fe9bf58b6b98bfacd65 (/dev/asmdisk1_udev_sdf1) [OCR] 2. ONLINE 88c2a08b4c8c4f85bf0109e0990388e4 (/dev/asmdisk1_udev_sdg1) [OCR] 3. ONLINE 1108f9a41e814fb2bfed879ff0039dd0 (/dev/asmdisk1_udev_sdh1) [OCR] Located 3 voting disk(s). Debugging GIPCD and GPnPD daemons using strace As GIPCD and GPnPD daemon traces gets updated every 5s lets check the gipcd process with strace # ps -elf | egrep ‘gpnpd.bin|gipcd.bin‘ # strace -t -f -p 24376 2>&1 | grep ‘192.168‘ | grep eth [pid 24872] 09:17:28 <... ioctl resumed> 200, {{"lo", {AF_INET, inet_addr("127.0.0.1")}}, {"eth0", {AF_INET, inet_addr("10.0.2.15")}}, {"eth1", {AF_INET, inet_addr("192.168.2.101")}}, {"eth2", {AF_INET, inet_addr("192.168.1.101")}}, {"virbr0", {AF_INET, inet_addr("192.168.122.1")}}}}) = 0 [pid 24870] 09:17:28 <... ioctl resumed> , {ifr_name="eth1", ifr_addr={AF_INET, inet_addr("192.168.2.101")}}) = 0 [pid 24870] 09:17:28 <... ioctl resumed> , {ifr_name="eth1", ifr_broadaddr={AF_INET, inet_addr("192.168.2.255")}}) = 0 [pid 24872] 09:17:28 <... ioctl resumed> , {ifr_name="eth1", ifr_addr={AF_INET, inet_addr("192.168.2.101")}}) = 0 [pid 24870] 09:17:28 <... ioctl resumed> , {ifr_name="eth2", ifr_addr={AF_INET, inet_addr("192.168.1.101")}}) = 0 [pid 24872] 09:17:28 <... ioctl resumed> , {ifr_name="eth1", ifr_broadaddr={AF_INET, inet_addr("192.168.2.255")}}) = 0 [pid 24870] 09:17:28 <... ioctl resumed> , {ifr_name="eth2", ifr_broadaddr={AF_INET, inet_addr("192.168.1.255")}}) = 0 [pid 24872] 09:17:28 <... ioctl resumed> , {ifr_name="eth2", ifr_addr={AF_INET, inet_addr("192.168.1.101")}}) = 0 [pid 24872] 09:17:28 <... ioctl resumed> , {ifr_name="eth2", ifr_broadaddr={AF_INET, inet_addr("192.168.1.255")}}) = 0 .. [pid 24872] 09:17:33 <... ioctl resumed> 200, {{"lo", {AF_INET, inet_addr("127.0.0.1")}}, {"eth0", {AF_INET, inet_addr("10.0.2.15")}}, {"eth1", {AF_INET, inet_addr("192.168.2.101")}}, {"eth2", {AF_INET, inet_addr("192.168.1.101")}}, {"virbr0", {AF_INET, inet_addr("192.168.122.1")}}}}) = 0 [pid 24870] 09:17:33 <... ioctl resumed> , {ifr_name="eth1", ifr_addr={AF_INET, inet_addr("192.168.2.101")}}) = 0 [pid 24870] 09:17:33 <... ioctl resumed> , {ifr_name="eth1", ifr_broadaddr={AF_INET, inet_addr("192.168.2.255")}}) = 0 [pid 24870] 09:17:33 <... ioctl resumed> , {ifr_name="eth2", ifr_addr={AF_INET, inet_addr("192.168.1.101")}}) = 0 [pid 24870] 09:17:33 <... ioctl resumed> , {ifr_name="eth2", ifr_broadaddr={AF_INET, inet_addr("192.168.1.255")}}) = 0 [pid 24872] 09:17:33 <... ioctl resumed> , {ifr_name="eth1", ifr_addr={AF_INET, inet_addr("192.168.2.101")}}) = 0 [pid 24872] 09:17:33 <... ioctl resumed> , {ifr_name="eth1", ifr_broadaddr={AF_INET, inet_addr("192.168.2.255")}}) = 0 [pid 24872] 09:17:33 <... ioctl resumed> , {ifr_name="eth2", ifr_addr={AF_INET, inet_addr("192.168.1.101")}}) = 0 .. --> Again we don‘t get an OS error but we are looping running the same ioctl() command Seems the kernel is not happy with the inforamtion we get from ioctl() call and tries to reread the information every 5 seconds Check GPnP profile [[email protected] Desktop]# gpnptool get > profile.xml Edit profile.xml and extract the adapter usage <gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"> <gpnp:Network id="net1" IP="192.168.1.0" Adapter="eth1" Use="public"/> <gpnp:Network id="net2" IP="192.168.2.0" Adapter="eth2" Use="cluster_interconnect"/> Verify with ifconfig [[email protected] Desktop]# ifconfig | egrep ‘HWaddr|inet addr‘ eth1 Link encap:Ethernet HWaddr 08:00:27:89:E9:A2 inet addr:192.168.2.101 Bcast:192.168.2.255 Mask:255.255.255.0 eth2 Link encap:Ethernet HWaddr 08:00:27:6B:E2:BD inet addr:192.168.1.101 Bcast:192.168.1.255 Mask:255.255.255.0 inet addr:127.0.0.1 Mask:255.0.0.0 --> eth1 is using 192.168.2.101 but according GPnP Profile it should use 192.168.1.101 eth2 is using 192.168.1.101 but according GPnP Profile it should use 192.168.2.101 Problem found : During manuall editing ifcfg-eth1 and ifcfg-eth2 HWADR entry was wrongly filled ( /etc/sysconfig/network-scripts ) Reconfiguring/restart network and CRS [[email protected] network-scripts]# cat ifcfg-eth2 HWADDR=08:00:27:89:E9:A2 IPADDR=192.168.2.101 NAME=eth2 [[email protected] network-scripts]# cat ifcfg-eth1 IPADDR=192.168.1.101 NAME=eth1 HWADDR=08:00:27:6B:E2:BD After changing HWaddr to follow the above ifconfig output the network looks good [[email protected] network-scripts] service network restart [[email protected] network-scripts]# ifconfig | egrep ‘HWaddr|inet addr‘ eth1 Link encap:Ethernet HWaddr 08:00:27:89:E9:A2 inet addr:192.168.1.101 Bcast:192.168.1.255 Mask:255.255.255.0 eth2 Link encap:Ethernet HWaddr 08:00:27:6B:E2:BD inet addr:192.168.2.101 Bcast:192.168.2.255 Mask:255.255.255.0 Restart CRS [[email protected] network-scripts]# crsctl stop crs -f [[email protected] network-scripts]# crsctl start crs [[email protected] network-scripts]# crsctl check cluster -all ************************************************************** grac41: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** grac42: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** grac43: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** Lessons learned - Verify carefully that IP addresses and Network Device names are clusterwide consistent Scenario 2: Filesystem full ( 12c ) [[email protected] Desktop]# crsi ***** Local Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.asm 1 ONLINE OFFLINE - STABLE ora.cluster_interconnect.haip 1 ONLINE OFFLINE - STABLE ora.crf 1 ONLINE OFFLINE - STABLE ora.crsd 1 ONLINE OFFLINE - STABLE ora.cssd 1 ONLINE OFFLINE - STABLE ora.cssdmonitor 1 OFFLINE OFFLINE - STABLE ora.ctssd 1 ONLINE OFFLINE - STABLE ora.diskmon 1 OFFLINE OFFLINE - STABLE ora.drivers.acfs 1 ONLINE ONLINE gract1 STABLE ora.evmd 1 ONLINE OFFLINE gract1 STARTING ora.gipcd 1 ONLINE OFFLINE - STABLE ora.gpnpd 1 ONLINE OFFLINE - STABLE ora.mdnsd 1 ONLINE OFFLINE gract1 STARTING ora.storage 1 ONLINE OFFLINE - STABLE Related client trace 2014-08-22 10:57:07.750: [ OCRMSG][2296473152]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2) 2014-08-22 10:57:07.750: [ OCRMSG][2296473152]GIPC error [29] msg [gipcretConnectionRefused] 2014-08-22 10:57:07.750: [ OCRMSG][2296473152]prom_connect: error while waiting for connection complete [24] 2014-08-22 10:57:07.821: [ OCRMSG][2296473152]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2) 2014-08-22 10:57:07.821: [ OCRMSG][2296473152]GIPC error [29] msg [gipcretConnectionRefused] 2014-08-22 10:57:07.821: [ OCRMSG][2296473152]prom_connect: error while waiting for connection complete [24] Root Cause : File system full : 100% - No traces can be written # df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_oel64-lv_root 39603624 37798864 0 100% / tmpfs 4194304 272 4194032 1% /dev/shm /dev/sda1 495844 101751 368493 22% /boot Scenario 3 : Firwall ON ***** Cluster Resources: ***** Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.asm 1 ONLINE OFFLINE - STABLE ora.cluster_interconnect.haip 1 ONLINE OFFLINE - STABLE ora.crf 1 ONLINE OFFLINE - STABLE ora.crsd 1 ONLINE OFFLINE - STABLE ora.cssd 1 ONLINE OFFLINE - STABLE ora.cssdmonitor 1 ONLINE ONLINE gract2 STABLE ora.ctssd 1 ONLINE OFFLINE - STABLE ora.diskmon 1 OFFLINE OFFLINE - STABLE ora.drivers.acfs 1 ONLINE ONLINE gract2 STABLE ora.evmd 1 ONLINE INTERMEDIATE gract2 STABLE ora.gipcd 1 ONLINE ONLINE gract2 STABLE ora.gpnpd 1 ONLINE ONLINE gract2 STABLE ora.mdnsd 1 ONLINE ONLINE gract2 STABLE ora.storage 1 ONLINE OFFLINE - STABLE --> CSSD doesn‘t become ONLINE Client log : 014-08-23 11:49:21.920: [ OCRMSG][2580342528]GIPC error [29] msg [gipcretConnectionRefused] 2014-08-23 11:49:42.948: [ OCRMSG][2580342528]GIPC error [29] msg [gipcretConnectionRefused] 2014-08-23 11:50:10.978: [ OCRMSG][2580342528]GIPC error [29] msg [gipcretConnectionRefused] 2014-08-23 11:50:46.008: [ OCRMSG][2580342528]GIPC error [29] msg [gipcretConnectionRefused] 2014-08-23 11:51:28.042: [ OCRMSG][2580342528]GIPC error [29] msg [gipcretConnectionRefused] 2014-08-23 11:51:28.042: [ OCRMSG][2580342528]GIPC error [29] msg [gipcretConnectionRefused] 20665 <... connect resumed> ) = 0 20665 connect(66, {sa_family=AF_FILE, path="/var/tmp/.oracle/sOHASD_UI_SOCKET"}, 110 <unfinished ...> 20665 <... connect resumed> ) = 0 20665 connect(73, {sa_family=AF_FILE, path="/var/tmp/.oracle/sprocr_local_conn_0_PROC"}, 110 <unfinished ...> 20665 <... connect resumed> ) = -1 ECONNREFUSED (Connection refused) occsd.log : 2014-08-23 12:32:58.427: [ CSSD][1279260416]clssnmvDHBValidateNCopy: node 1, gract1, has a disk HB, but no network HB, DHB has rcfg 304252836, wrtcnt, 3207223, LATS 4294823390, lastSeqNo 3207220, uniqueness 1408783210, timestamp 1408789980/5988764 2014-08-23 12:32:58.427: [ CSSD][1283991296]clssnmvDHBValidateNCopy: node 1, gract1, has a disk HB, but no network HB, DHB has rcfg 304252836, wrtcnt, 3207224, LATS 4294823390, lastSeqNo 3207221, uniqueness 1408783210, timestamp 1408789980/5988864 Fix : Disable Firewall References Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip (Doc ID 1210883.1) Grid Infrastructure Installation root.sh Failed with “Failed to start CTSS” (Doc ID 1277307.1) Troubleshoot Grid Infrastructure Startup Issues (Doc ID 1050908.1) Top 5 Grid Infrastructure Startup Issues (Doc ID 1368382.1)
以上是关于CRS does not start GIPC error: [29] msg [gipcretConnectionRefused]的主要内容，如果未能解决你的问题，请参考以下文章
does not kick start when marker is detected
Starting the application on Mac does not work（拷贝platforms到不同的位置，才能解决问题）
Service Fabric 反向代理 - 404 - FABRIC_E_SERVICE_DOES_NOT_EXIST
java.lang.RuntimeException: Unable to start activity ComponentInfo……AppCompat does not support the c
错误记录Android Studio 编译报错 ( AAPT: error: failed to read PNG signature: file does not start with PN )
react-native run-android Starting: Intent Error type 3 Error: Activity class does not exist