ASM Disk Group Will not Mount In Presence Of Duplicate Disks / Devices: ORA-15032, ORA-15017, ORA-15

Posted 2020-11-20 oman

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.
***Checked for relevance on 11-May-2014*** 

SYMPTOMS

2 node RAC - 11201 - Solaris10


ASM does not mount ASM disk groups because it finds insufficient disks.  This prevents Oracle Clusterware‘s CRS to start as the OCR diks are stored in ASM:

alert__ASM1.log:
~~~~~~~~~~~~~
...
Fri Oct 19 13:09:35 2012
NOTE: No asm libraries found in the system
ERROR: -5(Duplicate disk DATA:DATA_0000)   <----------- Notice this entry as it tells that ASM discovers duplicate disk (1)
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)
...
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
WARNING: using default parameter settings without any parameter file <---------------- not using a pfile/spfile (1)
...
Fri Oct 19 13:09:44 2012
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent */
NOTE: Diskgroup used for Voting files is:
       DATASOA
Diskgroup used for OCR is:DATA
Diskgroup used for OCR is:DATASOA
NOTE: cache registered group DATASOA number=1 incarn=0xa9eb204b
NOTE: cache registered group DATA number=2 incarn=0xaa0b204c
ERROR: no PST quorum in group: required 2, found 0    <------- ASM disk discovery problem
...
NOTE: cache ending mount (fail) of group DATASOA number=1 incarn=0xa9eb204b
...
ERROR: diskgroup DATASOA was not mounted  <--------- Therefore ASM disk groups do not mount
...
NOTE: cache ending mount (fail) of group DATA number=2 incarn=0xaa0b204c
...
ERROR: diskgroup DATA was not mounted
...
WARNING: Disk Group DATASOA containing configured OCR is not mounted
WARNING: Disk Group DATASOA containing voting files is not mounted
WARNING: Disk Group DATA containing configured OCR is not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA" <---- ASM discovers not sufficient disks
ORA-15017: diskgroup "DATASOA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATASOA"

(1) These entries are similar to Note:1384692.1

CAUSE

However, the problem is not with ASM not finding all disks but finding duplicate disks (i,e, instead of ASM discovering only all 5 needed ASM disk, it is discovering twice that amount, or 10 disks).

When ASM discovers the same disk more than once, it ignores them (v$asm_disk.mount_status=IGNORED), so it is like it finds not sufficient disks. 

Hence, the ASM alert logs shows the entry "ERROR: -5(Duplicate disk ..." at the instance starting up sequence and later gives the insufficient disk errors (ORA-15063).  
.
 

SOLUTION

Note: Finding duplicate disks will depend on the platform used.

For this case (Solaris), the following has done to identify the duplicate devices:

* From previous successful attempts to mount all ASM disk groups from ASM alert log entries, the following asm disks are needed:

   NOTE: cache registered group DATASOA number=1 incarn=0xa9eac7b8
   NOTE: cache registered group DATA number=2 incarn=0xaa0ac7b9
   NOTE: Assigning number (1,0) to disk (/oracle_crs_install/shr_disk_links/ora_dbdsk1) <--- 4 disks for DATASOA
   NOTE: Assigning number (1,2) to disk (/oracle_crs_install/shr_disk_links/ora_dbdsk2)
   NOTE: Assigning number (1,3) to disk (/oracle_crs_install/shr_disk_links/ora_dbdsk3)
   NOTE: Assigning number (1,1) to disk (/oracle_crs_install/shr_disk_links/ora_dbdsk4)
   NOTE: Assigning number (2,0) to disk (/dev/rdsk/c0t60050768018105880000000000000023d0s6) <--- 1 disk for DATA

* The Grid-plug-and-play profile or <Grid Infrastructure Home>/gpnp/<hostname>/profiles/peer/profile.xml, lists the disk discovery string that will be used (along with the spfile that will be use):

DiscoveryString="/oracle_crs_install/shr_disk_links/ora_dbdsk*,/dev/rdsk/c*23d0s6" SPFile="+DATA/wlsoadbt1-clust/asmparameterfile/registry.253.778542929"

* From the above, it was found that ASM was discovering It was found that

  /dev/rdsk/c0t60050768018105880000000000000035d0s6 <-- dupl
  /dev/rdsk/c0t60050768018105880000000000000034d0s6 <-- dupl
  /dev/rdsk/c0t60050768018105880000000000000033d0s6 <-- dupl
  /dev/rdsk/c0t60050768018105880000000000000032d0s6 <-- dupl
  /dev/rdsk/c0t60050768018105880000000000000038d0s6 <-- dupl
  /dev/rdsk/c0t60050768018105880000000000000023d0s6
  /oracle_crs_install/shr_disk_links/ora_dbdsk1
  /oracle_crs_install/shr_disk_links/ora_dbdsk2
  /oracle_crs_install/shr_disk_links/ora_dbdsk3
  /oracle_crs_install/shr_disk_links/ora_dbdsk4

* Since the duplicate disks were identified and isolated, please change their device‘s owner:group or file premissions in a manner that ASM will no longer discover (access) these.
   ASM can tolerate the discovery of each ASM disk only once (not more than once or it will be considered a ‘duplicate disk‘ and hence we will likely encounter this scenario.

REFERENCES

ORA-15032: not all alterations performed
ORA-15017: diskgroup "TESTADG" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "TESTDG_1"