Recently we had a power outage in our lab which caused all the Linux servers running Oracle database, switches, storage to go down. When the power was restored the servers, storages, and switches were brought up cleanly.
I noticed that Oracle ASM instance was down, so I tried to manually start it and got the ORA-01078 and ORA-29701 error messages.
Tried to use crsctl to start all the Oracle resources
On the Linux server running Oracle
When I looked at the owner and group of the disks, I noticed that the they had changed to owner root and group disk. No wonder Oracle is not seeing them.
Changed the owner and group of the volumes to oracle and dba
When I brought up asmca. I saw that the my +DATA diskgroup could now be seen, but was not mounted. I mounted the diskgroup, and everything worked fine after that.
I noticed that Oracle ASM instance was down, so I tried to manually start it and got the ORA-01078 and ORA-29701 error messages.
sh-3.2$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.1.0 Production on Wed Sep 28 15:20:59 2011 Copyright (c) 1982, 2009, Oracle. All rights reserved. Connected to an idle instance. SQL> startup; ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service SQL> |
-bash-3.2$ . ./.bash_profile_grid -bash-3.2$ echo $ORACLE_HOME /u01/app/oracle/product/11.2.0/grid -bash-3.2$ which crsctl /u01/app/oracle/product/11.2.0/grid/bin/crsctl |
Tried to use crsctl to start all the Oracle resources
-bash-3.2$ crsctl start resource -all CRS-5702: Resource 'ora.LISTENER.lsnr' is already running on 'isvx7' CRS-2672: Attempting to start 'ora.cssd' on 'isvx7' CRS-2679: Attempting to clean 'ora.diskmon' on 'isvx7' CRS-2681: Clean of 'ora.diskmon' on 'isvx7' succeeded CRS-2672: Attempting to start 'ora.diskmon' on 'isvx7' CRS-2676: Start of 'ora.diskmon' on 'isvx7' succeeded CRS-2676: Start of 'ora.cssd' on 'isvx7' succeeded CRS-2672: Attempting to start 'ora.asm' on 'isvx7' CRS-2676: Start of 'ora.asm' on 'isvx7' succeeded CRS-2672: Attempting to start 'ora.DATA.dg' on 'isvx7' ORA-15032: not all alterations performed ORA-15017: diskgroup "DATA" cannot be mounted ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA" CRS-2674: Start of 'ora.DATA.dg' on 'isvx7' failed CRS-2679: Attempting to clean 'ora.DATA.dg' on 'isvx7' CRS-2681: Clean of 'ora.DATA.dg' on 'isvx7' succeeded CRS-4000: Command Start failed, or completed with errors. -bash-3.2$ |
On the Linux server running Oracle
[root@isvx7 ~]# multipath -d -l mulipath.conf line 111, invalid keyword: prio mpath144 (36005076802828000c000000000000050) dm-3 IBM,2145 [size=300G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 5:0:3:3 sdaf 65:240 [active][undef] \_ 6:0:3:3 sdag 66:0 [active][undef] \_ 5:0:0:3 sdh 8:112 [active][undef] \_ 6:0:0:3 sdi 8:128 [active][undef] \_ 6:0:1:3 sdp 8:240 [active][undef] \_ 5:0:1:3 sdq 65:0 [active][undef] \_ 5:0:2:3 sdx 65:112 [active][undef] \_ 6:0:2:3 sdy 65:128 [active][undef] mpath143 (36005076802828000c000000000000053) dm-2 IBM,2145 [size=300G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 5:0:3:2 sdad 65:208 [active][undef] \_ 6:0:3:2 sdae 65:224 [active][undef] \_ 5:0:0:2 sdf 8:80 [active][undef] \_ 6:0:0:2 sdg 8:96 [active][undef] \_ 5:0:1:2 sdn 8:208 [active][undef] \_ 6:0:1:2 sdo 8:224 [active][undef] \_ 5:0:2:2 sdv 65:80 [active][undef] \_ 6:0:2:2 sdw 65:96 [active][undef] mpath142 (36005076802828000c000000000000052) dm-1 IBM,2145 [size=300G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 5:0:3:1 sdab 65:176 [active][undef] \_ 6:0:3:1 sdac 65:192 [active][undef] \_ 5:0:0:1 sdc 8:32 [active][undef] \_ 6:0:0:1 sde 8:64 [active][undef] \_ 6:0:1:1 sdl 8:176 [active][undef] \_ 5:0:1:1 sdm 8:192 [active][undef] \_ 5:0:2:1 sdt 65:48 [active][undef] \_ 6:0:2:1 sdu 65:64 [active][undef] mpath141 (36005076802828000c000000000000051) dm-0 IBM,2145 [size=300G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 5:0:3:0 sdaa 65:160 [active][undef] \_ 5:0:0:0 sdb 8:16 [active][undef] \_ 6:0:0:0 sdd 8:48 [active][undef] \_ 6:0:1:0 sdj 8:144 [active][undef] \_ 5:0:1:0 sdk 8:160 [active][undef] \_ 6:0:2:0 sdr 65:16 [active][undef] \_ 5:0:2:0 sds 65:32 [active][undef] \_ 6:0:3:0 sdz 65:144 [active][undef] |
When I looked at the owner and group of the disks, I noticed that the they had changed to owner root and group disk. No wonder Oracle is not seeing them.
[root@isvx7 ~]# ls -l /dev/mapper/mpath141 brw-rw---- 1 root disk 253, 0 Sep 28 09:57 /dev/mapper/mpath141 [root@isvx7 ~]# ls -l /dev/mapper/mpath142 brw-rw---- 1 root disk 253, 1 Sep 28 09:57 /dev/mapper/mpath142 [root@isvx7 ~]# ls -l /dev/mapper/mpath143 brw-rw---- 1 root disk 253, 2 Sep 28 09:57 /dev/mapper/mpath143 [root@isvx7 ~]# ls -l /dev/mapper/mpath144 brw-rw---- 1 root disk 253, 3 Sep 28 09:57 /dev/mapper/mpath144 |
[root@isvx7 ~]# chown -R oracle:dba /dev/mapper/mpath141 [root@isvx7 ~]# chown -R oracle:dba /dev/mapper/mpath142 [root@isvx7 ~]# chown -R oracle:dba /dev/mapper/mpath143 [root@isvx7 ~]# chown -R oracle:dba /dev/mapper/mpath144 [root@isvx7 ~]# ls -l /dev/mapper/mpath143 brw-rw---- 1 oracle dba 253, 2 Sep 28 09:57 /dev/mapper/mpath143 [root@isvx7 ~]# |
When I brought up asmca. I saw that the my +DATA diskgroup could now be seen, but was not mounted. I mounted the diskgroup, and everything worked fine after that.
