Friday, January 11, 2008
at
12:50 PM
|
What to do when "vxdisk list" shows status of 'online dgdisabled'.
Details:
This situation can happen when every disk in a disk group is lost from a bad power supply, power turned off to the disk array, cable disconnected, zoning problems, etc.
The disk group will not show in the output from
The disk group will show as disabled in
This is the output of
Your filesystems will of course fail and the operating system will report it as corrupted.
Once you have confirmed that the disk storage is powered-up, running, and operational and if the LUNs are in a SAN, zoning is configured right, this problem can be remedied by deporting, and then importing the disk group:
If volume manager can't see the disks, and your SAN or storage administrator has confirmed that the LUNs were fine and presented to your server, then rescan the disks.
Otherwise, your diskgroup should be showing up as enabled.
The disk group now shows in
Verify that the disks on the diskgroup are all online.
Now the volumes can be started:
The filesystems on these volumes may not be in consistent state. So, run a filesystem check before mounting them.
note: This example was taken from an AIX server, but all veritas commands here will work on all UNIX platforms.
Details:
aixsrv01:#vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
EMC_CLARiiON0_0 auto:cdsdisk EMC_CLARiiON0_0 dygy2502 online
EMC_CLARiiON0_1 auto:cdsdisk - (dvgy2500) online
EMC_CLARiiON0_2 auto:cdsdisk EMC_CLARiiON0_4 dvgyappl online
EMC_CLARiiON0_3 auto:cdsdisk EMC_CLARiiON0_3 dvgy2503 online
EMC_CLARiiON0_4 auto:cdsdisk EMC_CLARiiON0_4 dvgy2504 online
EMC_CLARiiON0_5 auto:cdsdisk EMC_CLARiiON0_5 dvgy25 online
EMC_CLARiiON0_6 auto:cdsdisk EMC_CLARiiON0_9 dvgy26 online dgdisabled
EMC_CLARiiON0_7 auto:cdsdisk EMC_CLARiiON0_8 dygy2501 online
EMC_CLARiiON0_8 auto:cdsdisk - (dvgy2506) online
EMC_CLARiiON0_9 auto:cdsdisk - (dvgy2505) online
EMC_CLARiiON0_10 auto:cdsdisk - (dvgy2507) online
EMC_CLARiiON0_11 auto:cdsdisk EMC_CLARiiON0_11 dvgy25db2 online
This situation can happen when every disk in a disk group is lost from a bad power supply, power turned off to the disk array, cable disconnected, zoning problems, etc.
The disk group will not show in the output from
vxprint -ht.aixsrv01:#vxprint -htg dvgy26
VxVM vxprint ERROR V-5-1-582 Disk group dvgy26: No such disk group
The disk group will show as disabled in
vxdg list:aixsrv01:#vxdg list
NAME STATE ID
dygy2501 enabled,cds 1189621899.78.aixsrv01
dvgyappl enabled,cds 1190904062.52.aixsrv01
dvgy25 enabled,cds 1189622068.88.aixsrv01
dvgy25db2 enabled,cds 1189622043.86.aixsrv01
dvgy26 disabled 1189538508.74.aixsrv01
dvgy2503 enabled,cds 1189621988.82.aixsrv01
dvgy2504 enabled,cds 1189622014.84.aixsrv01
dygy2502 enabled,cds 1189621955.80.aixsrv01
This is the output of
vxdg list dvgy26:aixsrv01:#vxdg list dvgy26
Group: dvgy26
dgid: 1189538508.74.aixsrv01
import-id: 1024.22
flags: disabled
version: 0
alignment: 0 (bytes)
local-activation: read-write
ssb: off
detach-policy: invalid
copies: nconfig=default nlog=default
config: seqno=0.1103 permlen=1280 free=1259 templen=11 loglen=192
config disk EMC_CLARiiON0_6 copy 1 len=1280 state=clean online
log disk EMC_CLARiiON0_6 copy 1 len=192
Your filesystems will of course fail and the operating system will report it as corrupted.
aixsrv01:#df -k > /dev/null
df: /db2/dwins26q: I/O error
df: /backup: I/O error
df: /db/dwdb26q/dwins25q/NODE0000: I/O error
df: /db/dwins26q/dwdb26q/syscatspace/NODE0000: I/O error
df: /db/dwins26q/dwdb26q/tempspace01/NODE0000: I/O error
df: /dba/dwins26q: I/O error
df: /db2/dwmysld: I/O error
df: /backup/wiminst: I/O error
Once you have confirmed that the disk storage is powered-up, running, and operational and if the LUNs are in a SAN, zoning is configured right, this problem can be remedied by deporting, and then importing the disk group:
#vxdg deport dvgy26#vxdg import dvgy26
VxVM vxdg ERROR V-5-1-587 Disk group dvgy26: import failed: No valid disk found containing disk group
If volume manager can't see the disks, and your SAN or storage administrator has confirmed that the LUNs were fine and presented to your server, then rescan the disks.
aixsrv01:#vxdisk scandisksaixsrv01:#vxdctl enableaixsrv01:#vxdg import dvgy26
Otherwise, your diskgroup should be showing up as enabled.
aixsrv01:#vxdg list
NAME STATE ID
dygy2501 enabled,cds 1189621899.78.aixsrv01
dvgyappl enabled,cds 1190904062.52.aixsrv01
dvgy25 enabled,cds 1189622068.88.aixsrv01
dvgy25db2 enabled,cds 1189622043.86.aixsrv01
dvgy26 enabled,cds 1189538508.74.aixsrv01
dvgy2503 enabled,cds 1189621988.82.aixsrv01
dvgy2504 enabled,cds 1189622014.84.aixsrv01
dygy2502 enabled,cds 1189621955.80.aixsrv01
The disk group now shows in
vxprint -ht with the volumes and plexes disabled:aixsrv01:#vxprint -htg dvgy26
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
dg dvgy26 default default 9000 1189538508.74.aixsrv01
dm EMC_CLARiiON0_9 EMC_CLARiiON0_6 auto 2048 67102464 -
v backup - DISABLED ACTIVE 4194304 SELECT - fsgen
pl backup-01 backup DISABLED ACTIVE 4194304 CONCAT - RW
sd EMC_CLARiiON0_9-02 backup-01 EMC_CLARiiON0_9 8388608 4194304 0 EMC_CLARiiON0_6 ENA
v db - DISABLED ACTIVE 1048576 SELECT - fsgen
pl db-01 db DISABLED ACTIVE 1048576 CONCAT - RW
sd EMC_CLARiiON0_9-04 db-01 EMC_CLARiiON0_9 16777216 1048576 0 EMC_CLARiiON0_6 ENA
v dba - DISABLED ACTIVE 4194304 SELECT - fsgen
pl dba-01 dba DISABLED ACTIVE 4194304 CONCAT - RW
sd EMC_CLARiiON0_9-03 dba-01 EMC_CLARiiON0_9 12582912 4194304 0 EMC_CLARiiON0_6 ENA
v db2 - DISABLED ACTIVE 8388608 SELECT - fsgen
pl db2-01 db2 DISABLED ACTIVE 8388608 CONCAT - RW
sd EMC_CLARiiON0_9-01 db2-01 EMC_CLARiiON0_9 0 8388608 0 EMC_CLARiiON0_6 ENA
v dwmysld - DISABLED ACTIVE 2097152 SELECT - fsgen
pl dwmysld-01 dwmysld DISABLED ACTIVE 2097152 CONCAT - RW
sd EMC_CLARiiON0_9-09 dwmysld-01 EMC_CLARiiON0_9 55574528 2097152 0 EMC_CLARiiON0_6 ENA
v lg1 - DISABLED ACTIVE 10485760 SELECT - fsgen
pl lg1-01 lg1 DISABLED ACTIVE 10485760 CONCAT - RW
sd EMC_CLARiiON0_9-08 lg1-01 EMC_CLARiiON0_9 45088768 10485760 0 EMC_CLARiiON0_6 ENA
v syscat - DISABLED ACTIVE 2097152 SELECT - fsgen
pl syscat-01 syscat DISABLED ACTIVE 2097152 CONCAT - RW
sd EMC_CLARiiON0_9-05 syscat-01 EMC_CLARiiON0_9 17825792 2097152 0 EMC_CLARiiON0_6 ENA
v tp01 - DISABLED ACTIVE 4194304 SELECT - fsgen
pl tp01-01 tp01 DISABLED ACTIVE 4194304 CONCAT - RW
sd EMC_CLARiiON0_9-07 tp01-01 EMC_CLARiiON0_9 40894464 4194304 0 EMC_CLARiiON0_6 ENA
v ts01 - DISABLED ACTIVE 20971520 SELECT - fsgen
pl ts01-01 ts01 DISABLED ACTIVE 20971520 CONCAT - RW
sd EMC_CLARiiON0_9-06 ts01-01 EMC_CLARiiON0_9 19922944 20971520 0 EMC_CLARiiON0_6 ENA
Verify that the disks on the diskgroup are all online.
aixsrv01:#vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
EMC_CLARiiON0_0 auto:cdsdisk EMC_CLARiiON0_0 dygy2502 online
EMC_CLARiiON0_1 auto:cdsdisk - (dvgy2500) online
EMC_CLARiiON0_2 auto:cdsdisk EMC_CLARiiON0_4 dvgyappl online
EMC_CLARiiON0_3 auto:cdsdisk EMC_CLARiiON0_3 dvgy2503 online
EMC_CLARiiON0_4 auto:cdsdisk EMC_CLARiiON0_4 dvgy2504 online
EMC_CLARiiON0_5 auto:cdsdisk EMC_CLARiiON0_5 dvgy25 online
EMC_CLARiiON0_6 auto:cdsdisk EMC_CLARiiON0_9 dvgy26 online
EMC_CLARiiON0_7 auto:cdsdisk EMC_CLARiiON0_8 dygy2501 online
EMC_CLARiiON0_8 auto:cdsdisk - (dvgy2506) online
EMC_CLARiiON0_9 auto:cdsdisk - (dvgy2505) online
EMC_CLARiiON0_10 auto:cdsdisk - (dvgy2507) online
EMC_CLARiiON0_11 auto:cdsdisk EMC_CLARiiON0_11 dvgy25db2 online
Now the volumes can be started:
aixsrv01:#vxvol -g dvgy26 startallaixsrv01:#vxprint -htg dvgy26 | egrep '^v|^pl'
v backup - ENABLED ACTIVE 4194304 SELECT - fsgen
pl backup-01 backup ENABLED ACTIVE 4194304 CONCAT - RW
v db - ENABLED ACTIVE 1048576 SELECT - fsgen
pl db-01 db ENABLED ACTIVE 1048576 CONCAT - RW
v dba - ENABLED ACTIVE 4194304 SELECT - fsgen
pl dba-01 dba ENABLED ACTIVE 4194304 CONCAT - RW
v db2 - ENABLED ACTIVE 8388608 SELECT - fsgen
pl db2-01 db2 ENABLED ACTIVE 8388608 CONCAT - RW
v dwmysld - ENABLED ACTIVE 2097152 SELECT - fsgen
pl dwmysld-01 dwmysld ENABLED ACTIVE 2097152 CONCAT - RW
v lg1 - ENABLED ACTIVE 10485760 SELECT - fsgen
pl lg1-01 lg1 ENABLED ACTIVE 10485760 CONCAT - RW
v syscat - ENABLED ACTIVE 2097152 SELECT - fsgen
pl syscat-01 syscat ENABLED ACTIVE 2097152 CONCAT - RW
v tp01 - ENABLED ACTIVE 4194304 SELECT - fsgen
pl tp01-01 tp01 ENABLED ACTIVE 4194304 CONCAT - RW
v ts01 - ENABLED ACTIVE 20971520 SELECT - fsgen
pl ts01-01 ts01 ENABLED ACTIVE 20971520 CONCAT - RW
The filesystems on these volumes may not be in consistent state. So, run a filesystem check before mounting them.
aixsrv01:#for i in `grep dvgy26 /etc/filesystems | awk '{ print $3 }'`
> do
> fsck -y $i
> mount $i
> done
note: This example was taken from an AIX server, but all veritas commands here will work on all UNIX platforms.
Posted by
JAUGHN
Labels:
Veritas Volume Manager,
VxVM - Problem Solving
3 comments:
Thanks very much for this post. You are the man.
This is quite interesting, I got the same thing but without ANY impact. trying to fix it online now is a whole new puzzle...
DEVICE TYPE DISK GROUP STATUS
cciss/c0d0 auto:none - - online invalid
eva80000_0 auto:cdsdisk eva80000_0 dgsapXYexe online dgdisabled
eva80000_1 auto:cdsdisk eva80000_1 dgoraXYhome online dgdisabled
eva80000_2 auto:cdsdisk eva80000_2 dgsapXYarch online
eva80000_3 auto:cdsdisk eva80000_3 dgsapXYdata online dgdisabled
eva80000_4 auto:cdsdisk eva80000_4 dgsapXYdata online dgdisabled
eva80000_5 auto:cdsdisk eva80000_5 dgsapXYdata online dgdisabled
eva80000_6 auto:cdsdisk eva80000_6 dgsapXYmirrlog online dgdisabled
eva80000_7 auto:cdsdisk eva80000_7 dgsapXYoriglog online dgdisabled
eva80000_8 auto:none - - online invalid
eva80000_9 auto:none - - online invalid
eva80000_10 auto:none - - online invalid
eva80000_11 auto:none - - online invalid
eva80000_12 auto:none - - online invalid
eva80000_13 auto:none - - online invalid
eva80000_14 auto:none - - online invalid
eva80000_15 auto:none - - online invalid
eva80000_16 auto:none - - online invalid
eva80000_17 auto:none - - online invalid
xp10k-12k0_110b auto:cdsdisk xp10k-12k0_110b dgsapXXtranspool online dgdisabled
xp10k-12k0_110d auto:cdsdisk xp10k-12k0_110d dgsapXXtranspool online dgdisabled
xp10k-12k0_110f auto:cdsdisk xp10k-12k0_110f dgsapXXtranspool online dgdisabled
xp10k-12k0_1108 auto:cdsdisk xp10k-12k0_1108 dgsaplocal online dgdisabled
xp10k-12k0_1109 auto:cdsdisk xp10k-12k0_1109 dgsapXXtranspool online dgdisabled
xp10k-12k0_1111 auto:cdsdisk xp10k-12k0_1111 dgsapXXtranspool online dgdisabled
xp10k-12k0_1113 auto:cdsdisk xp10k-12k0_1113 dgsapXXtranspool online dgdisabled
xp10k-12k0_1115 auto:cdsdisk xp10k-12k0_1115 dgsapXXtranspool online dgdisabled
xp10k-12k0_1117 auto:cdsdisk xp10k-12k0_1117 dgsapXXtranspool online dgdisabled
xp10k-12k0_1119 auto:none - - online invalid
xp10k-12k0_1178 auto:cdsdisk xp10k-12k0_1178 dgsapXYnfs online dgdisabled
xp10k-12k1_110b auto:cdsdisk xp10k-12k1_110b dgsapXXtranspool online dgdisabled
xp10k-12k1_110d auto:cdsdisk xp10k-12k1_110d dgsapXXtranspool online dgdisabled
xp10k-12k1_110f auto:cdsdisk xp10k-12k1_110f dgsapXXtranspool online dgdisabled
xp10k-12k1_1108 auto:cdsdisk xp10k-12k1_1108 dgsaplocal online dgdisabled
xp10k-12k1_1109 auto:cdsdisk xp10k-12k1_1109 dgsapXXtranspool online dgdisabled
xp10k-12k1_1111 auto:cdsdisk xp10k-12k1_1111 dgsapXXtranspool online dgdisabled
xp10k-12k1_1113 auto:cdsdisk xp10k-12k1_1113 dgsapXXtranspool online dgdisabled
xp10k-12k1_1115 auto:cdsdisk xp10k-12k1_1115 dgsapXXtranspool online dgdisabled
xp10k-12k1_1117 auto:cdsdisk xp10k-12k1_1117 dgsapXXtranspool online dgdisabled
xp10k-12k1_1119 auto:none - - online invalid
xp10k-12k1_1178 auto:cdsdisk xp10k-12k1_1178 dgsapXYnfs online dgdisabled
all those FS are still working happily, whereas a vxdg list dgname shows it as totally broken.
I assume this broke over some san lun rescan.
Let me assure you I do NOT love those linux hosts, even with VxVM installed :)
It worked for me
Post a Comment