ASM丢失disk header导致ORA-15032、ORA-15040、ORA-15042 Diskgroup无法mount

ASM丢失disk header导致ORA-15032、ORA-15040、ORA-15042 Diskgroup无法mount的案例不少，这里我们介绍下如何解决。         SQL> select * from v$version; BANNER -------------------------------------------------------------------------------- Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production PL/SQL Release 11.2.0.3.0 - Production CORE 11.2.0.3.0 Production TNS for Linux: Version 11.2.0.3.0 - Production NLSRTL Version 11.2.0.3.0 - Production   SQL> alter diskgroup datadg mount; alter diskgroup datadg mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "5" is missing from group number "1" ERROR: alter diskgroup datadg mount Wed Mar 13 07:42:03 2013   SQL> alter diskgroup datadg mount NOTE: cache registered group DATADG number=1 incarn=0xccb845cd NOTE: cache began mount (first) of group DATADG number=1 incarn=0xccb845cd NOTE: Assigning number (1,2) to disk (/dev/asm-diskg) NOTE: Assigning number (1,1) to disk (/dev/asm-diskf) NOTE: Assigning number (1,0) to disk (/dev/asm-diske) Wed Mar 13 16:42:09 2013 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 20 for pid 27, osid 5439 NOTE: Assigning number (1,5) to disk () GMON querying group 1 at 21 for pid 27, osid 5439 NOTE: cache dismounting (clean) group 1/0xCCB845CD (DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 5439, image: oracle@vmac1 (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xCCB845CD (DATADG) NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xccb845cd NOTE: cache deleting context for group DATADG 1/0xccb845cd GMON dismounting group 1 at 22 for pid 27, osid 5439 NOTE: Disk in mode 0x8 marked for de-assignment NOTE: Disk in mode 0x8 marked for de-assignment NOTE: Disk in mode 0x8 marked for de-assignment NOTE: Disk in mode 0x8 marked for de-assignment ERROR: diskgroup DATADG was not mounted ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "5" is missing from group number "1" ERROR: alter diskgroup datadg mount Wed Mar 13 16:42:10 2013 ASM Health Checker found 1 new failures   [grid@vmac1 ~]$ kfed read /dev/asm-diskh kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 7FA1DA233400 00000000 00000000 00000000 00000000 [................] Repeat 255 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0] col path for a20 set linesize 200 pagesize 1400 select path,header_status,state from v$asm_disk; PATH HEADER_STATUS STATE -------------------- ------------------------------------ ------------------------ /dev/asm-diskh CANDIDATE NORMAL /dev/asm-diskg MEMBER NORMAL /dev/asm-diskf MEMBER NORMAL /dev/asm-diske MEMBER NORMAL /dev/asm-diskc MEMBER NORMAL /dev/asm-diskd MEMBER NORMAL /dev/asm-diskb MEMBER NORMAL 7 rows selected. [grid@vmac1 ~]$ kfed repair /dev/asm-diskh KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]   [grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576 KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]     关闭ASM实例       优先备份 问题ASM的header dd if=<bad disk> of=<file> bs=4096 count=1       3. 检查现有磁盘找出拥有file 1 block 1的 [grid@vmac1 ~]$ kfed read /dev/asm-diske |grep f1b1 kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002 [grid@vmac1 ~]$ [grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1 kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000 [grid@vmac1 ~]$ kfed read /dev/asm-diskg |grep f1b1 kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000   这里asm-diske上出现了f1b1非零值，则其拥有file 1 block 1，可以通过检查第二个au的类型是否是KFBTYP_LISTHEAD来确认 [grid@vmac1 ~]$ kfed read /dev/asm-diske aun=2|grep kfbh.type kfbh.type: 5 ; 0x002: KFBTYP_LISTHEAD     若丢失的磁盘包含了"file 1 block 1 F1B1"则扫描 该磁盘上所有的 AU直到找到KFBTYP_LISTHEAD ， 如果找不到LISTHEAD 那么别无选择只能重建diskgroup。   从版本11.1.0.7开始(10g是从10.2.0.5开始，所以尽量别用10.2.0.5之前的版本上的ASM)，当每一个I/O写提交向ASM disk header(AU 0 blocknum 0)，都会复制到 AU 1中，最后第二个块。基于不同的AU size，该块的位置不同 Allocation Unit Size Block Number on AU 1 1048576 254 4194304 1022 8388608 2046 16777216 4094   首先使用kfed 验证该位置是否有正确的disk header，否则手动找到合适的header。 表1如参考： [grid@vmac1 ~]$ kfed read /dev/asm-diske ausz=1048576 aun=1 blkn=254|less kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 254 ; 0x004: blk=254 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 2086475720 ; 0x00c: 0x7c5d17c8 kfbh.fcn.base: 31322 ; 0x010: 0x00007a5a kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 之后使用kfed repair命令修复disk header [grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576     如果disk header的自动备份也丢失了，那么会报如下错误 KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]   若上述kfed repair无效则需要通过 手动恢复disk header的办法了：   对于版本10.2.0.5之前没有kred repair可用的情况：   找一个与问题disk 在同一个diskgroup中的，不包含f1b1 的好的disk header。 例如这里的asm-diskf [grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1 kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000   使用kfed read <device name> > fix.txt 命令保存其头部 [grid@vmac1 ~]$ kfed read /dev/asm-diskf > fix.txt 编辑fix.txt 修改kfdhdb.dsknum 、kfdhdb.dskname 、 kfdhdb.fgname 三个信息： 同时参考alert.log中的信息： [grid@vmac1 trace]$ grep "cache opening" alert_+ASM1.log NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/asm-diskf NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/asm-diskg NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/asm-diskf NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/asm-diskg NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diske NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/asm-diskf NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/asm-diskg NOTE: cache opening disk 0 of grp 2: SYSTEDG_0000 path:/dev/asm-diskb NOTE: cache opening disk 1 of grp 2: SYSTEDG_0001 path:/dev/asm-diskc NOTE: cache opening disk 2 of grp 2: SYSTEDG_0002 path:/dev/asm-diskd NOTE: cache opening disk 5 of grp 1: DATADG_0005 path:/dev/asm-diskh     原fix.txt中的内容： [grid@vmac1 ~]$ egrep "dsknum|grptyp|hdrsts|dskname|grpname|fgname" fix.txt kfdhdb.dsknum: 1 ; 0x024: 0x0001 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: DATADG_0001 ; 0x028: length=11 kfdhdb.grpname: DATADG ; 0x048: length=6 kfdhdb.fgname: DATADG_0001 ; 0x068: length=11   修改后： [grid@vmac1 ~]$ egrep "dsknum|grptyp|hdrsts|dskname|grpname|fgname" fix.txt kfdhdb.dsknum: 5 ; 0x024: 0x0005 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: DATADG_0005 ; 0x028: length=11 kfdhdb.grpname: DATADG ; 0x048: length=6 kfdhdb.fgname: DATADG_0005 ; 0x068: length=11   还需要修改 kfbh.block.obj [grid@vmac1 ~]$ grep kfbh.block.obj fix.txt kfbh.block.obj: 2147483649 ; 0x008: disk=1 2147483649==》0x80000001 0x80000001 最后一位是ASM DISK NUMBER , 应当等于kfdhdb.dsknum ， 在这里为 0x80000005 ==》 2147483653 [grid@vmac1 ~]$ grep kfbh.block.obj fix.txt kfbh.block.obj: 2147483653 ; 0x008: disk=5   若使用windows平台上的ASMLIB则还要麻烦地修改kfdhdb.driver.reserved[0]， 所幸很少人会用Windows上的ASMLIB       接着查看aunum=2 blknum=2来查找 disk directory， kfed read 拥有 f1b1的disk的 aunum=2 blknum=位置: kfed read <device name> aunum=2 blknum=2 | more [grid@vmac1 ~]$ kfed read /dev/asm-diske|grep f1b1 kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002   [grid@vmac1 ~]$ kfed read /dev/asm-diske aunum=2 blknum=2|more kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 4 ; 0x002: KFBTYP_FILEDIR kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 2 ; 0x004: blk=2 kfbh.block.obj: 1 ; 0x008: file=1 kfbh.check: 322527999 ; 0x00c: 0x133962ff kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfffdb.node.incarn: 1 ; 0x000: A=1 NUMM=0x0 kfffdb.node.frlist.number: 4294967295 ; 0x004: 0xffffffff ......................... kfffde[0].xptr.au: 3 ; 0x4a0: 0x00000003 kfffde[0].xptr.disk: 0 ; 0x4a4: 0x0000 kfffde[0].xptr.flags: 0 ; 0x4a6: L=0 E=0 D=0 S=0 kfffde[0].xptr.chk: 41 ; 0x4a7: 0x29 kfffde[1].xptr.au: 4294967295 ; 0x4a8: 0xffffffff kfffde[1].xptr.disk: 65535 ; 0x4ac: 0xffff kfffde[1].xptr.flags: 0 ; 0x4ae: L=0 E=0 D=0 S=0 kfffde[1].xptr.chk: 42 ; 0x4af: 0x2a ==》disk directory 位于 disk=0 的 aunum=3 NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske ==> 还是 asm-diske [grid@vmac1 ~]$ kfed read /dev/asm-diske aunum=3 blknum=0|more kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 6 ; 0x002: KFBTYP_DISKDIR kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2 ; 0x008: file=2 kfbh.check: 389127513 ; 0x00c: 0x17319d59 kfbh.fcn.base: 31299 ; 0x010: 0x00007a43 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kffdnd.bnode.incarn: 1 ; 0x000: A=1 NUMM=0x0 kffdnd.bnode.frlist.number: 4294967295 ; 0x004: 0xffffffff kffdnd.bnode.frlist.incarn: 0 ; 0x008: A=0 NUMM=0x0 kffdnd.overfl.number: 4294967295 ; 0x00c: 0xffffffff kffdnd.overfl.incarn: 0 ; 0x010: A=0 NUMM=0x0 kffdnd.parent.number: 0 ; 0x014: 0x00000000 kffdnd.parent.incarn: 1 ; 0x018: A=1 NUMM=0x0 kffdnd.fstblk.number: 0 ; 0x01c: 0x00000000 kffdnd.fstblk.incarn: 1 ; 0x020: A=1 NUMM=0x0 kfddde[0].entry.incarn: 1 ; 0x024: A=1 NUMM=0x0 kfddde[0].entry.hash: 0 ; 0x028: 0x00000000 kfddde[0].entry.refer.number:4294967295 ; 0x02c: 0xffffffff kfddde[0].entry.refer.incarn: 0 ; 0x030: A=0 NUMM=0x0 .......................................................... kfddde 结构为 disk directory结构，仅有kfddde[0].entry.incarn A=1的记录是已分配的记录，若A=0则说明该记录被删除。 [grid@vmac1 ~]$ grep "kfddde\[5\]" disk.txt kfddde[5].entry.incarn: 1 ; 0x8e4: A=1 NUMM=0x0 kfddde[5].entry.hash: 5 ; 0x8e8: 0x00000005 kfddde[5].entry.refer.number:4294967295 ; 0x8ec: 0xffffffff kfddde[5].entry.refer.incarn: 0 ; 0x8f0: A=0 NUMM=0x0 kfddde[5].dsknum: 5 ; 0x8f4: 0x0005 kfddde[5].state: 2 ; 0x8f6: KFDSTA_NORMAL kfddde[5].ddchgfl: 132 ; 0x8f7: 0x84 kfddde[5].dskname: DATADG_0005 ; 0x8f8: length=11 kfddde[5].fgname: DATADG_0005 ; 0x918: length=11 kfddde[5].crestmp.hi: 32984459 ; 0x938: HOUR=0xb DAYS=0xc MNTH=0x3 YEAR=0x7dd kfddde[5].crestmp.lo: 2470649856 ; 0x93c: USEC=0x0 MSEC=0xc8 SECS=0x34 MINS=0x24 kfddde[5].failstmp.hi: 0 ; 0x940: HOUR=0x0 DAYS=0x0 MNTH=0x0 YEAR=0x0 kfddde[5].failstmp.lo: 0 ; 0x944: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0 kfddde[5].timer: 0 ; 0x948: 0x00000000 kfddde[5].size: 5120 ; 0x94c: 0x00001400 kfddde[5].srRloc.super.hiStart: 0 ; 0x950: 0x00000000 kfddde[5].srRloc.super.loStart: 0 ; 0x954: 0x00000000 kfddde[5].srRloc.super.length: 0 ; 0x958: 0x00000000 kfddde[5].srRloc.incarn: 0 ; 0x95c: 0x00000000 kfddde[5].dskrprtm: 0 ; 0x960: 0x00000000 kfddde[5].zones[0].start: 0 ; 0x964: 0x00000000 kfddde[5].zones[0].size: 5120 ; 0x968: 0x00001400 kfddde[5].zones[0].used: 2 ; 0x96c: 0x00000002   回到编辑fix.txt上来， 调整 crestmp.hi 和 crestmp.lo 匹配上面显示的信息，若已经匹配则无需修改。 原本 [grid@vmac1 ~]$ egrep "hi|lo" fix.txt kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483653 ; 0x008: disk=5 kfdhdb.crestmp.hi: 32983191 ; 0x0a8: HOUR=0x17 DAYS=0x4 MNTH=0x2 YEAR=0x7dd kfdhdb.crestmp.lo: 2328519680 ; 0x0ac: USEC=0x0 MSEC=0x299 SECS=0x2c MINS=0x22 kfdhdb.mntstmp.hi: 32984468 ; 0x0b0: HOUR=0x14 DAYS=0xc MNTH=0x3 YEAR=0x7dd kfdhdb.mntstmp.lo: 1231840256 ; 0x0b4: USEC=0x0 MSEC=0x319 SECS=0x16 MINS=0x12 kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001 kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002 kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000 kfdhdb.grpstmp.hi: 32983191 ; 0x0e4: HOUR=0x17 DAYS=0x4 MNTH=0x2 YEAR=0x7dd kfdhdb.grpstmp.lo: 2328331264 ; 0x0e8: USEC=0x0 MSEC=0x1e1 SECS=0x2c MINS=0x22 修改后 kfdhdb.crestmp.hi: 32984459 ; 0x938: HOUR=0xb DAYS=0xc MNTH=0x3 YEAR=0x7dd kfdhdb.crestmp.lo: 2470649856 ; 0x93c: USEC=0x0 MSEC=0xc8 SECS=0x34 MINS=0x24 kfdhdb.mntstmp.hi: 32984468 ; 0x0b0: HOUR=0x14 DAYS=0xc MNTH=0x3 YEAR=0x7dd kfdhdb.mntstmp.lo: 1231840256 ; 0x0b4: USEC=0x0 MSEC=0x319 SECS=0x16 MINS=0x12   之后使用kfed merge命令合并disk header kfed merge <device name> text=fix.txt [grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt   若使用ASMLIb，则使用如下命令修复header中的asmlib信息   /etc/init.d/oracleasm force-renamedisk /dev/sdbg1 <ASMLIB Disk Name> /etc/init.d/oracleasm scandisks /etc/init.d/oracleasm listdisks     之后启动ASM实例到nomount SQL> startup nomount; SQL> col path for a20 SQL> set linesize 200 pagesize 1400 SQL> select path,header_status,state from v$asm_disk; PATH HEADER_STATUS STATE -------------------- ------------------------------------ ------------------------ /dev/asm-diskh MEMBER NORMAL /dev/asm-diskg MEMBER NORMAL /dev/asm-diskf MEMBER NORMAL /dev/asm-diske MEMBER NORMAL /dev/asm-diskc MEMBER NORMAL /dev/asm-diskd MEMBER NORMAL /dev/asm-diskb MEMBER NORMAL 7 rows selected. 检查头部信息是否为member       [grid@vmac1 ~]$ kfed read /dev/asm-diskh kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483653 ; 0x008: disk=5 kfbh.check: 3412972861 ; 0x00c: 0xcb6dd53d kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8 kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000 kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000   之后 alter diskgroup 问题DG mount; 若上述步骤无问题则可以正常mount dg; SQL> SQL> alter diskgroup datadg mount; Diskgroup altered. NOTE: cache registered group DATADG number=1 incarn=0x01c845f0 NOTE: cache began mount (first) of group DATADG number=1 incarn=0x01c845f0 NOTE: Assigning number (1,5) to disk (/dev/asm-diskh) NOTE: Assigning number (1,2) to disk (/dev/asm-diskg) NOTE: Assigning number (1,1) to disk (/dev/asm-diskf) NOTE: Assigning number (1,0) to disk (/dev/asm-diske) Wed Mar 13 19:39:49 2013 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 56 for pid 27, osid 8690 NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diske NOTE: F1X0 found on disk 0 au 2 fcn 0.31322 NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/asm-diskf NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/asm-diskg NOTE: cache opening disk 5 of grp 1: DATADG_0005 path:/dev/asm-diskh NOTE: cache mounting (first) external redundancy group 1/0x01C845F0 (DATADG) Wed Mar 13 19:39:49 2013 * allocate domain 1, invalid = TRUE kjbdomatt send to inst 2 Wed Mar 13 19:39:49 2013 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=11.2351 group=1 (DATADG) NOTE: advancing ckpt for group 1 (DATADG) thread=1 ckpt=11.2351 NOTE: cache recovered group 1 to fcn 0.33763 NOTE: redo buffer size is 256 blocks (1053184 bytes) Wed Mar 13 19:39:49 2013 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATADG) NOTE: LGWR found thread 1 closed at ABA 11.2350 NOTE: LGWR mounted thread 1 for diskgroup 1 (DATADG) NOTE: LGWR opening thread 1 at fcn 0.33763 ABA 12.2351 NOTE: cache mounting group 1/0x01C845F0 (DATADG) succeeded NOTE: cache ending mount (success) of group DATADG number=1 incarn=0x01c845f0 GMON querying group 1 at 57 for pid 18, osid 2911 Wed Mar 13 19:39:49 2013 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATADG was mounted SUCCESS: alter diskgroup datadg mount Wed Mar 13 19:39:49 2013 NOTE: diskgroup resource ora.DATADG.dg is online NOTE: diskgroup resource ora.DATADG.dg is updated Wed Mar 13 19:39:59 2013 NOTE: client PROD1:PROD registered, osid 10169, mbr 0x1 Wed Mar 13 19:40:11 2013 NOTE: ASM client PROD1:PROD disconnected unexpectedly. NOTE: check client alert log. NOTE: Trace records dumped in trace file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_10169.trc         注意 以上修复仅针对 asm header丢失磁盘头大约20MB(不一定)空间的数据， 若更多则仍可能无法mount diskgroup 若file numbe=4 Continuing Operations Directory (COD) - ASM file number 4也丢失了 那么一般很难mount了，在下面的例子中是前22MB丢失时即便kfed merge也无力回天了。 SELECT x.xnum_kffxp "Extent", x.au_kffxp "AU", x.disk_kffxp "Disk #", d.name "Disk name" FROM x$kffxp x, v$asm_disk_stat d WHERE x.group_kffxp=d.group_number and x.disk_kffxp=d.disk_number and x.group_kffxp=1 and x.number_kffxp=4 ORDER BY 1, 2; Extent AU Disk # Disk name ---------- ---------- ---------- ------------------------------------------------------------------------------------------ 0 21 5 DATADG_0005 1 16 1 DATADG_0001 2 33 2 DATADG_0002 3 34 0 DATADG_0000 4 22 5 DATADG_0005 5 34 2 DATADG_0002 6 35 0 DATADG_0000 7 33 1 DATADG_0001 SQL> alter diskgroup datadg dismount; Diskgroup altered. [grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=20 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.0165823 s, 1.3 GB/s   [grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt SQL> alter diskgroup datadg mount; Diskgroup altered. SQL> alter diskgroup datadg mount; Diskgroup altered. SQL> alter diskgroup datadg dismount; Diskgroup altered. [grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=21 21+0 records in 21+0 records out 22020096 bytes (22 MB) copied, 0.0182842 s, 1.2 GB/s [grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt SQL> alter diskgroup datadg mount; Diskgroup altered.   [grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=22 22+0 records in 22+0 records out 23068672 bytes (23 MB) copied, 0.0312157 s, 739 MB/s [grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt SQL> alter diskgroup datadg mount; alter diskgroup datadg mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15130: diskgroup "DATADG" is being dismounted ORA-15066: offlining disk "DATADG_0005" in group "DATADG" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]   Errors in file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8690.trc: ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] ERROR: cache failed to read group=1(DATADG) fn=4 blk=0 from disk(s): 5(DATADG_0005) ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] NOTE: cache initiating offline of disk 5 group DATADG NOTE: process _user8690_+asm1 (8690) initiating offline of disk 5.3915953639 (DATADG_0005) with mask 0x7e in group 1 WARNING: Disk 5 (DATADG_0005) in group 1 in mode 0x7f is now being taken offline on ASM inst 1 NOTE: initiating PST update: grp = 1, dsk = 5/0xe968b5e7, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 108 for pid 27, osid 8690 ERROR: Disk 5 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) WARNING: Offline of disk 5 (DATADG_0005) in group 1 and mode 0x7f failed on ASM inst 1 Wed Mar 13 20:00:56 2013 NOTE: halting all I/Os to diskgroup 1 (DATADG) System State dumped to trace file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8690.trc NOTE: AMDU dump of disk group DATADG created at /g01/orabase/diag/asm/+asm/+ASM1/trace ERROR: ORA-15130 signalled during mount of diskgroup DATADG NOTE: cache dismounting (clean) group 1/0xEB784617 (DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 8690, image: oracle@vmac1 (TNS V1-V3) NOTE: LGWR doing non-clean dismount of group 1 (DATADG) NOTE: LGWR sync ABA=18.2360 last written ABA 18.2360 kjbdomdet send to inst 2 detach from dom 1, sending detach message to inst 2 Wed Mar 13 20:00:57 2013 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 1, cluster inc 12) Global Resource Directory partially frozen for dirty detach * dirty detach - domain 1 invalid = TRUE 0 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete freeing rdom 1 WARNING: dirty detached from domain 1 NOTE: cache dismounted group 1/0xEB784617 (DATADG) NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xeb784617 NOTE: cache deleting context for group DATADG 1/0xeb784617 GMON dismounting group 1 at 109 for pid 27, osid 8690 NOTE: Disk in mode 0x8 marked for de-assignment NOTE: Disk in mode 0x8 marked for de-assignment NOTE: Disk in mode 0x8 marked for de-assignment NOTE: Disk in mode 0x8 marked for de-assignment ERROR: diskgroup DATADG was not mounted ORA-15032: not all alterations performed ORA-15130: diskgroup "DATADG" is being dismounted ORA-15066: offlining disk "DATADG_0005" in group "DATADG" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] ERROR: alter diskgroup datadg mount SQL> alter diskgroup datadg mount; alter diskgroup datadg mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15130: diskgroup "DATADG" is being dismounted ORA-15066: offlining disk "DATADG_0005" in group "DATADG" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 !=1] ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 !=1] [grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=4096 skip=3 seek=3 count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000617397 s, 6.6 MB/s   [grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=4096 skip=4 seek=4 count=1 kfffde[0].xptr.au: 21 ; 0x4a0: 0x00000015 kfffde[0].xptr.disk: 5 ; 0x4a4: 0x0005 kfffde[0].xptr.flags: 0 ; 0x4a6: L=0 E=0 D=0 S=0 kfffde[0].xptr.chk: 58 ; 0x4a7: 0x3a kfffde[1].xptr.au: 16 ; 0x4a8: 0x00000010   [grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=1048576 skip=21 seek=21 count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00296742 s, 353 MB/s   ORA-15066: offlining disk "DATADG_0005" in group "DATADG" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26077] [obj_kfbl] [4] [0] [3 != 4] ORA-15196: invalid ASM block header [kfc.c:26077] [obj_kfbl] [4] [0] [3 != 4] ERROR: alter diskgroup datadg mount force