読者です 読者をやめる 読者になる 読者になる

自宅サーバのRAID障害が出た日

以前構築した自宅サーバのRAID環境から障害通知メールが来た。


件名:mdadm monitoring

This is an automatically generated mail message from mdadm
running on localhost.localdomain

A Fail event had been detected on md device /dev/md0.

It could be related to component device /dev/sdc1.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1]
md0 : active raid1 sdc1[2](F) sdb1[0]
     976759936 blocks [2/1] [U_]

unused devices: 

/var/log/messagesを確認してそれっぽいエラーが出ているのがここらへん。

ar 18 05:43:04 localhost kernel: SCSI device sdc: 1953525168 512-byte hdwr sectors (1000205 MB)
Mar 18 05:43:11 localhost kernel: sdc: Write Protect is off
Mar 18 05:43:12 localhost kernel: SCSI device sdc: drive cache: write back
Mar 18 05:44:12 localhost kernel: ata2.01: exception Emask 0x10 SAct 0x0 SErr 0x400100 action 0x6 frozen
Mar 18 05:44:12 localhost kernel: ata2: SError: { UnrecovData Handshk }
Mar 18 05:44:12 localhost kernel: ata2.01: cmd 35/00:00:3f:05:c3/00:04:09:00:00/f0 tag 0 dma 524288 out
Mar 18 05:44:12 localhost kernel:          res 40/00:cf:61:05:c3/40:03:09:00:00/f0 Emask 0x14 (ATA bus error)             169,51        34%
Mar 18 05:43:04 localhost kernel: sdc: Current [descriptor]: sense key: Medium Error
Mar 18 05:43:04 localhost kernel:     Add. Sense: Unrecovered read error - auto reallocate failed
Mar 18 05:43:04 localhost kernel:
Mar 18 05:43:04 localhost kernel: Descriptor sense data with sense descriptors (in hex):
Mar 18 05:43:04 localhost kernel:         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Mar 18 05:43:04 localhost kernel:         09 c3 05 61
Mar 18 05:43:04 localhost kernel: end_request: I/O error, dev sdc, sector 163775841
Mar 18 05:43:04 localhost kernel: ata2: EH complete                                                                                         
Mar 18 05:43:04 localhost kernel: SCSI device sdc: 1953525168 512-byte hdwr sectors (1000205 MB)
Mar 18 05:43:11 localhost kernel: sdc: Write Protect is off
Mar 18 05:43:12 localhost kernel: SCSI device sdc: drive cache: write back
Mar 18 05:44:12 localhost kernel: ata2.01: exception Emask 0x10 SAct 0x0 SErr 0x400100 action 0x6 frozen
Mar 18 05:44:12 localhost kernel: ata2: SError: { UnrecovData Handshk }
Mar 18 05:44:12 localhost kernel: ata2.01: cmd 35/00:00:3f:05:c3/00:04:09:00:00/f0 tag 0 dma 524288 out
Mar 18 05:44:12 localhost kernel:          res 40/00:cf:61:05:c3/40:03:09:00:00/f0 Emask 0x14 (ATA bus error)
                                                                                                                          169,51        33%
Mar 18 05:43:04 localhost kernel: sdc: Current [descriptor]: sense key: Medium Error
Mar 18 05:43:04 localhost kernel:     Add. Sense: Unrecovered read error - auto reallocate failed
Mar 18 05:43:04 localhost kernel:
Mar 18 05:43:04 localhost kernel: Descriptor sense data with sense descriptors (in hex):
Mar 18 05:43:04 localhost kernel:         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Mar 18 05:43:04 localhost kernel:         09 c3 05 61
Mar 18 05:43:04 localhost kernel: end_request: I/O error, dev sdc, sector 163775841
Mar 18 05:43:04 localhost kernel: ata2: EH complete                                                                                         
Mar 18 05:43:04 localhost kernel: SCSI device sdc: 1953525168 512-byte hdwr sectors (1000205 MB)
Mar 18 05:43:11 localhost kernel: sdc: Write Protect is off
Mar 18 05:43:12 localhost kernel: SCSI device sdc: drive cache: write back
Mar 18 05:44:12 localhost kernel: ata2.01: exception Emask 0x10 SAct 0x0 SErr 0x400100 action 0x6 frozen
Mar 18 05:44:12 localhost kernel: ata2: SError: { UnrecovData Handshk }
Mar 18 05:44:12 localhost kernel: ata2.01: cmd 35/00:00:3f:05:c3/00:04:09:00:00/f0 tag 0 dma 524288 out
Mar 18 05:44:12 localhost kernel:          res 40/00:cf:61:05:c3/40:03:09:00:00/f0 Emask 0x14 (ATA bus error)
Mar 18 05:44:12 localhost kernel: ata2.01: status: { DRDY }                                                               169,51        34%
Mar 18 05:44:23 localhost kernel: ata2: hard resetting link
Mar 18 05:44:24 localhost kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 18 05:44:24 localhost kernel: ata2.01: revalidation failed (errno=-2)
Mar 18 05:44:24 localhost kernel: ata2.01: disabled
Mar 18 05:44:25 localhost kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x40)
Mar 18 05:44:25 localhost kernel: ata2.00: revalidation failed (errno=-5)
Mar 18 05:44:25 localhost kernel: ata2: failed to recover some devices, retrying in 5 secs
Mar 18 05:44:30 localhost kernel: ata2: hard resetting link                                                                                 
Mar 18 05:44:30 localhost kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Mar 18 05:44:30 localhost kernel: ata2.00: configured for UDMA/133
Mar 18 05:44:30 localhost kernel: ata2: EH complete

以下らへんを参考に復旧作業中だがだめだったらディスク買い換えるか、、、、

CentOSでソフトウェアRAIDの構築
smartd での sector error 復活作業
ext3でフォーマットする方法
HDDの不良ブロックをどうにかできないか – ubuntu badblocks mke2fs e2fsck

【追記】
badblocksコマンドで/dev/sdcのディスクについて不良セクタのチェックを行ったが特に検出されなかった(1TBのディスクなので--fullオプションをつけて実行したら一週間ぐらいかかった)
とりあえずRAID1を構築しなおして/dev/md0のRAIDデバイスについてext3でフォーマットしなおしたら復旧した。
ソフトウェアRAIDの書き込み失敗とかそういう論理的な問題だったのだろうか、、、、