[conspire] Happy fun HD failure

[Rick Moen]

Every bit of data that matters on linuxmafia.com is on a mirror pair of
hard drives (SCSI HDs /dev/sdb and /dev/sdc, 18 GB each).  sdb just 
failed.

Hilarity ensues?


----- Forwarded message from mdadm monitoring <root at linuxmafia.com> -----

Date: Fri, 04 Jun 2010 13:07:56 -0700
From: mdadm monitoring <root at linuxmafia.com>
To: root at linuxmafia.com
Subject: Fail event on /dev/md4:linuxmafia.com

This is an automatically generated mail message from mdadm
running on linuxmafia.com

A Fail event had been detected on md device /dev/md4.

It could be related to component device /dev/sdb10.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] 
md0 : active raid1 sdb5[2](F) sdc5[1]
      1951744 blocks [2/1] [_U]
      
md1 : active raid1 sdb6[2](F) sdc6[1]
      1461760 blocks [2/1] [_U]
      
md2 : active raid1 sdb7[2](F) sdc7[1]
      2931712 blocks [2/1] [_U]
      
md3 : active raid1 sdb9[2](F) sdc9[1]
      5855552 blocks [2/1] [_U]
      
md4 : active raid1 sdb10[2](F) sdc10[1]
      5229056 blocks [2/1] [_U]
      
unused devices: <none>

----- End forwarded message -----
----- Forwarded message from mdadm monitoring <root at linuxmafia.com> -----

Date: Fri, 04 Jun 2010 13:07:56 -0700
From: mdadm monitoring <root at linuxmafia.com>
To: root at linuxmafia.com
Subject: Fail event on /dev/md3:linuxmafia.com

This is an automatically generated mail message from mdadm
running on linuxmafia.com

A Fail event had been detected on md device /dev/md3.

It could be related to component device /dev/sdb9.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] 
md0 : active raid1 sdb5[2](F) sdc5[1]
      1951744 blocks [2/1] [_U]
      
md1 : active raid1 sdb6[2](F) sdc6[1]
      1461760 blocks [2/1] [_U]
      
md2 : active raid1 sdb7[2](F) sdc7[1]
      2931712 blocks [2/1] [_U]
      
md3 : active raid1 sdb9[2](F) sdc9[1]
      5855552 blocks [2/1] [_U]
      
md4 : active raid1 sdb10[0] sdc10[1]
      5229056 blocks [2/2] [UU]
      
unused devices: <none>

----- End forwarded message -----
----- Forwarded message from mdadm monitoring <root at linuxmafia.com> -----

Date: Fri, 04 Jun 2010 13:07:56 -0700
From: mdadm monitoring <root at linuxmafia.com>
To: root at linuxmafia.com
Subject: Fail event on /dev/md2:linuxmafia.com

This is an automatically generated mail message from mdadm
running on linuxmafia.com

A Fail event had been detected on md device /dev/md2.

It could be related to component device /dev/sdb7.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] 
md0 : active raid1 sdb5[2](F) sdc5[1]
      1951744 blocks [2/1] [_U]
      
md1 : active raid1 sdb6[2](F) sdc6[1]
      1461760 blocks [2/1] [_U]
      
md2 : active raid1 sdb7[2](F) sdc7[1]
      2931712 blocks [2/1] [_U]
      
md3 : active raid1 sdb9[2](F) sdc9[1]
      5855552 blocks [2/1] [_U]
      
md4 : active raid1 sdb10[0] sdc10[1]
      5229056 blocks [2/2] [UU]
      
unused devices: <none>

----- End forwarded message -----
----- Forwarded message from mdadm monitoring <root at linuxmafia.com> -----

Date: Fri, 04 Jun 2010 13:07:56 -0700
From: mdadm monitoring <root at linuxmafia.com>
To: root at linuxmafia.com
Subject: Fail event on /dev/md1:linuxmafia.com

This is an automatically generated mail message from mdadm
running on linuxmafia.com

A Fail event had been detected on md device /dev/md1.

It could be related to component device /dev/sdb6.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] 
md0 : active raid1 sdb5[2](F) sdc5[1]
      1951744 blocks [2/1] [_U]
      
md1 : active raid1 sdb6[2](F) sdc6[1]
      1461760 blocks [2/1] [_U]
      
md2 : active raid1 sdb7[2](F) sdc7[1]
      2931712 blocks [2/1] [_U]
      
md3 : active raid1 sdb9[2](F) sdc9[1]
      5855552 blocks [2/1] [_U]
      
md4 : active raid1 sdb10[0] sdc10[1]
      5229056 blocks [2/2] [UU]
      
unused devices: <none>

----- End forwarded message -----
----- Forwarded message from mdadm monitoring <root at linuxmafia.com> -----

Date: Fri, 04 Jun 2010 13:07:55 -0700
From: mdadm monitoring <root at linuxmafia.com>
To: root at linuxmafia.com
Subject: Fail event on /dev/md0:linuxmafia.com

This is an automatically generated mail message from mdadm
running on linuxmafia.com

A Fail event had been detected on md device /dev/md0.

It could be related to component device /dev/sdb5.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] 
md0 : active raid1 sdb5[2](F) sdc5[1]
      1951744 blocks [2/1] [_U]
      
md1 : active raid1 sdb6[2](F) sdc6[1]
      1461760 blocks [2/1] [_U]
      
md2 : active raid1 sdb7[2](F) sdc7[1]
      2931712 blocks [2/1] [_U]
      
md3 : active raid1 sdb9[2](F) sdc9[1]
      5855552 blocks [2/1] [_U]
      
md4 : active raid1 sdb10[0] sdc10[1]
      5229056 blocks [2/2] [UU]
      
unused devices: <none>

----- End forwarded message -----