Storage’R’Us

As you will recall (I command you!), I hit on the (frankly indefensible) idea of not deleting ripped DVDs and Bluray disks after I’ve watched them.  Because reasons.  So I ended up with a 12TB disk system to store all that.

And, surprise, half a year later that’s beginning to look too small:

big-tex:~# df -h /dvd/
Filesystem Size Used Avail Use% Mounted on
/dev/md0   11T  9.7T 1.4T  89%  /dvd

So now what?

In the mean time, the nice people at Seagate have started to make 8TB “archival” disks. And somebody was selling a 4 bay fanless disk dock.  Hm…

Surprise epic unboxing sequence!

Oops.  I forgot to take all the pictures in between.

Anyway, you can see that this is a “dock” type thing with the disks poking out on top, and not a cabinet.  That’s not very confidence inspiring.

It has both USB3 and e-SATA port multiplier functionality.  I decided to go with e-SATA, since USB is less stable than it should be.

The tiny Fit-PC3 machine in the cupboard there has an AMD SATA controller.

00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] 
        SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)

Apparently, some BIOS-es don’t enable the port multiplier (PM) stuff, but everything just worked on the Fit-PC3: All four disks showed up as /dev/sdx devices.

I wondered what the raw read speed would be from one of these disks via the port multiplier stuff, and I was shocked.  I got 190MB/s when reading the raw device sequentially.  That’s pretty awesome.

Then I put the four disks into a RAID5 configuration and waited a couple of days.  Let’s check in now:

big-tex:~# cat /proc/mdstat 
md2 : active raid5 sdg[4] sdf[2] sde[1] sdd[0]
 23441685504 blocks super 1.2 level 5, 512k chunk, algorithm 2 
             [4/3] [UUU_]
 [===================>.] recovery = 98.3% (7687937924/7813895168) 
                         finish=54.5min speed=38507K/sec

Hey! It’s almost done!  It took about 45 hours, though, and the speed varied between 50MB/s and 30MB/s.

When reading sequentially from the raw md device, I get a read speed of 140MB/s.  Not great, but not awful, either.

big-tex:~# df -h /newdvd/
Filesystem Size Used Avail Use% Mounted on
/dev/md2   22T  20K  21T   1%   /newdvd

Whoho!  I put ext4 on the RAID set.  The write speed is pretty awful: About 30MB/s.  The read speed is about 50MB/s.  That’s worse than I had expected.  Is it the e-SATA port multiplier that makes things that slow?  Weird.

Hm…   Mounting with -o data=writeback seems to help a bit.  I then get 45MB/s.  With the journal completely switched off, I get 54MB/s.  Still really slow.  Perhaps the CPU is underpowered?  But read speed shouldn’t be an issue CPU wise…  The read speed is really puzzling.

Anyway, I started rsyncing over the files, and I get about 35MB/s when doing that.  *sigh*  This is going to take a few days, especially since it seems like the Seagate disks sometimes go catatonic, and you can’t write anything to them for hours at a time…  Binging around a bit, it seems like this is a known issue with these 8T disks: Sometimes the write speed drops down to zero.  But they usually recover again.

*time passes*

After four days, all the .mkv files have been rsynced.  The write speed hickup didn’t occur again, so I’ve now got 10GB of movie goodness on the new RAID set.

Time for some benchmarking.

[larsi@big-tex ~]$ cat /dvd/*/*.mkv | pv > /dev/null
27.3GB 0:01:52 [ 294MB/s] [<=> ]

Wha…?  That’s, like, really fast.  Is pv miscalculating, or something?

Nope:

[larsi@big-tex ~]$ vmstat 60
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r b swpd free buff cache si so bi bo in cs us sy id wa
 1 0 0 30712 21504 3439784 0 0 295779 1 2876 1242 11 79 10 0
 1 0 0 33004 21556 3437512 0 0 292166 2 2850 1246 11 78 11 1

Geez.  I’m really getting read speeds of 290MB/s, which is kinda spiffy.

So!  Success!  Now let’s see whether these disks survive being used for a while…

Edit a couple hours later:

May 14 20:52:25 big-tex kernel: [ 3189.109798] ata5.03: SError: { Handshk }
 May 14 20:52:25 big-tex kernel: [ 3189.109907] ata5.03: failed command: WRITE FP
 DMA QUEUED
 May 14 20:52:25 big-tex kernel: [ 3189.110021] ata5.03: cmd 61/08:70:18:03:04/00
 :00:00:00:00/40 tag 14 ncq 4096 out
 May 14 20:52:25 big-tex kernel: [ 3189.110021] res 41/84:08:18:03:04/00:00:00:00:00/00 Emask 0x410 (ATA bus error) 
 May 14 20:52:25 big-tex kernel: [ 3189.110405] ata5.03: status: { DRDY ERR }
 May 14 20:52:25 big-tex kernel: [ 3189.110511] ata5.03: error: { ICRC ABRT }
 May 14 20:52:25 big-tex kernel: [ 3189.112594] ata5.03: hard resetting link
 ...
 May 14 20:53:32 big-tex kernel: [ 3255.728753] ata5.03: detaching (SCSI 4:3:0:0)

Well, that didn’t take long.  All the disks went AWOL, and the RAID failed.  A reboot brought the RAID back up again, but it’s going to have to resync, and that’s going to take days.

*sigh*

Well, we’ll see whether that was a one time error or…  not…