Storage… To The Max!

For some years now, I’ve been ripping DVDs (and blu-rays) before watching, both for practical reasons and stupid reasons. One theoretical nice side effect of all this is that I can re-watch movies without rummaging through boxes of stuff… And I actually did re-watch something this year, although I should have re-bought it on blu-ray.

OK, so this thing still doesn’t make much sense, but meanwhile:

My current disk system is full! Aaargh! Now what!

Well, the sensible thing would be to start deleting movies that I’m definitely never going to watch again, but I’ve never ever been sensible, so I instead started looking around for A Different Solution.

To recap, for new readers:

I’ve been using these Mediasonic 4 bay docks, because I need a solution without fans, and you basically can’t get anything fanless if you go over two disks. These things work fine… if you happen on the right combination of USB-3 chipsets and disks. I couldn’t get them to work in a RAID5 configuration with my current setup, so what you see above is two RAID0 (!) setups that are mirrors of each other, with four 12TB disks in each set. I get read speeds of 240MB/s from them, so that’s fine.

So… my current sets are full, and if there’s any point in expanding, I have to go at least 2x as big, which means 8 16TB disks in RAID5. So I got some more of these Mediasonic things, and did some testing.

Building a RAID5 over them gave me a rebuild speed of 20MB/s, which means that it’d take me a month to build a new set, so that’s not fun. Not to mention that it’d take as long to rebuild if a disk dies.

So I googled and googled, and I found A-Tech Fabrication, which I’m assuming is a couple of guys in a machine shop, because the web site has that look. But guys in machine shops can be good bets for this sort of stuff, so I went ahead and got one of the four-disk enclosures (planning to get another one if the first one works).

Behold!

Made in the U S of the A.

Look at all these screws! It takes 14 of them to just get the top off!

Inside there’s a bunch of emptiness, because this comes as a set you have to build yourself.

So each disk has to be fastened to these thick slabs of metal, presumably to conduct heat well, or because A-Tech loves thick slabs of metal. So four screws each per disk to fasten those…

… and then the slabs are screwed onto the lower metal plate; four more screws.

So that’s, er, (* (+ 4 4) 4) => 32 screws to get the disks onto the lower plate…

… which then sits (unscrewed!) in the case on these rubber things. It’s supposed to reduce whirring sounds. Makes sense, but it does mean that heat isn’t really conducted to the outside of the cabinet..

So there’s the disks in the cabinet.

The disks talk to the computer via a SAS interface.

So there’s one external SAS connector and four internal SATA ones.

Look how tidy it all looks with the cables.

And then the upper plate, with another four screws per disk, so we’re not up to… er… a lot more screws? And more of those rubber isolator things.

And then the top cover, which is just 14 screws.

So… if one of the disks go AWOL and has to be replaced, you better be prepared to get screwing.

But… I stupidly only bought one of these, because I wanted to check whether it was actually working. I could build a temporary four disk set over them, I guess, but… that’s so unsatisfying…

So I made the brilliant decision to just attach four additional disks directly to the SATA connectors on the motherboard of the little computer I’m using for this.

Look! How pretty!

The complete, beautiful setup.

So… does it work?

It started off with a rebuild speed of 240MB/s (quite respectable), and it’s slowly dropping off, as usual, since the outer bits of the disks have more bandwidth than the inner bits.

The raw read speed via the SAS controller is 260MB/s, which is identical to the speed I get from the direct-attached SATA disks, so that’s nice.

The temperatures of the disks inside the enclosure is about 4-7C higher than the “naked” disks. That’s not very dramatic, and 46C shouldn’t be a problem. And this is during the rebuild, when all the disks are working continuously — during normal usage I’d expect them to be cooler.

But there are some oddities. I’m getting a lot of these in the kern.log file:

Mar  3 05:00:48 big-tex kernel: [49923.811438] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Mar  3 05:00:48 big-tex kernel: [49923.811439] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Mar  3 05:00:48 big-tex kernel: [49923.811441] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Mar  3 05:00:48 big-tex kernel: [49923.811443] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)

And I mean a lot. That’s annoying. And I’ve been unable to google for whatever 0x31080000 is supposed to mean.

There’s also some occasional more serious-looking error messages:

Mar  3 00:40:19 big-tex kernel: [34294.952967] sd 0:0:3:0: [sdd] tag#1479 CDB: R
ead(16) 88 00 00 00 00 03 16 8f 9e 88 00 00 02 e8 00 00
Mar  3 00:40:19 big-tex kernel: [34294.952969] scsi target0:0:3: handle(0x000c),
 sas_address(0x4433221103000000), phy(3)
Mar  3 00:40:19 big-tex kernel: [34294.952972] scsi target0:0:3: enclosure logic
al id(0x500605b002099904), slot(3) 
Mar  3 00:40:19 big-tex kernel: [34294.952975] sd 0:0:3:0: task abort: SUCCESS s
cmd(000000001dbdacdb)
Mar  3 00:40:19 big-tex kernel: [34294.952982] sd 0:0:3:0: attempting task abort
! scmd(000000001b3d968e)

But the rebuild keeps on working, and the SUCCESS sounds nice?

Mar  2 23:03:27 big-tex kernel: [28482.994797] sd 0:0:3:0: [sdd] tag#1737 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Mar  2 23:03:27 big-tex kernel: [28482.994798] sd 0:0:3:0: [sdd] tag#1737 CDB: Read(16) 88 00 00 00 00 02 9b 93 0b 10 00 00 04 00 00 00
Mar  2 23:03:27 big-tex kernel: [28482.994799] print_req_error: I/O error, dev sdd, sector 11200039696
Mar  2 23:03:27 big-tex kernel: [28482.994856] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)

These messages are apparently a “retry” thing, and aren’t serious.

Allegedly.

So… it’s looking OK? So I think I’ll be ordering the second cabinet from the A-Tech people.

The build quality on this A-Tech cabinet seems out of this world. Everything is machined to perfection, and everything is heavy and sturdy. And with a simply ridiculous amount of screws, but I’ve got an electrical screwdriver, so that’s fine.

But meanwhile:

Ooo. That’s a lot of movies.

Since each 4K blu-ray is just 50-100GB large, this should be sufficient for months and months.

Anyway: While copying stuff from the old RAID to the new, the temperature seems to be holding quite steady:

Uhm… Perhaps it’s not a good thing for the disks to be jumping around like that when I’m copying things from them? What do you think?

OK, I’ll order another cabinet from A-Tech.

[six weeks pass]

Look! I got my second cabinet from A-Tech! Same solid build quality. Such attention to detail. The screws all use the same sockets, even, so that you don’t have to change bits while doing them. These cabinets spark joy.

… same number of screws. (There’s twelve screws per disk, and then … 14? more for the top. So that’s (+ (* 12 4) 14) => 62 screws. And most of them have to be unscrewed if I have to swap out a broken disk.)

Looks better on the shelf, eh?

So how speedy is it with all of the disks connected via the SAS card?

Well, 320MB/s sounds OK to me. I mean, it’s not massive or anything, but it’s fine…

Temperatures are OK? Sure. They seem to top out at 44C when I’m doing something, and when they’re sleeping (which is 95% of the time), the temperatures bottoms out at 34C.

Anyway! It is possible! To have a largish RAID array at home without any annoying fans.

The only question is whether the disks will die before or after the RAID has grown too small, I guess.

[edit some weeks later: The RAID bounced hard when I was doing some heavy copying from it, and rebuilding failed. Oops! But I wondered whether the problem was the SAS PCI Express card — I was getting those weird error messages, right? So I bought a “LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator]” card, and installed it. It required wiping the disks completely and rebuilding the RAID array (*sigh*) because the disks were subtly smaller when communicating via that card. I guess it put some of its own RAID stuff onto the disks? Anyway, with that card, there were no warnings from the OS and the rebuild (and re-sync of all the files) went off without a hitch.]

[edit a month later than that: The system has been totally stable after replacing the SAS card, so I think these A-Tech enclosures are pretty good.]