Battle Ground, RAID

Hey! I bought a new RAID system. Everything worked perfectly, so I’ll just stop this blog post here.

JUST KIDDING!!! This is one of those blog posts where you scroll forever while I natter on about things that are THE MOST IMPORTANT PROBLEM IN THE WORLD and so on. OK!?

First, some background: I buy movies on physical media — that is, DVDs, 2K blurays and 4K blurays. I rip them before watching them (using MakeMKV), and if I were sensible, I’d just delete the files afterwards, because realistically, am I going to be rewatching movies a lot? Nope.

But do I look sensible? So:

[larsi@rocket-sam ~]$ df -h /dvd/
Filesystem            Size  Used Avail Use% Mounted on
bigger-tex:/data/dvd  102T   79T   23T  78% /dvd

Yeah, I’ve got a big RAID system sitting in a cupboard here somewhere. Let’s see if I can find it…

Yeah, there it is. I’ve used it for about eight years, I guess, and it works well. It uses an LSI MegaRaid card, and it died last year, so I got a new (old) card, and it works again. But! The new LSI card has newer firmware, and that firmware doesn’t allow spinning down the disks. I know! Stupid firmware! And spinning down disks seems essential, since I basically watch movies approx. once a week, so they’re unused 95% of the time. At least.

(Especially since they’re in a fanless enclosure.)

I’ve also got a backup of the RAID system… over four USB3-connected disks. And that started having hickups a bit later.

My interface to this RAID is, of course, Emacs.

Look how nice.

So: I have a system that works, but not in a satisfactory way. I think I can work around the spin-down issue by just having the RAID machine totally go to sleep, and then I can use Wake On LAN, possibly?

But in any case, I need a better backup system than those USB3 disks, and I should have that before I begin experimenting with all this stuff.

So! That’s the background, and now the blog post starts.

It’s not trivial to find a sensible 8 disk enclosure. I want to run Linux on the RAID system, and use mdraid, because all hardware RAID for the consumer market sucks — either it’s slow, or it’s just weird (like the non-spindown LSI card). These enclosures, of course, tend to be very noisy because once you go past 4 disks, the market tends more towards data centres.

But I found this nice-looking thing from SilentPC. Look how nice:

OK, the feature set on the motherboard they’re using is odd, but eh, what the hey. VGA? COM port? Oh, right, this is a Supermicro data centre card.

So in mid May I ordered one (without disks, because I thought I’d buy them locally), because that means less shipping for them, and less bumping around. And how right I was:

It was sent via USPS. SilentPC lives in Battle Ground, Washington. It spent a month in Los Angeles… and was then sent back up there.

Is this part of Trump’s new tariff regime somehow?

They apparently sent it out again (without getting in touch with me or anything; I just got a new tracking message from stamps.com out of the blue)… AND WAS “DELIVERED” IN ANAHEIM!?

Or Compton, possibly? After delivering?

Yes, but after doing some more research via other trackers, it seemed like it changed tracking number after the delivery, so using the new tracking number (not linked anywhere)…

… I find out that it’s available for pickup here. On July 7th, just a month and a half after it was sent out originally.

And after inputting the tracking number in an app here on my phone, I was allowed to pick it up, too! (I got absolutely no notifications from anything about this.)

Logistics companies, eh? They’re not so good at the logistics.

BUT BEHOLD! I”VE GOT IT

I think you’re supposed to do an unboxing series if you’re serious about being an influencer.

Boxes!

Yes!

More unpacking!

THE MACHINE

It’s got fans.

So here’s the bays…

And the all-important backplate.

And I’ve even got VGA cables! Mystery Cable Box to the rescue, once again.

Here’s my home lab setup… very tidy… Most of those components are from the LSI MegaRaid futzing around.

Yes, it’s Supermicro.

Whoosh.

I’m a great fan of these fans. SilentPC says that they use the absolutely most silent components that they can find, and I can attest to that — when these move, I can’t hear them two meters away from the machine.

Oh yeah! Disks. I got some disks locally.

Unboxing!

These are 26TB WD Gold disks. I tried to figure out what the difference between Red, Gold and Purple was, but I gave up. I assured myself that WD Gold doesn’t have shingles, and went with that one on random.

Then install into these plasticky brackets.

I installed Linux on the machine, put the disks in there, and… *crickets*. None of the disks showed up.

I did not plan on actually opening up this machine, but I guess I have to. I see that the SilentPC people at least installed all the cables…

And the cables are indeed connected somewhere on the motherboard. Hm!

Er… lshw does show /dev/sda, but this turns out to be a USB-routed device that’s apparently part of the IPMI system, somehow? That is, it’s a device on the motherboard, and not actually a real disk.

“TEX” is the vendor?

Anyway. Time to read some manuals.

The disks are connected to two ports on the motherboard using a connector called “SlimSAS”:

Now, this connector type is new to me, but I’ve googled until my fingers were sore, and this is my understanding: SlimSAS (not to be confused with MiniSAS) is a physical connector standard. Just like M.2, it can carry different protocols. The two protocols that are relevant here are PCIe and SAS/SATA. Thank you for the unconfusing naming standards here, people. Great job!

Even Wikipedia doesn’t really explain much. SlimSAS is also called SFF-8654, MiniSAS (internal version) is called SFF-8087, and there’s also Compact MiniSAS, external versions, and etc etc etc. And of course there’s two versions of SlimSAS, both called SFF-8654, but one is 4X and one is 8X and has about twice as many pins. Of course.

The Supermicro X13SCL-IF motherboard has two 4X SlimSAS connectors, and they default to being PCIe, not SAS/SATA.

There’s a jumper on the motherboard, and that jumper is indeed set in the PCIe position. I moved it over to the SAS/SATA position.

And… yes! Four disks showed up!

… four disks? Aren’t there eight disks?

All the cables are indeed plugged in, so, hm…

Oh, look at that again. It says “2 SlimSAS”… Check… and then a parentheses that says “2 PCIe 4.0×4” (yes that’s why no disks showed up) “or”, and then… “or 1 PCIe 4.0×4 & 4 SATA”!!! What!? You can never have both SlimSAS ports be SATA? One is always PCIe!!?

This just seems incredible — that SilentPC would sell me an eight disk RAID machine with a motherboard that’s fundamentally unable to drive eight disks. I fired off an email to the SilentPC people, but now (after two weeks), I still haven’t heard back.

Looking more at the motherboard specs, it says elsewhere explicitly that it supports six SATA devices. And there are indeed two physical SATA ports on the motherboard, and along with the 4 on the one SlimSAS port, that’s (if my university maths education is anything to go by) 6 (six). I think.

So now what? My first thought was that perhaps there was some sort of little board you could put into the PCIe SlimSAS connector to get SATA, but nope. And that makes sense: A SATA chipset takes some space, and the SlimSAS connector is just too small, and there’s nowhere to screw something to support a larger card. Fortunately, there’s a free real PCIe slot on the mainboard, so:

More shopping.

I bought a PCIe card with more SATA goodness.

This supports eight disks, so it’s overkill, but it’s the only card like that I could here.

So I put it in there…

Look at the nice cable management… Anyway, I tried plugging in the SlimSAS plug into the card… and DOH!!! The card is MiniSAS, not SlimSAS!!! The feature set is basically identical, but the physical connectors are different.

So, more shopping.

Here’s some MiniSAS cables.

So here’s the difference — MiniSAS to the left, and SlimSAS to the right. See how much sturdier the MiniSAS is.

They’ve really squeezed the size of the SlimSAS down, which makes sense for a motherboard connector.

So, lots of connecting…

And then I tried putting the disk cabinet thing back into the cabinet and… DOH!!! The PCIe card is too big! It just doesn’t fit!

*sigh*

Anyway, I can still experiment with it with the disk part outside the cabinet.

Yes, it finds the controllers.

BEHOLD! ALL THOSE DISKS!

Looking good. So it’s mdadm time:

Looks good! I’m using RAID6, because why not. Meanwhile, it’s time for more shopping.

I couldn’t find any smaller MiniSAS cards via the usual suppliers, so I tried Aliexpress. This probably means that it’ll be uploading all my movies to China, but eh so what.

And… “Mining Accessories”?! I don’t wanna mine anything.

Finding the card isn’t easy — it’s apparently an unusual product category.

There’s a bazillion cards like this — but these aren’t SATA cards. They’re just physical converters from normal PCIe ports to MiniSAS ports — so they’re PCIe only; no SATA.

There’s also M.2 PCIe to MiniSAS, amusingly enough. But these are SATA! Impressive that they’ve managed to squeeze two SATA controllers into that small size…

The motherboard in my box just has one M.2 slot, though, and it’s taken up with the NVMe disk used for the OS.

The ambient temperature in the room is 32C (we’re having a tropical heatwave), so that’s not a bad temperature while building the RAID.

I put the disk cabinet more into the cabinet to get some breeze from the fans.

While looking for smaller cards, I also bought a simple SATA card.

It’s small…

… and has four connectors. But four stiff SATA cables? I’d rather use MiniSAS cables.

OK, the mdraid has finished building, so let’s look at some stats:

I’m getting 460MB/s write speed, and 672MB/s read speed (on sequential writes/reads). Very nice.

While I’m waiting for the AliExpress card, I can continue with the rest of the setup.

The old RAID machine runs headless, and that’s mostly fine. But once in a blue moon, I need to debug something (like when installing that new LSI MegaRaid card), and I have to schlep a monitor from somewhere. Getting the monitor plugged in is usually ten times as much work as whatever the machine is complaining about — usually something along the lines of the BIOS saying “can’t find keyboard; press F1 to continue booting” or something equally thrilling.

So I thought that this time around, I’d just gaffa a 10″ monitor onto the machine, so I don’t ever have to do that.

But remember that the motherboard only has a VGA port? Right, so I got a VGA to HDMI converter thing.

And a 10″ monitor.

It’s USB powered and takes HDMI, of course.

And I plug it in, and nothing.

Er… the machine complains about the USB power? I can’t use a normal USB port to power it?

It came with this power thingie, but with a UK prong only. *sigh* I have to do… something… but what?

Anyway, then the new MiniSAS card arrived.

And angled MiniSAS to SATA cables, because that would give me even more space, I think. (The thick part of these cables is very stiff.)

Look how cute!

Much smaller than the first card.

But… DOH! I BOUGHT THE WRONG ANGLE! Oh well, it seems to fit anyway…

And connect, and…

IT FITS!!! Excellent cable management, too.

Seems to be working…

Yes, indeed.

And I seem to be getting similar speeds with the new card.

So… yay! Everything works!

But I don’t understand why SilentPC sold me a RAID system that wasn’t RAIDing. That’s just odd. I mean, I must be misunderstanding something — as you could probably tell from scrolling through this blog post, all this SlimSAS/MiniSAS stuff was new to me. But I can’t see what.

Is it possible that if SilentPC sells these with disks installed, they also install an extra card (like I’ve done)? If that’s the case, I think the specs should have said something. Or they should have sent me an email. “Hey, guy! I saw that you ordered this without any disks, but are you still going to install some disks and actually use the machine? If so, we have to install a MiniSAS card, so that’s $20 extra.” OR SOMETHING!!!

You’d think?

Or have they never sold one of these before, and didn’t know how SlimSAS works, and that this motherboard had only one SlimSAS that was SATA capable?

The machine only has 1Gbps ethernet, so rsyncing from the old RAID machine is going to take a couple weeks. I think it’s been rsyncing for a week now, and it’s gotten 60TB over.

So in a week or so, I should be able to start experimenting with Wake On LAN and other fun stuff, and then, er, it’s movie watching time!

See? It’s so simple. I can’t imagine why people are using streaming services.

Oh yeah, I forgot the 10″ screen.

I do have other adapters with removable jacks, but none that are exactly the same shape.

I ran out to do some more shopping…

There. I fixed it.

I tried plugging this power adapter into the phone, and it accepted charging — so it seems like it’s at least somewhat standard USB-C, which is a bad sign.

Yup! Still dead!

I thought it was worth a try — I have several gadgets that stubbornly refuse to be charged by anything but the adapter they come with, even if they have a USB-C-shaped charging hole. But I guess this one wasn’t one of those? It’s just dead.

I’ve also tested the screen without the VGA to HDMI adapter, just in case it doesn’t say anything unless it has a valid HDMI stream. The frustrating thing about the screen is that there doesn’t seem to be any LEDs, or any feedback to say whether it’s doing something or not. (Which I guess it has in common with many screens — screen makers love to hide the power buttons, and go into deep sleep while you’re trying to figure stuff out…)

I guess the next step is to buy a new 10″ screen, then.

What about… this one.

*sigh*