October 5th

Dear Diary,

today the LSI MegaRAID SAS 9240-8i card finally completed building the RAID5 set over five 1TB Samsung SSDs.  It only took about 18 hours.

So time to do some benchmarking!  I created an ext4 file system on the volume and wrote /dev/zero to it.

DSC00890

Err…  40 MB/s?  40MB/s!??!  These are SATA3 6Gbps disks that should have native write speeds of over 400MB/s.  And writing RAID5 to them should be even faster.  40 MB/s is impossibly pathetic.  And the reading speed was about the same.

If I hadn’t seen it myself, I wouldn’t have believed it.

Dear Diary, I finally did what I should have done in the first place: I binged the card.  And I found oodles of people complaining about how slow it is.

This is apparently LSI’s bottom-rung RAID card.  One person speculates that it does the RAID5 XOR calculations on the host side instead of having it implemented on the RAID card.  That doesn’t really account for how incredibly slow it is, though.

I think LSI just put a lot of sleep() calls into the firmware so that they could have a “lower-end” card that wouldn’t compete with the cards they charge a lot more money for.

I went back to the office and reconfigured the SSDs as JBOD, and then I created an ext4 file system on one of them, and then wrote /dev/zero to it, just to see what the native write and read rates are:

DSC00891

DSC00892

Around 400 MB/s.  It’s not astoundingly good, but it’s not completely pitiful, either.  These disks should do over 500 MB/s, but…

Then I created a soft RAID5 over the disks.  How long would it take to build it?

DSC00893

80 minutes.  That’s better than 16 hours, but it seems a bit slow…

Turns out there’s a SPEED_LIMIT_MAX that needs to be tweaked.

DSC00894

With that in place, I get 300 MB/s while building the RAID.  51 minutes.

DSC00895

DSC00896

And it sustains until it’s done, which it has while I was typing this.

Now to check the real performance…

Making the file system was really fast.  It peaked at 1GB/s.  Writing a huge sequential file to the ext4 file system gives me around 180 MB/s.  Which isn’t fantastic, but it’s acceptable.  Reading the same sequential file gives me 1.4 GB/s!  That’s pretty impressive.

It’s incredible that the LSI MegaRAID SAS 9240-8i is 6x-25x slower than Linux soft RAID.  Even if the card offloads the XOR-ing to the host CPU, it still doesn’t explain how its algorithms are that much slower than md’s algorithms.

Anyway: Avoid this card like the plague.  It’s unusable as a RAID card if you need reasonable performance.  40 MB/s is not reasonable.

Signature Elided

I stopped buying stuff from Amazon like a year ago (I know, I’m slow on the uptake about their eeeevil), so I’m now buying all my DVDs and BluRays via play.com, which is a shopping portal. All the orders apparently go to smaller shops that send out the stuff.

Which has to be better.  I mean, Amazon don’t get any money.

But it’s occasionally a bit odd, like this Godard DVD I got from Taiwan (!) with the following card included:

DSC00882

 

October 4th

Dear Diary,

today was the day I was going to install a new SSD RAID system for the Gmane news spool.  The old spool kinda ran full four months ago, but I kept deleting (and storing off-spool) the largest groups and waddled through.

I had one server that seemed like a good fit for the new news server: It had 8 physical disk slots.  But the motherboard only had six SATA connectors, so I bought an LSI MegaRAID SAS 9240-8i  card.

DSC00850
Installing the 2.5″ SSDs in a 3.5″ adapter.  Five of the disks
DSC00851
So screwed. I mean, so many screws
DSC00853
I decided to add 2x 4TB spinning mechanical disks to the remaining two slots for, uhm, a search index perhaps?
DSC00854
Oops. I forgot to film the unboxing
DSC00849
Ribbed for your coldness
DSC00852
A tower of SSDs
DSC00857
All seven disks installed in their Supermicro caddies

DSC00859
Look at that gigantic hole, yearning to be filled.
DSC00860
Uhm… these captions took an odd turn back there…
DSC00861
I pull the server out of the rack and totally pop its top
DSC00862
Look! Innards!
DSC00863
Err… is that a photo of the RAID card? I think it might be…
DSC00864
All wired up
DSC00865
The disk backplane had six slots already hooked up, so I viciously ripped three of the connectors out
DSC00866
And plugged five of the connectors from the RAID card back in
DSC00867
And then my biggest fans were reinstalled. Thank you thank you

Now the hardware was all installed, but getting to the LSI WebBIOS configuration was impossible.

DSC00871

I hit Ctrl+H repeatedly while booting, but I always just got the Linux boot prompt instead.  I binged around a bit, and it turns out that if any other non-RAID disks are present in the system, the WebBIOS won’t appear at all.

So I popped the three spinning disks out of the machine, and presto:

DSC00872

I configured the RAID5 over the five 1TB SSDs.  This would give me about 4TB, which is twice what the current news spool has.

However, building the RAID seems to take forever:

DSC00874

WTF?  2% in 20 minutes?

These are SATA3 SSDs.  Read/write speed is over 500MB/s.  That means that reading or writing a single disk should take under 30 minutes.  Since the card can access all the disks independently, computing the RAID5 XOR shouldn’t take more than that.

But let’s be generous.  Let’s say it has to read the disks sequentially, and write the parity disk sequentially.  That’s 2.5 hours.

Instead it’s going to take 16 hours.  WTF is up with that?  Does the LSI MegaRAID SAS 9240-8i have the slowest CPU ever or something?

That’s just unbelievably slow.

Diary, I’m going to let it continue building the RAID and then do some speed tests.  If it turns out that it’s this slow during operation, I’m going to rip it out and just do software RAID.