Hardware Sucks

Once upon a time, I had a nice Hush machine with a PCI slot.  It had a SATA Multilane card in it, connected to an external box with three SATA CD-ROMs installed.  I used it to rip the CDs I bought so that I could listen to them.

It was perfect.  It was rock solid.  It was fast.  I got reading speeds of 50x — from all three CD-ROMs simultaneously.  So I got an aggregated speed of 150x when ripping a lot of CDs.  Life was wonderful.

Then the Hush machine died.  And I bought a new, spiffy machine.  That only had a PCI Express slot.  But it had USB3 and E-SATA. 

USB3/SATA adapter

So I first tried connecting the CD-ROMs via a mixture of USB3 adapters and E-SATA.

E-SATA backplane for the box

It kinda worked.  Ripping speeds were not impressive, but OK.  But it was unstable.  It would totally crash the machine like every twenty minutes.

So I tried going with a pure USB3 solution.  Good reading speeds, but equally unstable.  My guess would be that extracting audio from CDs via USB3 hasn’t received a lot of testing in Linux. Which is understandable, but it still sucks.

USB3/SATA PM backplane

Next, I noticed that the Addonics web site listed a SATA Port Multipler/USB3 backplane.  It seemed to say that it would allow just using a single USB3 port, and access all the CD players.  Perhaps that would be stable!

Unfortunately, I didn’t read the fine print.  Accessing the drives individually was only possible when using the SATA Port Multiplier interface, and not the USB3 one.

SATA Port Multiplier card

So I bought a SATA PM-capable card, since the Intel SATA chipset doesn’t do PM.

It almost fit into the machine when I removed the bracket.  And it worked!  And was totally stable.

Unfortunately, it’s dog slow.  When ripping a single CD, I get only 20x.  When ripping CDs in parallel, the speeds vary a lot, but ends up evening out at 8x. 

That’s pitiful.

Which brings us up to date.  Today I got a PCI/PCI Express adapter, which  would allow me to try using the old SATA Multilane card.

PCI Express/PCI Adapter
With the Multilane card installed

Multilane plugs are big

 Now, installing all this into the rather small machine took some… persuasion.

 Bendy CPU coolers FTW.

I installed the Multilane backplane back into the CD-ROM box.

 And….  success?

No.

It’s stable.  It doesn’t crash.  But it’s slow!  Slow!  Ripping a single CD gives me 26x.  Ripping three ones in parallel gives me 10x per CD.  So it seems like it tops out at a bandwidth of 30x aggregated.  That’s pitiful.  My old machine gave me 150x.  Is that progress?

It just seems kinda inconceivable that the machine itself is the bottleneck here.  30x CD is (* 44100 2 2 30) => 5292000 bytes per second.  That is 5MB/s.  That is nothing.  The machine has an Intel SSD.  PCI speed is 133MB/s.  The SSD should do 250MB/s. 

But it doesn’t.

I almost feel like giving up and try to refurbish an old Hush machine, but I’m not sure I have the stamina.

Hardware sucks.

There. I Fixed It.

I really like the fanless machines from Aleutia.  They have a nice form factor, they’re really silent, and they have kinda amusing web pages.

I’ve got three of the Sandy Bridge machines now, and they just like work and stuff.  Nine thumbs up.

However!

These are Sandy Brigde machines.  That means built-in Intel graphics.  Built-in Intel graphics have always sucked.  It still sucks.  They will probably always suck.

This doesn’t matter for two of the machines, but the third one is the one I use to watch tv with.  And that’s a problem.

If I use “-vo xv”, then I get tearing, but I can watch really “H” D.  If I use “-vo gl2”, then I get no tearing, but the CPU overheats and gets throttled, which means that I get stuttering video.

This is only a problem when watching 1080p and higher, so I haven’t really bothered doing anything about it.

Until today.

I got a fancy (i.e. the cheapest) fan-less low-profile nVidia card.

Epic unboxing sequence:

The Box!

 

Opened Slightly!

 

Opened Fully!

 

The Manual Removed!

 

And Something Else!

 

The Bag With The Card Is Revealed!

 

The Bag!

 

The Bag Has Sticky Tape On The Back!

 

The Sticky Tape Has Been Vanquished!

 

Peeking Out!

 

Almost Fully Revealed!

 

The Back Of The Card Revealed!

 

The Front Of The Card!

 

Where’s My Pulitzer!

 

Popping The Covers!

 

Pop!  Pop! Pop!

 

Naked!

 

Unscrewed!

 

Lo  Profiled!

Anyway, I got the Aleutia machine down from the shelf.

That’s A Bad Picture!

 

Quite Roomy Inside, Despite Size!

 

See?  There Should Be Room!

 

Er.  It pokes Out At The Top.

 

And The HDMI Port Is Concealed!

 

Bwa Ha HA!  I’ve Got Pliers!

 

Vanquished!

 

There!  I Fixed It!

http://vignette1.wikia.nocookie.net/godzilla/images/8/81/Genius-meme.png/revision/latest?cb=20130511201401

And it seems to actually work.  I can now watch the highest “D” that I’ve got on disk without any problems.

March 9th, 2012

Dear Diary,

I’ve switched the SSD disks back to the LSI MegaRAID SAS 9260-8i controller again, and did some final benchmarks.

I’m comparing reading bits of the Gmane spool in semi-random (i.e. ext4 directory) order, name order (which is also the same order as the files were created), and finally, just reading a big hunking file.  I’ve done this on a spinning disk, with soft RAID, and with the MegaRAID. The last two are over five Samsung 830 512GB in RAID5 mode.

  Semi-random order Name order Sequential reads
Spinning disk 234 files/s 2 MB/s 6062 files/s 48 MB/s 130 MB/s
Soft RAID5 over SSDs 4076 files/s 32 MB/s 8103 files/s 64 MB/s 264 MB/s
LSI MegaRAID SAS 9260-8i over SSDs 4564 files/s 36 MB/s 18708 files/s 148 MB/s 580 MB/s 

So, for the thing that matters — semi-random reads — we get a 20x improvement going from spinning magnetic disks to SSD.  However, the difference between using soft RAID over SATA2.0 versus using that spiffy MegaRAID SATA3.0 card isn’t very thrilling in my realistic use case.  

However, being able to read the 19K files per second when I’m doing, say, re-indexing or statistical analysis, will certainly be nice.  I’ll probably not get that speed when the machine is under load and serving out messages at the same time, but it’ll be a lot better than a contested spinning disk.

So to conclude:  The LSI MegaRAID 9260-8i was a waste of time and money.  I should just have soft RAID-ed the thing, as was my original plan.  But since I’ve got the card now, I might as well just use it.

I’ve now started the real syncing of the entire spool to the new machine.  That’ll probably take a few days, but I’m aiming for replacing the Gmane news server with the new machine sometime next week.  This will hopefully mean the end of all those “load over 16” login denials.

And, Diary, since it seems that some people are able to hack into you and read my most secret thoughts, I’m going to go back to writing you with a quill pen on parchment again.  Surely Google won’t be able to access you there.

March 7th, 2012

Brace yourselves.Winter is coming back

Dear Diary,

I’ve now found some time to do further speed testing of the SSD machine that’s going to replace the Gmane news server.  I know that you remember everything, Diary, but let me give you a brief recap.

I got a bunch of 512GB Samsung 830 SATA3 SSDs that are supposed to have IOPSes coming out of their ears. I got a RAID card, hooked it up, and was somewhat underwhelmed by the performance.  I got a read performance of 4400 files per second when reading in kinda-random order, despite the RAID card having SATA3 and a special “key” that was supposed to disable the built-in slowdowns. That’s ok, but not extremely impressive.

So today I hooked the SSDs up the the motherboard directly.  Unfortunately the motherboard only has SATA2, so the theoretical single-disk bandwidth is about 270MB/s.  Pitiful.

And, wouldn’t you know it, large-file performance towards a single disk was 250MB/s when writing, and 265MB/s when reading. For small files, I get 4000 files per seconds, and 32MB/s.

So pretty much as expected.

Then I created a soft RAID5 over the disks.  Large-file performance is 520MB/s, which is less than half of what I got on the SATA3 RAID card.  The small file performance is 4200 files per second, which is just vaguely south of what that supposedly spiffy RAID card gave me.

This is all rather disappointing.

Diary, I’ll try once more with the RAID card, mixing up the RAID parameters, but I have little hope that much will change.  I’ll have to settle for 4K/s performance when serving out messages from news.gmane.org, I think.

January 20th, 2012

The magical key!

Dear Diary,

today the IOPS-unlocking FastPath key arrived for the LSI MegaRAID 9265-8i card arrivied!  I was so excited, until I opened the box an found what’s basically something that short-circuits two headers on the RAID card.

What a rip-off.  It probably disables some sleep(1) calls in the firmware on the card.

Anyway, I plugged it in, and did the benchmarks, and I got…  absolutely no (zero, zip) difference.  No difference whatsoever.  Zilch.  Nada.

So I googled around a bit, and found someone who claimed that the RAID card should be configured as write-through, no read-ahead, direct I/O to make the FastPath stuff be all awesome and stuff.

I did that.  Annoyingly enough, there seems to be no way to just flip those settings without rebuilding the RAID, so testing takes a while.

And benchmarking afterwards shows that…  throughput drops to one fifth of the standard settings, and random reads of files is absolutely no faster.

I have no more time for benchmarking, since I have to go to Rome.  Diary, I’ll continue fiddling with the RAID card when I get back.

But the strangest thing has happened.  I was looking through some of the previous pages in my secret diary, and I found that somebody else has been making comments!  How is that possible!?  Are other people able to read my diary?  Is Google betraying my secrets to the world?

Anyway, the comment said that I should perhaps just use the SATA 2.0 connectors on the motherboard to see what IOPS I get without using the MegaRAID card.  That makes tons of sense, because I don’t know whether the lack-lustre random read performance I’m getting is because of the disks or the RAID card.

I’ll do that when I get back in about four weeks. 

And I’ll see whether I can secure my diary further somehow.

January 15th, 2012

Dear Diary,

after the rather disappointing random read benchmarks (I mean, 4K files a second? It’s pitiful) I almost went into a severe depression and started thinking about using viAgain.

But then I went back to the hardware pusher’s web site and noticed something strange.  They’re selling something that’s supposed to make their LSI MegaRAID 9265 card suck less!  It’s apparently a firmware upgrade that removes all the sleep(1) calls in their code!  Or something!  At least that’s what I got from reading what they’re saying.  It’s gonna make the IOPS-es on SSD be three times better!

On the one hand, one could be annoyed that they didn’t just leave out the apparent sleep() calls in the standard firmware.  On the other hand, perhaps this is the answer to everything!

Dear Diary, I forked over more money, and I’m now waiting all aflutter for the magical firmware stick to arrive.

Meanwhile, here’s a picture of Oslo.  Winter has sort of arrived, at last:

January 14th, 2012

Dear Diary,

today I’ve written a small benchmark utility to try to emulate NNTP server performance.  A one-file-per-article spool has somewhat unusual performance characteristics, totally dominated by stat-ing and stuff.

So my little utility is a C program that recursively reads a real news spool, and then just discards the result.  It’s extremely single-threaded, which isn’t typical of NNTP usage patterns, but otherwise it should be kinda ok.  It’s on GitHub.

To test, I copied over a 26GB portion of the read Gmane news spool (3.3M files) over to three different partitions: One btrfs on the MegaRAID, one ext4 on the MegaRAID, and one ext4/btrfs on the spinning system disk, just to get a baseline.

(And always do echo 3 > /proc/sys/vm/drop_caches before testing anything.)

btrfs wastes a lot of room, though.  What takes 32GB on ext4 takes 42GB on btrfs.  But with max_inline=0 that shinks to 36GB.  Still kinda sucky.

Anyway, the results are, when reading files in readdir() order:

btrfs on ssd: 10600 files per second, 84MB/s

ext4 on ssd: 4460 files per second, 35MB/s

btrfs on spinning disk: 5030 files per second, 40MB/s

ext4 on spinning disk: 238 files per second (yes, I know.  With noatime.  Yes.  Yes.  Try it yourself.)
And when sorting the files in alphabetical order:

btrfs on ssd: 7800 files per second, 62MB/s

ext2 on ssd: 19200 files per second, 152MB/s

ext4 on ssd: 19100 files per second, 152MB/s

ext4 on spinning disk: 6100 files per second, 48MB/s

So two things stand out here:

1) ext4 is really sensitive to the order you read files
3) the LSI MegaRAID SAS 9265-8I is quite slow on small files

I mean, when reading large files, I get 1.2GB/s!  This is bullshit!  Where are my IOPSes!  I want more IOPS!

Perhaps I should set the stripe size on the RAID to something smaller than the default, which is 128KB.  I mean, the mean file size in the spool is 8K, which means that it’s probably reading a lot more than it has to.

It has to!