There. I Fixed It.

I really like the fanless machines from Aleutia.  They have a nice form factor, they’re really silent, and they have kinda amusing web pages.

I’ve got three of the Sandy Bridge machines now, and they just like work and stuff.  Nine thumbs up.

However!

These are Sandy Brigde machines.  That means built-in Intel graphics.  Built-in Intel graphics have always sucked.  It still sucks.  They will probably always suck.

This doesn’t matter for two of the machines, but the third one is the one I use to watch tv with.  And that’s a problem.

If I use “-vo xv”, then I get tearing, but I can watch really “H” D.  If I use “-vo gl2”, then I get no tearing, but the CPU overheats and gets throttled, which means that I get stuttering video.

This is only a problem when watching 1080p and higher, so I haven’t really bothered doing anything about it.

Until today.

I got a fancy (i.e. the cheapest) fan-less low-profile nVidia card.

Epic unboxing sequence:

The Box!

 

Opened Slightly!

 

Opened Fully!

 

The Manual Removed!

 

And Something Else!

 

The Bag With The Card Is Revealed!

 

The Bag!

 

The Bag Has Sticky Tape On The Back!

 

The Sticky Tape Has Been Vanquished!

 

Peeking Out!

 

Almost Fully Revealed!

 

The Back Of The Card Revealed!

 

The Front Of The Card!

 

Where’s My Pulitzer!

 

Popping The Covers!

 

Pop!  Pop! Pop!

 

Naked!

 

Unscrewed!

 

Lo  Profiled!

Anyway, I got the Aleutia machine down from the shelf.

That’s A Bad Picture!

 

Quite Roomy Inside, Despite Size!

 

See?  There Should Be Room!

 

Er.  It pokes Out At The Top.

 

And The HDMI Port Is Concealed!

 

Bwa Ha HA!  I’ve Got Pliers!

 

Vanquished!

 

There!  I Fixed It!

http://vignette1.wikia.nocookie.net/godzilla/images/8/81/Genius-meme.png/revision/latest?cb=20130511201401

And it seems to actually work.  I can now watch the highest “D” that I’ve got on disk without any problems.

March 9th, 2012

Dear Diary,

I’ve switched the SSD disks back to the LSI MegaRAID SAS 9260-8i controller again, and did some final benchmarks.

I’m comparing reading bits of the Gmane spool in semi-random (i.e. ext4 directory) order, name order (which is also the same order as the files were created), and finally, just reading a big hunking file.  I’ve done this on a spinning disk, with soft RAID, and with the MegaRAID. The last two are over five Samsung 830 512GB in RAID5 mode.

  Semi-random order Name order Sequential reads
Spinning disk 234 files/s 2 MB/s 6062 files/s 48 MB/s 130 MB/s
Soft RAID5 over SSDs 4076 files/s 32 MB/s 8103 files/s 64 MB/s 264 MB/s
LSI MegaRAID SAS 9260-8i over SSDs 4564 files/s 36 MB/s 18708 files/s 148 MB/s 580 MB/s 

So, for the thing that matters — semi-random reads — we get a 20x improvement going from spinning magnetic disks to SSD.  However, the difference between using soft RAID over SATA2.0 versus using that spiffy MegaRAID SATA3.0 card isn’t very thrilling in my realistic use case.  

However, being able to read the 19K files per second when I’m doing, say, re-indexing or statistical analysis, will certainly be nice.  I’ll probably not get that speed when the machine is under load and serving out messages at the same time, but it’ll be a lot better than a contested spinning disk.

So to conclude:  The LSI MegaRAID 9260-8i was a waste of time and money.  I should just have soft RAID-ed the thing, as was my original plan.  But since I’ve got the card now, I might as well just use it.

I’ve now started the real syncing of the entire spool to the new machine.  That’ll probably take a few days, but I’m aiming for replacing the Gmane news server with the new machine sometime next week.  This will hopefully mean the end of all those “load over 16” login denials.

And, Diary, since it seems that some people are able to hack into you and read my most secret thoughts, I’m going to go back to writing you with a quill pen on parchment again.  Surely Google won’t be able to access you there.

March 7th, 2012

Brace yourselves.Winter is coming back

Dear Diary,

I’ve now found some time to do further speed testing of the SSD machine that’s going to replace the Gmane news server.  I know that you remember everything, Diary, but let me give you a brief recap.

I got a bunch of 512GB Samsung 830 SATA3 SSDs that are supposed to have IOPSes coming out of their ears. I got a RAID card, hooked it up, and was somewhat underwhelmed by the performance.  I got a read performance of 4400 files per second when reading in kinda-random order, despite the RAID card having SATA3 and a special “key” that was supposed to disable the built-in slowdowns. That’s ok, but not extremely impressive.

So today I hooked the SSDs up the the motherboard directly.  Unfortunately the motherboard only has SATA2, so the theoretical single-disk bandwidth is about 270MB/s.  Pitiful.

And, wouldn’t you know it, large-file performance towards a single disk was 250MB/s when writing, and 265MB/s when reading. For small files, I get 4000 files per seconds, and 32MB/s.

So pretty much as expected.

Then I created a soft RAID5 over the disks.  Large-file performance is 520MB/s, which is less than half of what I got on the SATA3 RAID card.  The small file performance is 4200 files per second, which is just vaguely south of what that supposedly spiffy RAID card gave me.

This is all rather disappointing.

Diary, I’ll try once more with the RAID card, mixing up the RAID parameters, but I have little hope that much will change.  I’ll have to settle for 4K/s performance when serving out messages from news.gmane.org, I think.

January 20th, 2012

The magical key!

Dear Diary,

today the IOPS-unlocking FastPath key arrived for the LSI MegaRAID 9265-8i card arrivied!  I was so excited, until I opened the box an found what’s basically something that short-circuits two headers on the RAID card.

What a rip-off.  It probably disables some sleep(1) calls in the firmware on the card.

Anyway, I plugged it in, and did the benchmarks, and I got…  absolutely no (zero, zip) difference.  No difference whatsoever.  Zilch.  Nada.

So I googled around a bit, and found someone who claimed that the RAID card should be configured as write-through, no read-ahead, direct I/O to make the FastPath stuff be all awesome and stuff.

I did that.  Annoyingly enough, there seems to be no way to just flip those settings without rebuilding the RAID, so testing takes a while.

And benchmarking afterwards shows that…  throughput drops to one fifth of the standard settings, and random reads of files is absolutely no faster.

I have no more time for benchmarking, since I have to go to Rome.  Diary, I’ll continue fiddling with the RAID card when I get back.

But the strangest thing has happened.  I was looking through some of the previous pages in my secret diary, and I found that somebody else has been making comments!  How is that possible!?  Are other people able to read my diary?  Is Google betraying my secrets to the world?

Anyway, the comment said that I should perhaps just use the SATA 2.0 connectors on the motherboard to see what IOPS I get without using the MegaRAID card.  That makes tons of sense, because I don’t know whether the lack-lustre random read performance I’m getting is because of the disks or the RAID card.

I’ll do that when I get back in about four weeks. 

And I’ll see whether I can secure my diary further somehow.

January 15th, 2012

Dear Diary,

after the rather disappointing random read benchmarks (I mean, 4K files a second? It’s pitiful) I almost went into a severe depression and started thinking about using viAgain.

But then I went back to the hardware pusher’s web site and noticed something strange.  They’re selling something that’s supposed to make their LSI MegaRAID 9265 card suck less!  It’s apparently a firmware upgrade that removes all the sleep(1) calls in their code!  Or something!  At least that’s what I got from reading what they’re saying.  It’s gonna make the IOPS-es on SSD be three times better!

On the one hand, one could be annoyed that they didn’t just leave out the apparent sleep() calls in the standard firmware.  On the other hand, perhaps this is the answer to everything!

Dear Diary, I forked over more money, and I’m now waiting all aflutter for the magical firmware stick to arrive.

Meanwhile, here’s a picture of Oslo.  Winter has sort of arrived, at last:

January 14th, 2012

Dear Diary,

today I’ve written a small benchmark utility to try to emulate NNTP server performance.  A one-file-per-article spool has somewhat unusual performance characteristics, totally dominated by stat-ing and stuff.

So my little utility is a C program that recursively reads a real news spool, and then just discards the result.  It’s extremely single-threaded, which isn’t typical of NNTP usage patterns, but otherwise it should be kinda ok.  It’s on GitHub.

To test, I copied over a 26GB portion of the read Gmane news spool (3.3M files) over to three different partitions: One btrfs on the MegaRAID, one ext4 on the MegaRAID, and one ext4/btrfs on the spinning system disk, just to get a baseline.

(And always do echo 3 > /proc/sys/vm/drop_caches before testing anything.)

btrfs wastes a lot of room, though.  What takes 32GB on ext4 takes 42GB on btrfs.  But with max_inline=0 that shinks to 36GB.  Still kinda sucky.

Anyway, the results are, when reading files in readdir() order:

btrfs on ssd: 10600 files per second, 84MB/s

ext4 on ssd: 4460 files per second, 35MB/s

btrfs on spinning disk: 5030 files per second, 40MB/s

ext4 on spinning disk: 238 files per second (yes, I know.  With noatime.  Yes.  Yes.  Try it yourself.)
And when sorting the files in alphabetical order:

btrfs on ssd: 7800 files per second, 62MB/s

ext2 on ssd: 19200 files per second, 152MB/s

ext4 on ssd: 19100 files per second, 152MB/s

ext4 on spinning disk: 6100 files per second, 48MB/s

So two things stand out here:

1) ext4 is really sensitive to the order you read files
3) the LSI MegaRAID SAS 9265-8I is quite slow on small files

I mean, when reading large files, I get 1.2GB/s!  This is bullshit!  Where are my IOPSes!  I want more IOPS!

Perhaps I should set the stripe size on the RAID to something smaller than the default, which is 128KB.  I mean, the mean file size in the spool is 8K, which means that it’s probably reading a lot more than it has to.

It has to!

January 4th, 2012

Dear Diary,

today the RAID card arrived for my new server. I had apparently only ordered a single SATA cable instead of the six I had meant to order.

But it turned out that the card didn’t use normal SATA cables at all, but a weird one-to-four connector.  It’s a big connector on one side, and four separate SATA cables comes out of it.

So for once messing up the order saved me some money.

I installed the card in my 2U machine.  As you can see, those weird connectors are pretty high up on the card, which means that the cables take a rather dramatic 90 degree turn.  I hope that’s not going to cause any problems…  At least the lid closes.

I connected it all up and booted the machine.

04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 01)

[    8.636577] megaraid_sas 0000:04:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    8.637236] megaraid_sas 0000:04:00.0: setting latency timer to 64
[    8.640920] megasas: FW now in Ready state
[    8.641300] megaraid_sas 0000:04:00.0: irq 46 for MSI/MSI-X
[    8.663036] megasas:IOC Init cmd success

It works!

After fiddling aorund in the glorious and very oddly named “WebBIOS” (it’s not actually on the web at all), I managed to set up my test SSD in RAID0.  That is, a single Corsair Force 3 disk.  Apparently it doesn’t like doing JBOD…  At least, I couldn’t find any setting for it.

Writing to the device gives me 480MB/s, while reading gives me 375MB/s.  Not all that impressive, but at least is demonstrates that the old backplane really really doesn’t lead to any major problems, since that’s clearly SATA 3 speed.

Now I  just have to wait for the Samsung 830s to arrive.  Surely they can’t be serious about that horribly late arrival date… I’ll be in Rome by then…  I won’t know whether this all works until months and months from now…

Woe is me, Diary.

(Continue reading my secret diary.)