January 14th, 2012

Dear Diary,

today I’ve written a small benchmark utility to try to emulate NNTP server performance.  A one-file-per-article spool has somewhat unusual performance characteristics, totally dominated by stat-ing and stuff.

So my little utility is a C program that recursively reads a real news spool, and then just discards the result.  It’s extremely single-threaded, which isn’t typical of NNTP usage patterns, but otherwise it should be kinda ok.  It’s on GitHub.

To test, I copied over a 26GB portion of the read Gmane news spool (3.3M files) over to three different partitions: One btrfs on the MegaRAID, one ext4 on the MegaRAID, and one ext4/btrfs on the spinning system disk, just to get a baseline.

(And always do echo 3 > /proc/sys/vm/drop_caches before testing anything.)

btrfs wastes a lot of room, though.  What takes 32GB on ext4 takes 42GB on btrfs.  But with max_inline=0 that shinks to 36GB.  Still kinda sucky.

Anyway, the results are, when reading files in readdir() order:

btrfs on ssd: 10600 files per second, 84MB/s

ext4 on ssd: 4460 files per second, 35MB/s

btrfs on spinning disk: 5030 files per second, 40MB/s

ext4 on spinning disk: 238 files per second (yes, I know.  With noatime.  Yes.  Yes.  Try it yourself.)
And when sorting the files in alphabetical order:

btrfs on ssd: 7800 files per second, 62MB/s

ext2 on ssd: 19200 files per second, 152MB/s

ext4 on ssd: 19100 files per second, 152MB/s

ext4 on spinning disk: 6100 files per second, 48MB/s

So two things stand out here:

1) ext4 is really sensitive to the order you read files
3) the LSI MegaRAID SAS 9265-8I is quite slow on small files

I mean, when reading large files, I get 1.2GB/s!  This is bullshit!  Where are my IOPSes!  I want more IOPS!

Perhaps I should set the stripe size on the RAID to something smaller than the default, which is 128KB.  I mean, the mean file size in the spool is 8K, which means that it’s probably reading a lot more than it has to.

It has to!

January 13th, 2012

Dear diary,

today is the most joyful day of my entire life.  The Samsung 830s for my new server arrived!  Sort of out of the blue!  The web shop insisted upon them not arriving until the end of the month, and then they sent them anyway.

Look at how pretty they are.  One might even suspect Samsung having learnt something from Apple by manufacturing most of Apple’s stuff.

The disks are so pretty that it’s almost a shame to put them into the server, don’t you think, Diary?

 Anyway, 5x 512GB of SSD-ey goodness.  I installed them in some 2.5″->3.5″ adapters that we had laying around.  They were originally used for some WD VelociRaptor disks that had all died over the years.  Not very good disks, but nice adapters.  And that’s what counts.

So the SSDs are plugged into adapters, which are then plugged into the backplace, which is then plugged into the RAID card.

I configured up the LSI MegaRAID SAS 9265-8I card in RAID5 mode, and then put ext4 on the partition.

Diary, you won’t believe this.  With a simple “cat /dev/zero > /mnt/ssd/file” command, iostat told me that I had a writing speed of 1.2GB/s.  Yes!  Jiggobytes!  Not bits!  Bytes!  Jiggobytes!

 I only got 600MB/s when reading the same huge file back.

So to get slightly more serious about benchmarking, I installed iozone, and made it do its test on five 100GB files in parallell.  It reports 1.5GB/s on large writes, and 1.2GB/s on large reads!

So I wasn’t just imagining things!  This thing is blazingly fast on pointlessly huge files! I mean, the theoretical max should be about 2GB/s, since each of the five SSDs have a writing speed of about halv a gig each, and it’s a five-disk RAID5 system, but I hadn’t actually expected the LSI RAID card to deliver anything like this, because, you know, Diary, all hardware sucks.

Now I just have to write something to emulate read NNTP server load (i.e., lots of small files in huge directories), and then do some benchmarking on differents file systems to see what happens.

Oh, the excitement!

January 4th, 2012

Dear Diary,

today the RAID card arrived for my new server. I had apparently only ordered a single SATA cable instead of the six I had meant to order.

But it turned out that the card didn’t use normal SATA cables at all, but a weird one-to-four connector.  It’s a big connector on one side, and four separate SATA cables comes out of it.

So for once messing up the order saved me some money.

I installed the card in my 2U machine.  As you can see, those weird connectors are pretty high up on the card, which means that the cables take a rather dramatic 90 degree turn.  I hope that’s not going to cause any problems…  At least the lid closes.

I connected it all up and booted the machine.

04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 01)

[    8.636577] megaraid_sas 0000:04:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    8.637236] megaraid_sas 0000:04:00.0: setting latency timer to 64
[    8.640920] megasas: FW now in Ready state
[    8.641300] megaraid_sas 0000:04:00.0: irq 46 for MSI/MSI-X
[    8.663036] megasas:IOC Init cmd success

It works!

After fiddling aorund in the glorious and very oddly named “WebBIOS” (it’s not actually on the web at all), I managed to set up my test SSD in RAID0.  That is, a single Corsair Force 3 disk.  Apparently it doesn’t like doing JBOD…  At least, I couldn’t find any setting for it.

Writing to the device gives me 480MB/s, while reading gives me 375MB/s.  Not all that impressive, but at least is demonstrates that the old backplane really really doesn’t lead to any major problems, since that’s clearly SATA 3 speed.

Now I  just have to wait for the Samsung 830s to arrive.  Surely they can’t be serious about that horribly late arrival date… I’ll be in Rome by then…  I won’t know whether this all works until months and months from now…

Woe is me, Diary.

(Continue reading my secret diary.)

December 23rd, 2011

Dear Diary,

today I was rummaging through the drawers in the office, and I happened upon an SSD there.  It was a Corsair Force 3, which is a SATA 3.0 disk!  Yay!  I can finally do some benchmarking, I thought.

I put the SSD into the server and switched it on, but it came up as a 3Gbps device.  And there were only four ports visible, even though there’s six headers on the motherboard.  Just look:

ahci 0000:00:11.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0x3f impl SATA mode

After fiddling around in the BIOS a bit (I had to switch off IDE emulation and “combined mode”, whatever that is), all six ports appeared in Linux:

ahci 0000:00:11.0: AHCI 0001.0100 32 slots 6 ports 3 Gbps 0x3f impl SATA mode

By all that’s holy!  By Emacs!  I misread the specifications on the motherboard! It says “6x SATA2 3.0 Gbps Ports”, not “6x SATA 3.0 ports”!


Diary, it turns out that Supermicro has absolutely no SATA3 motherboards.  


Instead of turning to drink, despair and vi, as some might do, I turned to the browser and ordered a low-profice PCI Express SATA 3 card instead.  The reviews says that it’s supposed to play well with SSDs, too, and has realish hardware RAID.  I know that I earlier vowed never to use anything but soft RAID again after many dissappointing experiences with cheap-ish hardware RAID setups, but surely this time there won’t be any problems.


Fingers crossed!


(Continue reading my secret diary.)

December 22nd, 2011

Dear Diary,

the Samsungs haven’t arrived yet, but they’re supposed to arrive any day now. I’m starting to wonder whether they actually exist out there, or if they are a phantom product that’s announced, but not actually shipped.

But I just looked on Newegg, for instance, and they claim to have them in stock, so they must exist.  They must!

It’s odd that they say “limit 1 per customer”, though.  And that they are more expensive in the US than in Norway.  That never happens.  I’m pretty sure when I looked earlier that they were more expensive here than there.  As they should be.

Is Samsung rationing the SSDs?

Aren’t they thinking about what the effect is on me?  They are so self-centered.

(Continue reading my secret diary.)

December 19th, 2011

Dear diary,

Sinking Heat

today I got the heat sink and the RAM for the new server. I installed them and hit the power switch.

It’s alive!

I haven’t built a server in ever so long, so I’m happy and kinda surprised that it works.  The Samsung 830s I ordered haven’t arrived yet, but they’re supposed to ship in a few days.  The OS is going to be on a spinning disk, anyway, so I installed a normal Western Digital Enterprisey and untarred Debian onto it.

It still works!

SATA Galore

However, the disk performance isn’t all that great.  Since the disk is quite modern, and the motherbord is quite modern, I had expected sequential writes of 120MB/s, but I only get 105MB/s.  Boo.

To bad I don’t have any SATA3.0 disks to test with.  The spinning disk is just a 3Gbps SATA disk.  At least it means that les backplanes sont des connecteurs simples sans électronique ni intelligence.  For SATA2, at least.

Oh well.  The Samsungs will arrive any day now and I can start testing!

(Continue reading my secret diary.)