Random Thoughts

The magical key!

Dear Diary,

today the IOPS-unlocking FastPath key arrived for the LSI MegaRAID 9265-8i card arrivied!  I was so excited, until I opened the box an found what’s basically something that short-circuits two headers on the RAID card.

What a rip-off.  It probably disables some sleep(1) calls in the firmware on the card.

Anyway, I plugged it in, and did the benchmarks, and I got…  absolutely no (zero, zip) difference.  No difference whatsoever.  Zilch.  Nada.

So I googled around a bit, and found someone who claimed that the RAID card should be configured as write-through, no read-ahead, direct I/O to make the FastPath stuff be all awesome and stuff.

I did that.  Annoyingly enough, there seems to be no way to just flip those settings without rebuilding the RAID, so testing takes a while.

And benchmarking afterwards shows that…  throughput drops to one fifth of the standard settings, and random reads of files is absolutely no faster.

I have no more time for benchmarking, since I have to go to Rome.  Diary, I’ll continue fiddling with the RAID card when I get back.

But the strangest thing has happened.  I was looking through some of the previous pages in my secret diary, and I found that somebody else has been making comments!  How is that possible!?  Are other people able to read my diary?  Is Google betraying my secrets to the world?

Anyway, the comment said that I should perhaps just use the SATA 2.0 connectors on the motherboard to see what IOPS I get without using the MegaRAID card.  That makes tons of sense, because I don’t know whether the lack-lustre random read performance I’m getting is because of the disks or the RAID card.

I’ll do that when I get back in about four weeks. 

And I’ll see whether I can secure my diary further somehow.

Dear Diary,

after the rather disappointing random read benchmarks (I mean, 4K files a second? It’s pitiful) I almost went into a severe depression and started thinking about using vi.  Again.

But then I went back to the hardware pusher’s web site and noticed something strange.  They’re selling something that’s supposed to make their LSI MegaRAID 9265 card suck less!  It’s apparently a firmware upgrade that removes all the sleep(1) calls in their code!  Or something!  At least that’s what I got from reading what they’re saying.  It’s gonna make the IOPS-es on SSD be three times better!

On the one hand, one could be annoyed that they didn’t just leave out the apparent sleep() calls in the standard firmware.  On the other hand, perhaps this is the answer to everything!

Dear Diary, I forked over more money, and I’m now waiting all aflutter for the magical firmware stick to arrive.

Meanwhile, here’s a picture of Oslo.  Winter has sort of arrived, at last:

Dear Diary,

today I’ve written a small benchmark utility to try to emulate NNTP server performance.  A one-file-per-article spool has somewhat unusual performance characteristics, totally dominated by stat-ing and stuff.

So my little utility is a C program that recursively reads a real news spool, and then just discards the result.  It’s extremely single-threaded, which isn’t typical of NNTP usage patterns, but otherwise it should be kinda ok.  It’s on GitHub.

To test, I copied over a 26GB portion of the read Gmane news spool (3.3M files) over to three different partitions: One btrfs on the MegaRAID, one ext4 on the MegaRAID, and one ext4/btrfs on the spinning system disk, just to get a baseline.

(And always do echo 3 > /proc/sys/vm/drop_caches before testing anything.)

btrfs wastes a lot of room, though.  What takes 32GB on ext4 takes 42GB on btrfs.  But with max_inline=0 that shinks to 36GB.  Still kinda sucky.

Anyway, the results are, when reading files in readdir() order:

btrfs on ssd: 10600 files per second, 84MB/s

ext4 on ssd: 4460 files per second, 35MB/s

btrfs on spinning disk: 5030 files per second, 40MB/s

ext4 on spinning disk: 238 files per second (yes, I know.  With noatime.  Yes.  Yes.  Try it yourself.)
And when sorting the files in alphabetical order:

btrfs on ssd: 7800 files per second, 62MB/s

ext2 on ssd: 19200 files per second, 152MB/s

ext4 on ssd: 19100 files per second, 152MB/s

ext4 on spinning disk: 6100 files per second, 48MB/s

So two things stand out here:

1) ext4 is really sensitive to the order you read files
3) the LSI MegaRAID SAS 9265-8I is quite slow on small files

I mean, when reading large files, I get 1.2GB/s!  This is bullshit!  Where are my IOPSes!  I want more IOPS!

Perhaps I should set the stripe size on the RAID to something smaller than the default, which is 128KB.  I mean, the mean file size in the spool is 8K, which means that it’s probably reading a lot more than it has to.

It has to!