B&CB: Aïoli Monstre w/ Aïoli

New year, new decade, new food.

The previous salt cod dish from the Bistro Cooking book was delicious, so I’m all excited about this one: Lots of veggies, cod and aioli. The only thing that’s odd about this recipe is its name: Grand aioli. I mean, that’s just the sauce.

So this is a two-for-one dish, because I’m also making this:

And it’s the first time I’ve ever made aioli from scratch, so I wonder how that’s doing to go.

So it’s the normal setup: It’s salt cod, so it has to be desalinated for two days:

So, into the fridge… While that’s happening, I can read a book? Yes.

Assassin’s Fate by Robin Hobb.

Now she’s just fucking with us.

Two previous trilogies were named:

Assassin’s Apprentice / Royal Assassin / Assassin’s Quest

Fool’s Errand / The Golden Fool / Fool’s Fate

See the pattern? Now this:

Fool’s Assassin / Fool’s Quest / Assassin’s Fate

Hobb! Nobody can remember which books is which! Doing a combinatorial mix of words in titles is bad!

Anyway!

This is the sixteenth book in this series, and as you can see, it’s rather hefty. But I think this book is bigger than most of them? If we say that the books are, say, 600 pages on average, that’s (* 600 16) => 9600 pages in total. So my guess is that this is the longest series of books I’ve read.

I mean, counting pages. Reading-wise, it’s a breeze. Robin Hobb has a way of writing that makes these books hard to put down: Sometimes the plots are so exciting that it all becomes too much and I just plotz. She’s also really great at world-building; her different societies do not feel like Feudal Japan With Identifying Numbers Filed Off or whatever.

Let’s read the first three pages:

Excitement!

And this is the culmination of this series, apparently. I mean… OK, no spoilers? No spoilers. But it’s the end, so it has a lot to live up to, and it does. While some writers are rather thin on plot development and draw things out (I remember a mid-trilogy 400 page book by Cherryh that could be summed up “The ship took off.”, and was summed up that way in the next book.) But there’s a lot going on here, and everything that happens has repercussions, and there’s no deus ex machina that swoops in and makes all the tribulations of the characters irrelevant, and… It’s solid.

That said, Hobb has some annoying tics. OK, I hate all the torture scenes, but those can mostly be skipped over, but more annoying is that she has two plot gadgets she relies on way too much: Characters withholding information from each other (I seem to remember an earlier book that … just wouldn’t have happened if they’d talked on the first page; this one isn’t as bad), and the other is having Fitz (the titular Assassin) being stoopid.

Fitz always thinks he knows best, and goes off harebrained to sacrifice himself and then the other people go “hm, surely it’s not as dire as that” and then they have to go after him, unlocking doors he’s locked, and kinda sorta save him. Again, there’s not as much of that in this book as in previous books, but it’s a really sad way of solving plotting problems.

Oh, and another really annoying thing, come to think of it: In these books, nobody believes anything anybody says. “Could you pass me the salt?” “Do you really want salt?” These characters can mind-read and dream prophetic dreams and everything, but whenever somebody says something like “Fitz is alive; he just mindcalled me” everybody says “You’re just feverish. Go to bed.” It’s really, really annoying, and it wastes so many pages. One thing I really love about Star Trek: The Next Generation is that they had unconditional faith in nobody trying to bullshit anybody else, and that just makes things move faster. Somebody: “Captain, I think… something’s vaguely kinda perhaps wrong?” Picard: “Everybody assemble, drop out of warp, run a full diagnostic; could we be in an alternate timeline? Did somebody take over our bodies? Find out what’s wrong!”

ANYWAY.

And the next-to-final scene, with page upon page being spent on Fitz and how sad everything is, boo hoo (well it is sad), and then half a page spent on the Fool, as an afterthought. It’s eyeroll inducing.

But it does work…

The cod’s been de-salted enough now, so it’s time to start on the aioli.

So it’s garlic, salt and egg yolks…

Whisked a lot and then add oil and EEEEK! It separated! It’s horrible and gruesome!

*pout*

Well, while I think about what to do, I’ll do the rest of the dish.

It’s a lot of chopping.

And then everything into a pan to steam. The recipe said to cook everything separately, because everything has different timing (from 40 minutes for the beets, to 20 for the potatoes/carrots, to 7 for the cauliflower, to 4 for the beans). But I don’t have that many steamer thingies, so I just put everything into one casserole at different times. Lots of math!

Finally almost everything in…

It’s a very simple recipe.

And done! The recipe specifies one hard-boiled egg (in its shell), too, so I did that separately. I guess the shell is to make it more rustic? It’s got that rustic vibe.

I redid the aioli while the veggies were cooking, and added the oil at a much slower pace, and then everything went swimmingly. It turned out a bit thick, though. Perhaps I should have added some water?

Anyway, this was really good. I love salt cod, and it goes great with the aioli. What a good idea. And the beets are also a great component here. I should have added 2x the cauliflower, though.

Some of the recipes in the Bistro Cooking book have been kinda dodgy (or I’m not able to execute them), but this was perfect.

This blog post is part of the Bistro
Cooking & Books series.

news.gmane.org is now news.gmane.io

As previously discussed, the gmane.org domain was no longer viable, and the NNTP server has now moved to news.gmane.io.

Likewise, mailing list subscriptions have been moved from m.gmane.org to m.gmane-mx.org.

As of this writing, neither service is up, because I’m doing the final resync before restarting the services on a new server. I expect the services to be back up again about 21:00 GMT (January 15th 2020), so don’t panic before that time.

DNS changes may also take some time to propagate.

[Edit at 15:30 GMT: I had misremembered how long the rsync took, so we’re now live six hours ahead of schedule. This Shouldn’t Possibly Happen. I mean, a computer project not being late. Anyway, both Gmane and Gwene feeds are now processing, but the news-to-mail bits aren’t up yet.]

OK, service announcement done, so I thought I’d write a bit about what happened the last week:

First of all, thank you for all the nice comments and PVT EMAILs of support. I wasn’t quite sure whether to continue running the NNTP server, but getting some feedback helps.

So then I started moving 15K mailing lists from a subscription on gmane.org to the new domain, and that was… er… interesting? The process works like this:

The Gmane configuration is a file that has one of these entries per list:

gmane.test   gmane-test@quimby.gnus.org
  Testing the Gmane hierarchy
  mailman gt-gmane-test
  validated=2020-01-13
  transfer=done
  crosspost-posting=no

I’ve got an Emacs mode to do a the maintenance work (subscribing, unsubscribing and the like), so I utilised that to write a function that would send out two unsubscription messages (because lists may be subscribed as @gmane.org or @m.gmane.org for historical reasons and there’s no record of which one) and one subscription message for the new @m.gmane-mx address. This bit is fully automated, so I could just sit there watching Emacs send out messages at a somewhat speedy clip. (Well, sending out all the messages took 12 hours in total due to how it’s done: It’s actually doing RPC via NNTP so that the messages are sent directly from the MTA instead of from me, because that looks less spammy.)

(While looking at Emacs doing this bit, I watched Witcher, which was surprisingly entertaining… in parts, and really, really tedious for the rest of the time. And, since it was Netflix, it looked cheap and shoddy.)

I did this in 1K batches, because when I’ve triggered this bit, all messages for the group in question go to a special gmane.admin group, so that I can see all the error messages and stuff, but most importantly: The “reply to confirm subscription” messages, which I then have to respond to (from Gnus). That’s semi-scriptable, but when the “please reply” message is in Chinese, I have to kinda guess.

Then after that, the “Welcome to foo” messages start pouring in, and again, handling the ones in English is fine, but then there’s all the other languages. I know Willkommen, and I can guess at bem-vinda and bienvenu, but Japanese is not my forte, so more guessing is involved.

So this took two days, and for the second day I watched both seasons of Fleabag (the first one is really fun and original and weird, and in the second one they removed everything that was interesting about it and made it into a normal boring dramedy, which explains why it’s on all the “best of” lists of 2019, and even won the Golden Globe. Well played!).

Fun bits from this process: If you’re sending out mail to email addresses that may not longer exist, you’ll end up being branded a spammer, but that wasn’t really much of a surprise.

What did surprise me was that Sourceforge has made it impossible to sign up to mailing lists via email, so we either had to abandon the 2K Sourceforge lists (I know! So many! I had no idea) or do something… semimanually. So I wrote a little bit to open Firefox on each list URL, which put the unique gmane-mx.org into the X selection, so doing each list was “Super-s Right-Mouse TAB TAB SPACE TAB SPACE RET Super-TAB”, and I could do one in five seconds without moving my hands from the keyboard (it’s got build in mouse buttons).

But then it turns out that if you do that a lot, Sourceforge will sic the old-fashioned “click on the three palms” captcha on you. Which took the throughput way, way down.

So for the first half of the Sourceforge lists, I utilised the power of crowdsourcing, and sent off 100 lists each to people who volunteered to do this mindless and boring bit. (Some came back for seconds!) Thank you all again for volunteering.

For the second half, I discovered the wonderful world of captcha solvers, and after signing up with a free wit.ai API key, it worked pretty reliably. So I did most of the second day myself, since it just added a new “TAB RET” bit to each list. (I watched the Alan Bennett at the BBC box set for this part. It’s quite extraordinary. I particularly loved the one with Mrs Bucket in the hospital… and the one with the photographer in the churchyard… and the one with the guy who retires, is unhappy about the retirement, and then gets a stroke and dies.)

Uhm… Any other observations? Ah, right; the IETF MTAs rate-limited me to about one email per ten minutes, so it took four days for all the un/resubscription messages to get through, and some probably just timed out, so I should do another sweep of those bits.

OK… that’s it? Now I just have to wait for the rsync and the DNS changes, which is why I’m going on for this long.

Let’s hear it for TV and wine; two invaluable companions when doing boring semi-manual labour.

SMTP Callouts

After doing several rounds of mop-ups after moving all the mailing lists from gmane.org to gmane-mx.org, I was wondering whether there were any way of checking whether a mailing list is dead for sure or not (mostly to see what percentage had been ported), and… of course there isn’t.

But you can use SMTP callouts to do a somewhat inaccurate check. If you don’t know what that it, it’s this:

You just say MAIL FROM and RCPT TO to the mail server, and it’ll say whether that looks okidoke or not, without delivering any mail to the victim I mean recipient.

The problems are, of course, manifold, and you really shouldn’t do this unless you have at least some handle on the possible repercussions, because SMTP is a war zone. Spammers try to send you spam, and everybody else tries to make that as difficult as possible.

But I wrote a little Emacs Lisp library for it, and put it on Microsoft Github.

Basic usage notes:

Some MTAs will reject the email if the host you’re running this on isn’t listed in the SPF records for the “MAIL FROM” address. In that case, you can add the host you’re running this on to the SPF record for that address:

v=spf1 mx a […] ip4:172.217.20.46 -all

Some MTAs will respond with a “451” code, which means “send the mail again later”, and this is usually because greylisting is in effect. It may or may not have checked whether the recipient exists first. Some MTAs respond with a “451” even if they never accept mail for that address. This library counts “451” as a failure, but you may not want to.

Some MTAs will say they accepts mail to all recipients, but then do the “real” check after data has been sent (and the MTA checks whether the contents look like spam or the like).

The actions this function takes may remind some MTAs of actions spammers take, and you may end up in RBLs as a suspected spammer. This is very much the case if the address you’re checking doesn’t actually exist, as people feed random delivery attempts to the spam-fighting RBLs, even if the addresses in question have, at one point in the past, been valid.

So I’m running the gmane.conf checker on a host that’s not my MTA at all, because I assume that it’ll be “burned” once I’m done checking.

Fun!

“Spamtraps”

When looking at the log on my mail server, it usually looks like this:

Just and endless stream of spammers sending spam to addresses that don’t exist. I’ve often wondered why people who run MTAs don’t just band together and work out was to just ban these obviously horrible spammy monsters from the interweb totally!

Surely that has to be the right thing to do!

Cut to yesterday when I sent out 2x 15K unsubscription messages for gmane.org and 15K subscription messages for gmane-mx.org, many of these to mailing lists that don’t exist any more.

Guess what happened:

Note amusing extortion.

And even more amusingly: Both of the services I ended up on (that I checked) claims that I sent messages to spamtrap addresses:

Since I’ve just sent messages to present and former mailing list admin addresses, I think the only way to interpret this is that people have implemented what I’ve been dreaming about all these years: Ban those nasty people that send messages to addresses that don’t exist.

Isn’t that ironic? Don’t you think?

Anyway, only a handful of mailing list providers seem to rely on these bans totally, and the only significant one seems to be OpenBSD. So I guess these won’t be there when news.gmane.io moves to the new address.

On the other hand, they can probably be resubscribed when the move is final, as the new IP won’t be in these “spamtrap” RBLs.

Whatever Happened To news.gmane.org?

I fucked up.

Short version: If you’re reading mailing lists with an NNTP news reader via news.gmane.org, you should update your news reader to point to news.gmane.io instead.

Over the past few years, people have asked me what happened to Gmane, and I’ve mostly clasped my hands over my ears and gone “la la la can’t hear you”, because there’s nothing about the story I’m now finally going to tell that I don’t find highly embarrassing. I had hoped I could just continue that way until I die, but perhaps it would be more constructive to actually tell people what’s going on instead of doing an ostrich impression.

So here’s the long, boring and stupid story.

But first, just some background, because there’s no reason you should know what I’m blathering on about: In 2002 I started a mailing list archive called Gmane. It was fun for many years: I got to write a whole bunch of software for the web site, and it was a useful and satisfying hobby (with some quirks, like not daring to visit India). After a decade or so I started getting burned out, and that’s when my problems started.

At that point I should have looked for somebody to take over Gmane in an orderly fashion, but there were (at least) two reasons I didn’t: I felt a childish attachment to the entire project (I think the feeling can best expressed summed up as “NO! MINE!”), and while I could easily see that somebody would want to take over the web part, the NNTP part (which was the one I used personally) seemed too obscure for anybody to be interested in.

So things limped along, me not having any fun at all at this point, and whenever anybody approached me with feelers towards taking over, I would mostly not reply, because when formulating a reason to say no, I didn’t really have a good reason.

And then the DDoS happened, taking out both Gmane and my kind employer, and I lost my shit. I can’t properly express how little I want to be typing these paragraphs, because I’m so embarrassed by my inadequate handling of the situation. But at least here was an opportunity to hand over Gmane to somebody who could continue running it responsibly.

I got a whole bunch of nice offers from people, among them from somebody working at C, a company with a good reputation and huge reach.

I thought it over, and I said yes to the offer from Y. Because I was thinking “those C. people are nice…” Somehow it had gotten into my mind that Y was C. I have no explanation or excuse: I looked at two totally different company names, and I thought in my head that I was saying yes to one while sending email to the other.

My proposal was that I would still own the gmane.org domain, but Y said they had to own it, so I just signed it over, sent them an SSD with the spool, and continued running the NNTP news server (now hosted in a slightly different place).

By that time I had realised that I’m in touch not with the company I was intending to give Gmane to, but at that point I thought it would be churlish to yank Gmane away from them, saying “oops! wrong company!”

Yes, everything here is embarrassing.

It took Y half a year to get the web site up again, and finally all those dead links stopped screaming. However, the site went down frequently (I didn’t get much communication from Y, but apparently there were more DDoS attacks), and it’s been down completely for at least the last year, I think?

Which brings us to last year, when the company I’d been a co-founder of back in 1997 and worked for since, entered into a sales process. (The other co-founder and majority owner got cancer and suddenly died some time earlier.) The new owner would probably not let me host the news.gmane.org server in the server room, so I got in touch with my contact M at Y to arrange a DNS change for the news.gmane.org server.

And I heard nothing back.

So I Cc’d everybody I’d ever known at Y, including the corporate counsel S, and then I got a response from M, CC’d to S, that started with:

I already responded to this on 22nd January.

So apparently M’s email can’t reach me unless he’s Cc’in S? I guess they must have extremely high security email at Y or something, because I can’t think of any other reason for M’s emails not reaching me. (I grepped the exim logs for the previous year: No contact from Y’s MTAs.)

But I thought, well, now I have contact at least, so I can get the DNS updated! Then S responded:

You have contact with M, please follow through with him directly.

I don’t need copying on any of this.

And then… crickets: I didn’t get any further communication with Y, and no DNS update for the NNTP server.

So at this point, I definitely should have done… something… but instead the sale of my employer was progressing, and then it was done, and then I took a long holiday, and then…

I’m an expert at procrastinating, especially when whatever I have to do involves some degree of confrontation.

Time passes, until… two weeks ago, when I got a message that I really have to have my server removed within a couple of weeks, so I sent off new emails to the Y people, even Cc-ing the guy I wasn’t supposed to Cc, and I heard nothing back, at all. So I have to assume the gmane.org domain can’t be updated any more and I have to do something.

I have set up a new server, and I have mirrored all the data to it, but the problem is the domain, of course. The name of the server is one thing: Update your .gnus files to point to news.gmane.io now; the old name will stop working in a couple of weeks (I got an extension, apparently).

The bigger problem is all the mailing lists: When the server goes away, 15K mailing lists will start to bounce.

So the question is: What should I do about that? There’s several options:

1) Let them bounce: It’s not a major disaster, but I can imagine list admins at places like vger being annoyed (Gmane subscribes to hundreds of vger lists).

2) Unsubscribe them all and let news.gmane.org die a final death: The unsubscription has to happen before the server goes away, because the unsubscription emails have to come from the IP address mentioned in the SPF record for gmane.org.

3) Unsubscribe and resubscribe under a different domain: I have registered a new domain gmane-mx.org and done some experimentation. Of twenty lists un/resubscribed (I’ve scripted this bit so it’s not a lot of work to do a few hundred lists in a batch), two were successfully resubscribed. It seems the reason for this is that sourceforge lists can no longer be subscribed without creating a sourceforge account? And many of the other lists just bounced, despite being active, so it means that they’ve moved without the gmane.conf being updated, but the Gmane address is still subscribed. Fixing this will take manual intervention. But do I want to do this? Asking mailing lists admins to trust me again, with a new domain after the previous debacle, is a bit too much, isn’t it?

4) Somebody at Y could wake up and do the fucking DNS update: Make blaine.gmane.org point to 159.69.161.202 (that’s all you have to do) and we can at least have time to do whatever transition we’re doing in an orderly fashion.

I don’t know what’s going to happen, but list admins: If gmane.org addresses start to bounce, and you want to fix things manually, just replace “gmane.org” with “gmane-mx.org” in all the email addresses. This works already.

OK, this blog post turned out to be even more meandering and unstructured than I had anticipated, but that’s probably because I don’t even know what I want to do here, or what people want to have happen.

  • Should I just let news.gmane.org die? Is it even useful for anybody any more?
  • Should I resubscribe all the lists as gmane-mx.org to continue having a “full” archive, and then try to make somebody more responsible and responsive take over it all again? I mean, there could be a new web interface with similar links as the old one?
  • Is the above point even relevant any more in these days of GDPR? What’s the upside for anybody running such a service?

In many ways, I do want Gmane to go away and not have to think about it any more, because it’s all tied up in feelings of inadequacy. I hope I’m not fishing for sympathy here; this has been really awkward to write, and the only conclusion anybody can draw from reading this is that I’m a fuck-up, and I fucked this up.

But I guess I’m asking for some feedback as to what should happen next. I think I have about a week until the old server goes away now, so whatever happens has to happen fast.