As previously discussed, the gmane.org domain was no longer viable, and the NNTP server has now moved to news.gmane.io.
Likewise, mailing list subscriptions have been moved from m.gmane.org to m.gmane-mx.org.
As of this writing, neither service is up, because I’m doing the final resync before restarting the services on a new server. I expect the services to be back up again about 21:00 GMT (January 15th 2020), so don’t panic before that time.
DNS changes may also take some time to propagate.
[Edit at 15:30 GMT: I had misremembered how long the rsync took, so we’re now live six hours ahead of schedule. This Shouldn’t Possibly Happen. I mean, a computer project not being late. Anyway, both Gmane and Gwene feeds are now processing, but the news-to-mail bits aren’t up yet.]
OK, service announcement done, so I thought I’d write a bit about what happened the last week:
First of all, thank you for all the nice comments and PVT EMAILs of support. I wasn’t quite sure whether to continue running the NNTP server, but getting some feedback helps.
So then I started moving 15K mailing lists from a subscription on gmane.org to the new domain, and that was… er… interesting? The process works like this:
The Gmane configuration is a file that has one of these entries per list:
gmane.test gmane-test@quimby.gnus.org Testing the Gmane hierarchy mailman gt-gmane-test validated=2020-01-13 transfer=done crosspost-posting=no
I’ve got an Emacs mode to do a the maintenance work (subscribing, unsubscribing and the like), so I utilised that to write a function that would send out two unsubscription messages (because lists may be subscribed as @gmane.org or @m.gmane.org for historical reasons and there’s no record of which one) and one subscription message for the new @m.gmane-mx address. This bit is fully automated, so I could just sit there watching Emacs send out messages at a somewhat speedy clip. (Well, sending out all the messages took 12 hours in total due to how it’s done: It’s actually doing RPC via NNTP so that the messages are sent directly from the MTA instead of from me, because that looks less spammy.)
(While looking at Emacs doing this bit, I watched Witcher, which was surprisingly entertaining… in parts, and really, really tedious for the rest of the time. And, since it was Netflix, it looked cheap and shoddy.)
I did this in 1K batches, because when I’ve triggered this bit, all messages for the group in question go to a special gmane.admin group, so that I can see all the error messages and stuff, but most importantly: The “reply to confirm subscription” messages, which I then have to respond to (from Gnus). That’s semi-scriptable, but when the “please reply” message is in Chinese, I have to kinda guess.
Then after that, the “Welcome to foo” messages start pouring in, and again, handling the ones in English is fine, but then there’s all the other languages. I know Willkommen, and I can guess at bem-vinda and bienvenu, but Japanese is not my forte, so more guessing is involved.
So this took two days, and for the second day I watched both seasons of Fleabag (the first one is really fun and original and weird, and in the second one they removed everything that was interesting about it and made it into a normal boring dramedy, which explains why it’s on all the “best of” lists of 2019, and even won the Golden Globe. Well played!).
Fun bits from this process: If you’re sending out mail to email addresses that may not longer exist, you’ll end up being branded a spammer, but that wasn’t really much of a surprise.
What did surprise me was that Sourceforge has made it impossible to sign up to mailing lists via email, so we either had to abandon the 2K Sourceforge lists (I know! So many! I had no idea) or do something… semimanually. So I wrote a little bit to open Firefox on each list URL, which put the unique gmane-mx.org into the X selection, so doing each list was “Super-s Right-Mouse TAB TAB SPACE TAB SPACE RET Super-TAB”, and I could do one in five seconds without moving my hands from the keyboard (it’s got build in mouse buttons).
But then it turns out that if you do that a lot, Sourceforge will sic the old-fashioned “click on the three palms” captcha on you. Which took the throughput way, way down.
So for the first half of the Sourceforge lists, I utilised the power of crowdsourcing, and sent off 100 lists each to people who volunteered to do this mindless and boring bit. (Some came back for seconds!) Thank you all again for volunteering.
For the second half, I discovered the wonderful world of captcha solvers, and after signing up with a free wit.ai API key, it worked pretty reliably. So I did most of the second day myself, since it just added a new “TAB RET” bit to each list. (I watched the Alan Bennett at the BBC box set for this part. It’s quite extraordinary. I particularly loved the one with Mrs Bucket in the hospital… and the one with the photographer in the churchyard… and the one with the guy who retires, is unhappy about the retirement, and then gets a stroke and dies.)
Uhm… Any other observations? Ah, right; the IETF MTAs rate-limited me to about one email per ten minutes, so it took four days for all the un/resubscription messages to get through, and some probably just timed out, so I should do another sweep of those bits.
OK… that’s it? Now I just have to wait for the rsync and the DNS changes, which is why I’m going on for this long.
Let’s hear it for TV and wine; two invaluable companions when doing boring semi-manual labour.