“Spamtraps”

When looking at the log on my mail server, it usually looks like this:

Just and endless stream of spammers sending spam to addresses that don’t exist. I’ve often wondered why people who run MTAs don’t just band together and work out was to just ban these obviously horrible spammy monsters from the interweb totally!

Surely that has to be the right thing to do!

Cut to yesterday when I sent out 2x 15K unsubscription messages for gmane.org and 15K subscription messages for gmane-mx.org, many of these to mailing lists that don’t exist any more.

Guess what happened:

Note amusing extortion.

And even more amusingly: Both of the services I ended up on (that I checked) claims that I sent messages to spamtrap addresses:

Since I’ve just sent messages to present and former mailing list admin addresses, I think the only way to interpret this is that people have implemented what I’ve been dreaming about all these years: Ban those nasty people that send messages to addresses that don’t exist.

Isn’t that ironic? Don’t you think?

Anyway, only a handful of mailing list providers seem to rely on these bans totally, and the only significant one seems to be OpenBSD. So I guess these won’t be there when news.gmane.io moves to the new address.

On the other hand, they can probably be resubscribed when the move is final, as the new IP won’t be in these “spamtrap” RBLs.

Whatever Happened To news.gmane.org?

I fucked up.

Short version: If you’re reading mailing lists with an NNTP news reader via news.gmane.org, you should update your news reader to point to news.gmane.io instead.

Over the past few years, people have asked me what happened to Gmane, and I’ve mostly clasped my hands over my ears and gone “la la la can’t hear you”, because there’s nothing about the story I’m now finally going to tell that I don’t find highly embarrassing. I had hoped I could just continue that way until I die, but perhaps it would be more constructive to actually tell people what’s going on instead of doing an ostrich impression.

So here’s the long, boring and stupid story.

But first, just some background, because there’s no reason you should know what I’m blathering on about: In 2002 I started a mailing list archive called Gmane. It was fun for many years: I got to write a whole bunch of software for the web site, and it was a useful and satisfying hobby (with some quirks, like not daring to visit India). After a decade or so I started getting burned out, and that’s when my problems started.

At that point I should have looked for somebody to take over Gmane in an orderly fashion, but there were (at least) two reasons I didn’t: I felt a childish attachment to the entire project (I think the feeling can best expressed summed up as “NO! MINE!”), and while I could easily see that somebody would want to take over the web part, the NNTP part (which was the one I used personally) seemed too obscure for anybody to be interested in.

So things limped along, me not having any fun at all at this point, and whenever anybody approached me with feelers towards taking over, I would mostly not reply, because when formulating a reason to say no, I didn’t really have a good reason.

And then the DDoS happened, taking out both Gmane and my kind employer, and I lost my shit. I can’t properly express how little I want to be typing these paragraphs, because I’m so embarrassed by my inadequate handling of the situation. But at least here was an opportunity to hand over Gmane to somebody who could continue running it responsibly.

I got a whole bunch of nice offers from people, among them from somebody working at C, a company with a good reputation and huge reach.

I thought it over, and I said yes to the offer from Y. Because I was thinking “those C. people are nice…” Somehow it had gotten into my mind that Y was C. I have no explanation or excuse: I looked at two totally different company names, and I thought in my head that I was saying yes to one while sending email to the other.

My proposal was that I would still own the gmane.org domain, but Y said they had to own it, so I just signed it over, sent them an SSD with the spool, and continued running the NNTP news server (now hosted in a slightly different place).

By that time I had realised that I’m in touch not with the company I was intending to give Gmane to, but at that point I thought it would be churlish to yank Gmane away from them, saying “oops! wrong company!”

Yes, everything here is embarrassing.

It took Y half a year to get the web site up again, and finally all those dead links stopped screaming. However, the site went down frequently (I didn’t get much communication from Y, but apparently there were more DDoS attacks), and it’s been down completely for at least the last year, I think?

Which brings us to last year, when the company I’d been a co-founder of back in 1997 and worked for since, entered into a sales process. (The other co-founder and majority owner got cancer and suddenly died some time earlier.) The new owner would probably not let me host the news.gmane.org server in the server room, so I got in touch with my contact M at Y to arrange a DNS change for the news.gmane.org server.

And I heard nothing back.

So I Cc’d everybody I’d ever known at Y, including the corporate counsel S, and then I got a response from M, CC’d to S, that started with:

I already responded to this on 22nd January.

So apparently M’s email can’t reach me unless he’s Cc’in S? I guess they must have extremely high security email at Y or something, because I can’t think of any other reason for M’s emails not reaching me. (I grepped the exim logs for the previous year: No contact from Y’s MTAs.)

But I thought, well, now I have contact at least, so I can get the DNS updated! Then S responded:

You have contact with M, please follow through with him directly.

I don’t need copying on any of this.

And then… crickets: I didn’t get any further communication with Y, and no DNS update for the NNTP server.

So at this point, I definitely should have done… something… but instead the sale of my employer was progressing, and then it was done, and then I took a long holiday, and then…

I’m an expert at procrastinating, especially when whatever I have to do involves some degree of confrontation.

Time passes, until… two weeks ago, when I got a message that I really have to have my server removed within a couple of weeks, so I sent off new emails to the Y people, even Cc-ing the guy I wasn’t supposed to Cc, and I heard nothing back, at all. So I have to assume the gmane.org domain can’t be updated any more and I have to do something.

I have set up a new server, and I have mirrored all the data to it, but the problem is the domain, of course. The name of the server is one thing: Update your .gnus files to point to news.gmane.io now; the old name will stop working in a couple of weeks (I got an extension, apparently).

The bigger problem is all the mailing lists: When the server goes away, 15K mailing lists will start to bounce.

So the question is: What should I do about that? There’s several options:

1) Let them bounce: It’s not a major disaster, but I can imagine list admins at places like vger being annoyed (Gmane subscribes to hundreds of vger lists).

2) Unsubscribe them all and let news.gmane.org die a final death: The unsubscription has to happen before the server goes away, because the unsubscription emails have to come from the IP address mentioned in the SPF record for gmane.org.

3) Unsubscribe and resubscribe under a different domain: I have registered a new domain gmane-mx.org and done some experimentation. Of twenty lists un/resubscribed (I’ve scripted this bit so it’s not a lot of work to do a few hundred lists in a batch), two were successfully resubscribed. It seems the reason for this is that sourceforge lists can no longer be subscribed without creating a sourceforge account? And many of the other lists just bounced, despite being active, so it means that they’ve moved without the gmane.conf being updated, but the Gmane address is still subscribed. Fixing this will take manual intervention. But do I want to do this? Asking mailing lists admins to trust me again, with a new domain after the previous debacle, is a bit too much, isn’t it?

4) Somebody at Y could wake up and do the fucking DNS update: Make blaine.gmane.org point to 159.69.161.202 (that’s all you have to do) and we can at least have time to do whatever transition we’re doing in an orderly fashion.

I don’t know what’s going to happen, but list admins: If gmane.org addresses start to bounce, and you want to fix things manually, just replace “gmane.org” with “gmane-mx.org” in all the email addresses. This works already.

OK, this blog post turned out to be even more meandering and unstructured than I had anticipated, but that’s probably because I don’t even know what I want to do here, or what people want to have happen.

  • Should I just let news.gmane.org die? Is it even useful for anybody any more?
  • Should I resubscribe all the lists as gmane-mx.org to continue having a “full” archive, and then try to make somebody more responsible and responsive take over it all again? I mean, there could be a new web interface with similar links as the old one?
  • Is the above point even relevant any more in these days of GDPR? What’s the upside for anybody running such a service?

In many ways, I do want Gmane to go away and not have to think about it any more, because it’s all tied up in feelings of inadequacy. I hope I’m not fishing for sympathy here; this has been really awkward to write, and the only conclusion anybody can draw from reading this is that I’m a fuck-up, and I fucked this up.

But I guess I’m asking for some feedback as to what should happen next. I think I have about a week until the old server goes away now, so whatever happens has to happen fast.

Adding a CSS File to WordPress

Of all things in the world that are frustrating, Googling for how to do things in WordPress is the absolute worst.

I guess I’m not used to, like, search for stuff that’s popular. Because whatever you search for related to WordPress, the top ten answers are from content farms that wants to sell you something, and in addition to being sleazy, their answers are all wrong.

So you have to resort to The Dark Web (i.e., page two of the Google search results) before you get to something that seems to half-way make sense… and then it doesn’t. Not really.

OK, here’s my issue: I wanted to edit the CSS of a WordPress site with Emacs. Because editing the CSS in a text box in a browser is miserable, horrible and no good. If you Google this, you’ll find out that there’s a bunch of plugins that… allows you to edit the CSS in a text box in a browser.

I just want a file somewhere! That I can edit!

So, on the Dark Web, I found somebody with a solution: Just put the extra code in the theme! Or put the code in functions.php! But won’t those changes be overwritten when WordPress is upgraded? Sure!

*sigh*

So after going to The Even Darker Web (page three), and putting four different answers together, I now have a sustainable solution: CSS in a file, loaded in a way that won’t be overwritten when you upgrade WordPress.

Here’s how. Create the directory /var/www/html/wp-content/plugins/site-css. Put the following in the file site-css.php inside that directory:

<?php
/*
Plugin Name: Site CSS
Description: Site specific CSS
*/

function add_custom_css() {
  wp_enqueue_style("site-css", "/wp-content/plugins/site-css/site-css.css");
}

add_action("wp_enqueue_scripts", "add_custom_css");

Go to the WordPress admin panel, and choose “Plugins”. You’ll see a new plugin there called “Site CSS”. Activate it.

That’s it! You can now edit the file called /var/www/html/wp-content/plugins/site-css/site-css.css to your heart’s delight via Tramp in Emacs and tweak those little rounded CSS corners until they look… just… right:

Did you find this blog on page seventeen in Google? You’re welcome!

unsilence

Hidden tracks on CDs used to be a pretty common thing. Not “real” hidden tracks: You could play tricks with the directory structure and put a track before the first one, so you have to skip back from 1 to get to 0.

No, the common way to do this is to pad the final track with, say, ten minutes of silence, and then the “hidden” song starts.

While the concept of hidden tracks is fun, in practice it’s really annoying, because it means that you’re sitting there listening to silence for ten minutes.

So once upon a time I wrote a little C program that would look for silences in the last track of all the CDs I have ripped, and if it finds (long) silences, then splitting happens.

As CDs are getting less popular as a means of distribution (to put it mildly), the hidden track thing has all but disappeared, too, but this week I got
the new Deathcrush album, and it employs this tactic.

I started looking around for the scripts to split the file, and then I discovered that it was in an obscure region of my home directory, not touched for ten years, and not put on Microsoft Github.

But now it is.

It’s probably a couple of decades too late to be useful to anybody, but there you go. It compiles and everything on the current Debian, which just amazes me. I mean, I wrote it in… like… 2003?

Go Linux.

(Not) HDR10 to sRGB

I’m going to be watching a bunch of 4K movies in High Dynamic Range (i.e., UHD HDR) later this year, and I’m going to be screenshotting a bit. Now, as you can see from that blog post, I’m using an HDMI splitter that sends the UHD HDR bits to the TV, and sends 2K SDR bits to the screenshotting box.

There’s a right way to do the SDR conversion and a wrong way, and the HDMI splitter does it the wrong way.

As that web page explains, if you just do the moral equivalent of

ffmpeg.exe -i input.mkv -vf select=gte(n\,360) -vframes 1 output.png

then you’re going to get washed-out images. HDR10 is a 10-bit BT.2020 colourspace, and we want to end up in the 8-bit BT.709 colour space.

And as that page tells us, you can do that in a pretty sensible way, or… you can just discard some bits and end up with a washed-out low-contrast image.

Here’s an example of what comes out of the SDR port:

You can do something simple like:

convert -contrast-stretch 0.20x0.10% IMG_53.JPG norm.jpg

and get something that looks acceptable:

But that’s obviously not the “correct” transform.

As Wikipedia explains, the HDR10 format uses a static non-linear transform in the Rec. 2020 colour space. The formula is here: It’s a “perceptual quantizer” (PQ). “PQ is a non-linear electro-optical transfer function (EOTF).” So there.

So! Here’s my question, that I also asked here:

Given that we have discarded some bits, there’s no way to get the ideal SDR version of these images. But: What has been discarded is predictable (I’m guessing the upper? lower? bits?), and all the HDR10 math stuff is static.

A helpful person on the ImageMagick forum suggests using a sigmoidal contrast stretch with lots of “auto” in the parameters, and that does give me pretty good results in all the test images, so it’s “good enough”.

But since all the things that have been done to the signal is static, there should be a way to write a static transform from these non-HDR10 images to SDR.

So, like, first convert back to 10-bit space (with zeroes for the missing bits?)

And then just do the inverse of the PQ EOTF, and then chop bits again into SDR, as this explains, using one of the nice algos there.

Surely this must be a fun math challenge for somebody (who isn’t me!).

Included below are a few more screenshots of movies and TV series that are HDR10 according to the TV.