SMTP Callouts

After doing several rounds of mop-ups after moving all the mailing lists from gmane.org to gmane-mx.org, I was wondering whether there were any way of checking whether a mailing list is dead for sure or not (mostly to see what percentage had been ported), and… of course there isn’t.

But you can use SMTP callouts to do a somewhat inaccurate check. If you don’t know what that it, it’s this:

You just say MAIL FROM and RCPT TO to the mail server, and it’ll say whether that looks okidoke or not, without delivering any mail to the victim I mean recipient.

The problems are, of course, manifold, and you really shouldn’t do this unless you have at least some handle on the possible repercussions, because SMTP is a war zone. Spammers try to send you spam, and everybody else tries to make that as difficult as possible.

But I wrote a little Emacs Lisp library for it, and put it on Microsoft Github.

Basic usage notes:

Some MTAs will reject the email if the host you’re running this on isn’t listed in the SPF records for the “MAIL FROM” address. In that case, you can add the host you’re running this on to the SPF record for that address:

v=spf1 mx a […] ip4:172.217.20.46 -all

Some MTAs will respond with a “451” code, which means “send the mail again later”, and this is usually because greylisting is in effect. It may or may not have checked whether the recipient exists first. Some MTAs respond with a “451” even if they never accept mail for that address. This library counts “451” as a failure, but you may not want to.

Some MTAs will say they accepts mail to all recipients, but then do the “real” check after data has been sent (and the MTA checks whether the contents look like spam or the like).

The actions this function takes may remind some MTAs of actions spammers take, and you may end up in RBLs as a suspected spammer. This is very much the case if the address you’re checking doesn’t actually exist, as people feed random delivery attempts to the spam-fighting RBLs, even if the addresses in question have, at one point in the past, been valid.

So I’m running the gmane.conf checker on a host that’s not my MTA at all, because I assume that it’ll be “burned” once I’m done checking.

Fun!

Emacs Supremacy

I use Emacs to control the stereo, so 95% of the screen is Emacs:

But I like to have a clock there too, so since I started this thing (in… 1997?) I’ve also had an xwatch in the corner. The other night I decided that this was completely and utterly untenable:

So I thought “it must be easy to make an SVG based clock, surely” and looked forward to some hacking, but unfortunately Ulf Jasper wrote an SVG clock back in 2011, and it’s on ELPA, so I just used that instead. (I tweaked it slightly to get the look I wanted.)

Look! Emacs rules supreme over the screen!

To get the look I wanted, I had to delve into portions of Emacs I’ve never seen before, but to remove the borders/continuation markers between the two buffers at the bottom (one showing the songs and the other the clock), I had to:

(set-face-foreground 'vertical-border "black")
(set-display-table-slot standard-display-table 0 ?\ ))

There you go.

2×10%

Since this summer, I’ve er had some time on my hands, so I thought I’d work at fixing Emacs bugs. (Or closing stuff that’s no longer relevant.) And since working towards some goal is more fun, I decided that 10% of the Emacs bug tracker would be nice.

There was about 4500 open bugs in the tracker, so that was 450 bugs, which I completed in August.

After taking some time off, I thought it was time for another 10%. Which is, of course, 410 bugs, since there’s now a smaller number of open bugs in the tracker.

I have the best math. And this way, each 10% is easier.

And today, tada, Emacs reported

11 today; 410 bugs; target 410; start 2019-08-07T20:40:02

and I’m done! Which is perfect timing, because I’ve got a cold, so writing a bragging blog post is just my speed.

So now there’s about 3500 remaining open bugs in the tracker. “But wait,” you say, “I can’t find my slide rule but according to all maths, 4100 less 410 isn’t 3500, probably”. You are indeed a mathematical genius. But I’m not the only one working at trolling I mean trawling the bug database. In particular, Stefan Kangas has closed hundreds and hundreds of bugs concurrently with me. (And, of course, the rest of the gaggle of Emacs people working at the usual rates of reporting/fixing.)

In addition, since I’m from finance, I can apply the newest financial science innovations: Linear extrapolation.

By using these latest technologies (i.e., a ruler and a sharpie), I can predict:

In 2021, there will be more closed bugs than have been opened! It’s an inversion of the OxC Index! It’s a brave new world! This time it’s different!

And now it’s time for another break, I think. Well, at least after my cold clears up; I might as well continue a bit more while feverish…

Parsing Exif Data

Emacs is moving away from ImageMagick support, and is instead handling all the major image formats (PNG, JPEG, etc) natively. The reason for this is that the ImageMagick libraries have a pretty bad track record: Over the years, a large number of Emacs crashes have turned out to stem from ImageMagick crashing. While things have been getting better, this is still a problem, especially as Emacs is being used as a web browser and a buffer overflow can lead to code execution.

But ImageMagick provides features that the native libraries do not, like image scaling and rotation, and without image scaling, everything’s sad. Have you tried looking at a web page in Emacs with all images being ENORMOUSxGINORMOUS pixels large?

Sad.

Fortunately, Emacs 27, while deprecating ImageMagick, has gained native image transforms at the same time, so we’re all set, I think. The only thing that’s missing is a way for image-mode and the like to suss out what the rotation of images should be, and that data is stored in the Exif portion of JPEG images.

So this weekend I finally got started with writing an Emacs Lisp package to parse Exif data (in JPEG images)… and I finished, too, because the format is way simple. Which shouldn’t come as a surprise, because it’s meant to be implemented by camera manufacturers, and they er well you know.

But it’s an interesting format. It has the smell of being cobbled together from whatever formats people had lying around, and it’s not… er… smart. I don’t think I’m being controversial when I’m saying that.

I’m not an Exif expert: All I know what was I googled while writing exif.el yesterday. If I say anything horribly wrong here, I’m er horribly wrong:

Basically, the Exif format is a TIFF file plonked into the APP1 field of a JPEG. The TIFF format is the weirdness: It’s based on four-byte (32 bit) offsets instead of something sensible like length specifications. And the offsets are always from the start of the TIFF file, so they’re absolute. (Well. Relative to the start. Relatively absolute.)

This means that there’s no way to say “just extend this string”: You have to recompute (or regenerate) the entire TIFF file, because everything that points to something after the string will have their offsets changed. One person writing one of the many web pages that try to explain the format laconically commented that no Exif editors do so correctly, or without corrupting at least some part of the data.

The main part of the TIFF is the IDF: The Image File Directory. It looks like this:

LL
TT|FF|LLLL|VVVV
TT|FF|LLLL|VVVV
TT|FF|LLLL|VVVV
...
NNNN

LL (two bytes; a 16 bit number) say how many entries there are, and then each entry is 12 bytes long. In each entry TT is the tag, FF says what “format” the data is, LLLL says how long the data is (a 32 bit number), and VVVV is the value of the data.

Simple, eh? Hah!

The format is stuff like “ascii”, “short” and “long”. A “short” is 2 bytes and s “long” is 4 bytes. To find the actual length of the data, you have to multiply LLLL with this number of bytes, so if you introduce a new format (with a different byte length), there’s no way for older parsers to know how long the data is! No wonder it’s common for Exif editors to mangle the data.

Besides not making sense, it’s not even any kind of optimisation, because no part in a TIFF file can be longer than what can be described by a 32-bit number, so having LLLL just specify the length directly would have been just as fine.

*sigh*

And then the real fun: If LLLL (times the format length) is shorter than 4, then VVVV is the value. If it’s longer than 4, then obviously it can’t fit into VVVV, so… VVVV (a 32 bit number) holds an offset value that points to a place in the TIFF file where the data really is.

But there’s nice things about the TIFF format. I mean, it has fractions. (They’re represented by eight bytes, the numerator is the four first and the denominator is the last four.) Ideal for parsing with Common Lisp, but unfortunately Emacs Lisp doesn’t have rational numbers.

Oh, I didn’t mention what NNNN is: It’s a pointer to the next directory section, which is er useful if er you have more than 65536 directory entries, I guess.

I think the funniest part of the Exif format is that the numbers embedded in it can be either little-endian and big-endian. Fortunately this is called out explicitly with a bit that says either “II” (Intel) or “MM” (Motorola), so it’s no biggie, but it’s just weird that they couldn’t decide on one or the other.

But isn’t it nice that all this archaeological technology lives inside our modern devices and programs? Just think of that the next time you see an image that’s correctly rotated for sure.

Setting up GPG for Emacs

I know, everybody hates GPG these days (and for good reasons), but I’ve been looking at the Emacs bug database and getting annoyed with all the SMIME etc bugs that aren’t getting fixed, and thought I should do something about it.

I last used GPG in the nineties, and I’ve forgotten everything I possibly could have know about it, so I thought I, as a complete born-again novice, would be in a good position to write a how to set it up, so I started with

rm -r ~/.gnupg

and then

gpg --gen-key

and I then got to

What… century… is… this.

On my laptop, it stayed that way for ten minutes, until it got enough entropy, because /dev/urandom is obviously too insecure.

*sigh*

And then I got:

gpg: agent_genkey failed: No such file or directory
Key generation failed: No such file or directory

Which after some Duckgoing is because I removed the .gnupg directory, and the gpg agent doesn’t understand that, and gpg of course didn’t say anything about this until after collecting entropy for ten minutes.

So did I have a gpg agent running?

Err… apparently? About five hundred?

None of the recommended ways of restarting the agent worked (presumably because the .gnupg directory was removed, which makes sense), so I just

pkill gpg-agent

and went through creating the key again, and waited another ten minutes, and then everything was created, even the revocation certificate, which all the recipes on the Interweb says I have to create by hand?

Who knows.

Anyway, then interweb recipes say I should send the key to a key
server, like:

gpg --keyserver gpg.mit.edu --send-keys 29AEBB3

and then gpg helpfully says:

larsi@marnie:~$ gpg --keyserver gpg.mit.edu --send-keys 29AEBB3DDF5E83147CB9EC61EE84A168D288B04F
gpg: sending key EE84A168D288B04F to hkp://gpg.mit.edu
gpg: keyserver send failed: No name
gpg: keyserver send failed: No name

“No name.”

What could that mean? No name? In the key? Didn’t I enter my name there? No, it turns out that the “gpg.mit.edu” domain name doesn’t resolve. Which the error message could have said, you’d think? No?

It’s “pgp.mit.edu”. After which the command apparently works, because there’s no feedback, which is the universal way to say “yes, that worked”, I guess?

OK, now everything is set up, so I just have to figure out how to use it on the Gnus/Message side, I think?

How about if I test with a

C-c C-m C-c

and I get a

It worked! I sent an encrypted message! To myself! And Gnus decrypted it automatically, and Emacs used the gpg agent so I didn’t have to type the passphrase a billion times.

Hey, now I’m totally qualified to finally have a look at the SMIME bug reports.