Emacs Supremacy

I use Emacs to control the stereo, so 95% of the screen is Emacs:

But I like to have a clock there too, so since I started this thing (in… 1997?) I’ve also had an xwatch in the corner. The other night I decided that this was completely and utterly untenable:

So I thought “it must be easy to make an SVG based clock, surely” and looked forward to some hacking, but unfortunately Ulf Jasper wrote an SVG clock back in 2011, and it’s on ELPA, so I just used that instead. (I tweaked it slightly to get the look I wanted.)

Look! Emacs rules supreme over the screen!

To get the look I wanted, I had to delve into portions of Emacs I’ve never seen before, but to remove the borders/continuation markers between the two buffers at the bottom (one showing the songs and the other the clock), I had to:

(set-face-foreground 'vertical-border "black")
(set-display-table-slot standard-display-table 0 ?\ ))

There you go.

2×10%

Since this summer, I’ve er had some time on my hands, so I thought I’d work at fixing Emacs bugs. (Or closing stuff that’s no longer relevant.) And since working towards some goal is more fun, I decided that 10% of the Emacs bug tracker would be nice.

There was about 4500 open bugs in the tracker, so that was 450 bugs, which I completed in August.

After taking some time off, I thought it was time for another 10%. Which is, of course, 410 bugs, since there’s now a smaller number of open bugs in the tracker.

I have the best math. And this way, each 10% is easier.

And today, tada, Emacs reported

11 today; 410 bugs; target 410; start 2019-08-07T20:40:02

and I’m done! Which is perfect timing, because I’ve got a cold, so writing a bragging blog post is just my speed.

So now there’s about 3500 remaining open bugs in the tracker. “But wait,” you say, “I can’t find my slide rule but according to all maths, 4100 less 410 isn’t 3500, probably”. You are indeed a mathematical genius. But I’m not the only one working at trolling I mean trawling the bug database. In particular, Stefan Kangas has closed hundreds and hundreds of bugs concurrently with me. (And, of course, the rest of the gaggle of Emacs people working at the usual rates of reporting/fixing.)

In addition, since I’m from finance, I can apply the newest financial science innovations: Linear extrapolation.

By using these latest technologies (i.e., a ruler and a sharpie), I can predict:

In 2021, there will be more closed bugs than have been opened! It’s an inversion of the OxC Index! It’s a brave new world! This time it’s different!

And now it’s time for another break, I think. Well, at least after my cold clears up; I might as well continue a bit more while feverish…

Parsing Exif Data

Emacs is moving away from ImageMagick support, and is instead handling all the major image formats (PNG, JPEG, etc) natively. The reason for this is that the ImageMagick libraries have a pretty bad track record: Over the years, a large number of Emacs crashes have turned out to stem from ImageMagick crashing. While things have been getting better, this is still a problem, especially as Emacs is being used as a web browser and a buffer overflow can lead to code execution.

But ImageMagick provides features that the native libraries do not, like image scaling and rotation, and without image scaling, everything’s sad. Have you tried looking at a web page in Emacs with all images being ENORMOUSxGINORMOUS pixels large?

Sad.

Fortunately, Emacs 27, while deprecating ImageMagick, has gained native image transforms at the same time, so we’re all set, I think. The only thing that’s missing is a way for image-mode and the like to suss out what the rotation of images should be, and that data is stored in the Exif portion of JPEG images.

So this weekend I finally got started with writing an Emacs Lisp package to parse Exif data (in JPEG images)… and I finished, too, because the format is way simple. Which shouldn’t come as a surprise, because it’s meant to be implemented by camera manufacturers, and they er well you know.

But it’s an interesting format. It has the smell of being cobbled together from whatever formats people had lying around, and it’s not… er… smart. I don’t think I’m being controversial when I’m saying that.

I’m not an Exif expert: All I know what was I googled while writing exif.el yesterday. If I say anything horribly wrong here, I’m er horribly wrong:

Basically, the Exif format is a TIFF file plonked into the APP1 field of a JPEG. The TIFF format is the weirdness: It’s based on four-byte (32 bit) offsets instead of something sensible like length specifications. And the offsets are always from the start of the TIFF file, so they’re absolute. (Well. Relative to the start. Relatively absolute.)

This means that there’s no way to say “just extend this string”: You have to recompute (or regenerate) the entire TIFF file, because everything that points to something after the string will have their offsets changed. One person writing one of the many web pages that try to explain the format laconically commented that no Exif editors do so correctly, or without corrupting at least some part of the data.

The main part of the TIFF is the IDF: The Image File Directory. It looks like this:

LL
TT|FF|LLLL|VVVV
TT|FF|LLLL|VVVV
TT|FF|LLLL|VVVV
...
NNNN

LL (two bytes; a 16 bit number) say how many entries there are, and then each entry is 12 bytes long. In each entry TT is the tag, FF says what “format” the data is, LLLL says how long the data is (a 32 bit number), and VVVV is the value of the data.

Simple, eh? Hah!

The format is stuff like “ascii”, “short” and “long”. A “short” is 2 bytes and s “long” is 4 bytes. To find the actual length of the data, you have to multiply LLLL with this number of bytes, so if you introduce a new format (with a different byte length), there’s no way for older parsers to know how long the data is! No wonder it’s common for Exif editors to mangle the data.

Besides not making sense, it’s not even any kind of optimisation, because no part in a TIFF file can be longer than what can be described by a 32-bit number, so having LLLL just specify the length directly would have been just as fine.

*sigh*

And then the real fun: If LLLL (times the format length) is shorter than 4, then VVVV is the value. If it’s longer than 4, then obviously it can’t fit into VVVV, so… VVVV (a 32 bit number) holds an offset value that points to a place in the TIFF file where the data really is.

But there’s nice things about the TIFF format. I mean, it has fractions. (They’re represented by eight bytes, the numerator is the four first and the denominator is the last four.) Ideal for parsing with Common Lisp, but unfortunately Emacs Lisp doesn’t have rational numbers.

Oh, I didn’t mention what NNNN is: It’s a pointer to the next directory section, which is er useful if er you have more than 65536 directory entries, I guess.

I think the funniest part of the Exif format is that the numbers embedded in it can be either little-endian and big-endian. Fortunately this is called out explicitly with a bit that says either “II” (Intel) or “MM” (Motorola), so it’s no biggie, but it’s just weird that they couldn’t decide on one or the other.

But isn’t it nice that all this archaeological technology lives inside our modern devices and programs? Just think of that the next time you see an image that’s correctly rotated for sure.

Setting up GPG for Emacs

I know, everybody hates GPG these days (and for good reasons), but I’ve been looking at the Emacs bug database and getting annoyed with all the SMIME etc bugs that aren’t getting fixed, and thought I should do something about it.

I last used GPG in the nineties, and I’ve forgotten everything I possibly could have know about it, so I thought I, as a complete born-again novice, would be in a good position to write a how to set it up, so I started with

rm -r ~/.gnupg

and then

gpg --gen-key

and I then got to

What… century… is… this.

On my laptop, it stayed that way for ten minutes, until it got enough entropy, because /dev/urandom is obviously too insecure.

*sigh*

And then I got:

gpg: agent_genkey failed: No such file or directory
Key generation failed: No such file or directory

Which after some Duckgoing is because I removed the .gnupg directory, and the gpg agent doesn’t understand that, and gpg of course didn’t say anything about this until after collecting entropy for ten minutes.

So did I have a gpg agent running?

Err… apparently? About five hundred?

None of the recommended ways of restarting the agent worked (presumably because the .gnupg directory was removed, which makes sense), so I just

pkill gpg-agent

and went through creating the key again, and waited another ten minutes, and then everything was created, even the revocation certificate, which all the recipes on the Interweb says I have to create by hand?

Who knows.

Anyway, then interweb recipes say I should send the key to a key
server, like:

gpg --keyserver gpg.mit.edu --send-keys 29AEBB3

and then gpg helpfully says:

larsi@marnie:~$ gpg --keyserver gpg.mit.edu --send-keys 29AEBB3DDF5E83147CB9EC61EE84A168D288B04F
gpg: sending key EE84A168D288B04F to hkp://gpg.mit.edu
gpg: keyserver send failed: No name
gpg: keyserver send failed: No name

“No name.”

What could that mean? No name? In the key? Didn’t I enter my name there? No, it turns out that the “gpg.mit.edu” domain name doesn’t resolve. Which the error message could have said, you’d think? No?

It’s “pgp.mit.edu”. After which the command apparently works, because there’s no feedback, which is the universal way to say “yes, that worked”, I guess?

OK, now everything is set up, so I just have to figure out how to use it on the Gnus/Message side, I think?

How about if I test with a

C-c C-m C-c

and I get a

It worked! I sent an encrypted message! To myself! And Gnus decrypted it automatically, and Emacs used the gpg agent so I didn’t have to type the passphrase a billion times.

Hey, now I’m totally qualified to finally have a look at the SMIME bug reports.

Technical Analysis

I thought I could use my prodigious financial know-how to do an analysis of Emacs Open Bugs chart.

First of all, we have clearly defined positive trend channels reaching back to 2009, broken by a period of recession.

But lately, this growth has been curbed and we’ve seen a clear development of a resistance line at 4100. It looked like we’d broken out of it in 2018, but then we have a reversal, but judging on this clear resistance, we’ll soon see a bounce-back.

Zooming even closer into the last couple of years, we see the emergence of a linear Fibonacci reversal trend coupled with a reverse Head & Shoulders® pattern complete with magnificent bubbles appearing and not the least hint of dandruff sloughing off of the chart, complete with a 10% (450 bugs) reduction due to me procrastinating a lot over the past month and a half and going through old bug reports, but now there’s a two week vacation from the vacation coming up.

This can only mean one thing, and I don’t have to spell it out for you, I think: Sell, sell, S-E-L SELL!

Your Emacs Statistics Service

I’ve been plugging away at the Emacs bug database the last month or so. I’m the kind of person who gets obsessed with something for a time and then I do something completely different the next month. So I’m away from Emacs developments for months on end (and this time it’s basically been a couple of years Due To Circumstances), but when I’m back and hacking away, I’m always wondering:

Are things getting better or are things getting worse?

Things are the same! The chart showing the number of open Emacs bugs has been remarkably flat over the last three years; three cheers for the Emacs maintainers and all their helpers!

But is this because there’s fewer bugs reported?

Hm… No, that red line (opened bugs) is almost a straight line, so it’s about fixing things and closing bug reports at a steady pace. (Interactive charts here.)

So let’s look at the number of commits and contributors:

Surprisingly enough the number of commits per month hasn’t been very large; about 300 per month. (The red line is when Emacs moved to git from bzr.) Or perhaps that’s not surprising; the fewer commits, the fewer new bugs?

In a longer perspective:

And here are the equivalent charts for the number of contributors:

Pretty steady at about 35 per month. (The recent spike is just me spelunking through the bugs database and applying a bunch of older patches that were submitted and not applied at the time they were submitted.) Or as the Emacs bug tracker graphs put it:

And in other Emacs-related news, I thought it was really amusing to see The New York Times using Emacs as an example when educating people about commit messages, and it was doubly amusing that the author used a commit message of mine when doing so.

So now I can say that I’m a New York Times-published author.

If it wasn’t for the small detail that that would be grossly misleading.

In conclusion:

Here’s a picture of some potatoes.

Towards a Cleaner Emacs Build

I’m planning on getting back into Emacs development after being mostly absent for a couple of years. One thing that’s long annoyed me when tinkering with the Lisp bits of Emacs is the huge number of compilation warnings. The C parts of Emacs were fixed up at least a decade ago, but this is what compiling the Lisp directory looks like:

For me, this has been a source of having to go slower when coding: I make a change, look at the output from the compilation window, and then do a double take when I see some warning about something I didn’t think I had touched.

And then it turns out to be an old warning about something completely different.

The number of warnings in an Emacs build has been fluctuating, but sort of growing. There were 440 Warning: lines output by the build process, totalling 1800 lines on stderr. That’s kinda a lot to look at when doing a build.

There’s a number of reasons that the Emacs build looks like this, but the most important is perhaps the somewhat unique way the Emacs Lisp code has traditionally been developed: Many of the major modules have been maintained out-of-tree, and often support a huge number of Emacs versions dating back to the 1980s. Not to mention XEmacs.

This leads to there being conditional calls to code that doesn’t exist in modern Emacs, and code that doesn’t use new calling conventions.

The other is that, well, Emacs has a long history, but the Emacs Lisp language is evolving constantly, what with lexical binding and all. What was good code in 1993 now uses outmoded idioms, and these idioms trigger compilation warnings.

The development situation has changed somewhat over the last few years: Now most of the code in the Emacs tree is developed in the Emacs git repository, and the external packages are instead distributed using the Emacs package system. So the half-external/half-internal development isn’t as big an issue any more (although there are still (very) significant packages developed this way, like CC mode).

And XEmacs compatibility isn’t a major issue any more for many people.

So I thought now was the time to roll up my t-shirt sleeves and get stuck in to the code and get organisised.

90% of the warnings took 10% of the time: They were easy syntactic changes to bring code up to date with the new Emacs Lisp standards. The next 9% took 90% of the time. And then the last 1% took another 90% of the time.

So there was a lot of questions asked and some new tests implemented to ensure that the changes didn’t break anything.

And a lot of questions answered by all the smart people on emacs-devel.

But now it’s over! That is, there’s one single Warning: left, and that’s being pondered.

The total output from a “make bootstrap” is down from 5200 lines to 2900 lines (on this machine; it may vary somewhat), which is a 40% reduction. Looking at Emacs compiling now is a calmer experience.

I also added some new progress messaging in parts where a single section takes so long that it looks like it’s crashed or something, so it’s not purely a “get rid of lines” project.

Virtually all of the warnings fixed were valid warnings (i.e., they were about things we’d rather not see in the Emacs Lisp code), but some warnings were false positives. For instance, Emacs has a method to mark functions as obsolete, and then you get a warning if you load that code, which is a nice way of letting users know that something is going to disappear in a few years. (And Emacs has a very conservative removal policy; obsolete functions are kept around for like a decade.)

But functions are sometimes obsoleted in groups, so you may have one obsolete function calling another obsolete function in that group, and that will issue a compilation warning… and it shouldn’t.

So we’ve introduced a new macro

(with-suppressed-warnings ((obsolete an-old-function))
  (an-old-function :foo :bar))

to make the byte compiler know that we know about this, and not issue any warnings about that. (And there are similar things about the number of arguments to the functions.)

I had to use the macro about a dozen places, which isn’t a lot, percentage wise.

I hope the somewhat less daunting compilation output will help developers, old and new, to get more stuff done. At least a little bit.