15×10%

Time for another one of these posts about Emacs development, I guess? It’s been two months since the last one, and I’ve basically been slacking off, mostly dealing with new bug reports instead of going dumpster diving. I mean, bug triage.

We started at 2460 open bugs in this stretch, and we’re now at 2402, so that’s not too bad. It’s been unusually quiet in the bug tracker, actually, and I’m not sure how to interpret that. Did we finally perfect Emacs so that there’s nothing to report, or did the war against Ukraine make people less apt to poke at Emacs, or did finally everybody start using TextPad Pro?

On the other other other hand, the bug tracker has been unusually active the preceding months, so perhaps it’s just a regression towards the mean.

And Emacs 28.1 has been released, so I’d expected a bump in the bug inflows from that, but nothing huge.

So instead of digging into the bug data more, let’s just look at some of the new stuff in Emacs 29 since the last blog post.

Less Recursive GC

I think the most important fix these past couple of months on the trunk has been the GC fixes courtesy of Mattias Engdegård. It’s been a long-standing er embarrassment that you can segfault Emacs by just consing up some structures. Since basically forever (at least the 80s, I think?) Emacs’ garbage collection has been implemented in a pretty recursive way. Not for normal lists, but there’s nothing, in principle, stopping anybody from creating lists that have the “rest” in the car cell. This makes the GC recurse, and if it recurses sufficiently, Emacs would just segfault.

*poof*

Now, people learned to Just Not Do That, so these segfaults are rare in practice, but still… yuck.

Mattias has now fixed this for Emacs 29, and the resulting GC is even slightly faster to boot.

Drag and Drop

Emacs has long had some basic support for dragging and dropping things into Emacs, but not really much support for dragging things out of Emacs.

Courtesy of Po Lu, you can now drag files from Dired to other programs, and of course, drag text from Emacs to other programs. This requires support on all the different operating systems and different window systems, so it’s a pretty daunting task implementing it across the board.

Animated webp support

There was a bug report about some webp images not working in Emacs 29. Emacs got native support (via libwebp courtesy of Stefan Kangas) earlier this year, and it mostly worked fine — except with animated webp images, which didn’t display at all. (As an aside, it’s kinda amusing that the webp format was developed from the VP8 video format… and now there’s animated webp images, which aren’t VP8 videos.) It turns out that parsing those files requires using the demux interface in libwebp, which is in a separate library file.

I thought that sounded like a fun little project, so I started reading the libwebp documentation, and a couple of days later:

Easy peasy lemon squeezy — the resulting code is short and nice, because the libwebp interface functions here are sane and well thought out. However, it took me several nights poking at this to actually find what interface to use, because the documentation doesn’t give any guidance on this.

If you look at the demux documentation, it starts with the WebPIterator iterator, so that was what I tried to use. After some head scratching, I had this:

Freddy is displayed fine, because his webp file is all “complete” frames, but the more advanced nyan cat is all garbled. That’s because this interface is the low-level interface that requires you to do the blending between frames yourself. What you’re supposed to do is use WebPAnimInfo instead, which does all the heavy lifting for you.

I’m not the only one confused by this. Here’s how ImageMagick displays the nyan cat:

I’m guessing the ImageMagick people implemented this with a test file that has only complete frames (like the Freddy webp file), so they didn’t catch the problem.

So: Good documentation, but it would have been helpful with some implementation guidance as a preamble, mm-kay?

Anyway, after poking away at the webp animation code, I realised that it would be pretty easy to finally fix the GIF animation code, too. I mean, it’s functionally OK, but it’s dog slow. After putting in the same sort of animation cache mechanism here, displaying an animated GIF is… a lot faster? Here’s the display from top while displaying the same GIF in Emacs 28.1 and in Emacs 29:

So it’s not fabulously fast or anything, but it’s 10x faster than before. (And more importantly, doesn’t slow down quadratically with the length of the GIF, which made looking at long GIFs in Emacs impossible.)

Hm… this reminds me… wasn’t there some other issue with the image cache when doing animated images? Yes! We were stashing data in the image object and that made display-level caching fail!

So with that fixed, we’re down to:

OK, now it’s really starting to help. 🙃 With a torture test page, containing dozens of animated GIFs:

100% in Emacs 28.1 (which means that it doesn’t have a chance to actually animate all the GIFs) and 14% in Emacs 29.

Nice!

Open Closures

A more internal new thing is the “open closure” concept, courtesy of Stefan Monnier. It’s like a struct that you can funcall? Or like a function object that you can interrogate about various data and metadata.

So — a closure that’s open, so it has the best name ever.

Variable Pitch Tables

Mixing fonts, images and text size in tabular displays has always been a pain in Emacs, so I’ve now written a new package for that: vtable. The thing I’ve been using to test it is the ewp package, which I’m writing this very blog post from:

All nice and lined up.

Emacs has had tabulated-list-mode for quite a while, but in addition to not dealing with proportional fonts, it’s also just awkward interface-wise: It expects to own the entire buffer it lives in, so you can’t have a table in the middle of some other text, and it stores all the data in buffer-local variables, so you can’t have several tables in the same buffer. And interacting with the table always seems to happen via setting some obscure variable and then calling some other obscure function, so I wanted to make an interface that fixes all of that, in addition to making it trivial to create simple tables, and easy to create complex tables.

My inspiration for the interface design was originally the Common Lisp/LispWorks table library developed for my former employer by Espen Vestre, but it diverged quite a bit in the end. The idea is basically still the same: You have data, you have the formatted data, and then you have the data that’s being presented, and you can alter these things at several levels. (Which is important when resizing a column and you also want to resize images shown in a column, for instance.)

Here’s the simplest table possible, I guess:

(make-vtable
 :objects '(("Foo" 1034)
            ("Gazonk" 45)))

This displays as:

You’d want a header, though, so:

(make-vtable
 :columns '("Name" "ID")
 :objects '(("Foo" 1034)
            ("Gazonk" 45)))

Which is:

And here’s how simple a very simple version of C-x C-b could be implemented:

(make-vtable
 :columns '("Name" "Size" "File")
 :objects (buffer-list)
 :actions '("k" kill-buffer
            "RET" display-buffer)
 :getter (lambda (object column vtable)
           (pcase (vtable-column vtable column)
             ("Name" (buffer-name object))
             ("Size" (buffer-size object))
             ("File" (or (buffer-file-name object) "")))))

Which gives us:

Finally, here’s a very simple image browser which displays most of the significant features:

(make-vtable
 :columns `(( :name "Thumb" :width "500px"
              :displayer
              ,(lambda (value max-width table)
                 (propertize "*" 'display
                             (create-image value nil nil
                                           :max-width max-width))))
            (:name "Size" :width 10
                   :formatter file-size-human-readable)
            (:name "Time" :width 10 :primary ascend)
            "Name")
 :objects-function (lambda ()
                     (directory-files "/tmp/" t "\\.jpg\\'"))
 :actions '("RET" find-file)
 :getter (lambda (object column table)
           (pcase (vtable-column table column)
             ("Name" (file-name-nondirectory object))
             ("Thumb" object)
             ("Size" (file-attribute-size (file-attributes object)))
             ("Time" (format-time-string
                      "%F" (file-attribute-modification-time
                            (file-attributes object))))))
 :separator-width 5
 :keymap (define-keymap
           "q" #'kill-buffer))

This results in (if you’ve been watching Japanese 60s movies):

And increasing the size of the first column allows the displayer to resize the thumbnails:

And so on.

Make Commands Query

You can now make any command query you before it’s executed. For instance, if you want to be queried before doing C-x C-c, you can just say

(command-query 'save-buffers-kill-terminal)

in your init file.

More Electric Completion

M-<down> and M-<up> can now be used to navigate completions directly, which makes for quicker selection between candidates. This, and many other user interface improvements in completion is courtesy of Juri Linkov and Jimmy Aguilar Mena.

Clean CI

After fixing some obscure bugs (only appearing in a few configurations) and a couple of not-so-obscure bugs (along with some judicious disabling of some tests that were timing out on EMBA), we now have a green board:

Whoho! That means that we hopefully can start to send out nagging mails to people that commit something that makes things go red.

(We should, of course, refactor the tests that take too long and re-enable them on EMBA, but having those timing-out (that were only triggering on EMBA) wasn’t productive.)

This is how things used to look:

Which was sad.