Onscreen Keyboards and Emacs and Stuff

This is part two of a series of blog posts that I’m sure is going to go on forever: Wherein I Bitch About Wayland And Modern Life In General.

So, last time, I finally bit the bullet and moved my travelling Yoga laptop to Wayland. This time around, I’m trying to get one more piece of the puzzle to work: The onscreen keyboard.

But first of all, I should perhaps explain my use case, because otherwise this will all seem incomprehensible. I mean, even more.

When I’m travelling, I bring a Lenovo Yoga laptop with me, because it’s so handy to watch movies on (in hotels and on planes etc). I guess most people solve this by bringing a tablet of some kind, but I’ve never found anything as convenient as just using one of these laptops. (Your mileage will vary.) Since the physical keyboard isn’t available in the “tent” position like up in the picture there, I’ve implemented a simple touch interface for Emacs (based on libevdev). Which means that I can just touch virtual squares on the screen to do various stuff:

Here’s the same view, but with labels on the squares to remind me where the various functions are.

Similarly, then actually playing the movie (with mpv), I have a whole bunch of convenient functions available by touching various squares…

… here with the grid shown with labels.

And all this works fine in Wayland, too, since it doesn’t really talk to Wayland stuff at all — it just pokes around in the Linux evdev hierarchy to get the events.

So what’s the problem? The problem is that sometimes I really, really need a keyboard, like when searching for things.

Now, all desktop environments have onscreen keyboards (or as some systems call them, “virtual keyboards” or the like). And… that’s the problem!

That is, there is no One True Onscreen Keyboard — instead they’re implemented for each specific desktop system, and the result is that they’re all lacking in features.

In the Good/Bad Olden Days, if people implemented a thing, they’d make sure that it worked for Linux, Macos, Windows, FreeBSD, whatever, and they’d slowly accrue a lot of features and become pretty useful. These days, a thing like this is implemented as a plugin — we’re here seeing the Onboard keyboard for Gnome Shell. But, as is always the case with these things:

Since it’s tied to Gnome Shell and uses internalish Gnome Shell features, it needs to be updated constantly as Gnome Shell changes. (And they usually aren’t.)

But if all these desktops have onscreen keyboards, why does it matter so much which one I’m using here? Because most keyboards don’t have the feature I need, which is to be able to pop up programmatically, and Onboard has:

qdbus org.onboard.Onboard /org/onboard/Onboard/Keyboard \
org.onboard.Onboard.Keyboard.Show

Beautiful, eh? All the other keyboards I’ve explored work “automatically” — that is, you enable the concept of “onscreen keyboards”, and they pop up when you focus on an input field or the like, like the amusingly named Improved OSK keyboard.

(The name is a typical result of these plugin application ecologies:

It’s a fork of something that already has a bad name, “Improved Onscreen Keyboard” — instead of making one app that works well, we get a large number of vague variations.)

So they’re all useless for my use case, which requires that I’m able to pop up a keyboard when I hit the right square on the screen.

And besides, it “pushes” all the windows up to occupy its own region, which is annoying.

Even if an ideal onscreen keyboard does exist, that fits my requirements like Onboard did — I have no faith in that it’ll still exist and be usable next month.

So what to do? Well, I guess I’ll just… implement yet another crappy onscreen keyboard, won’t I? Won’t I just.

My first thought was to use xdotool, but that doesn’t work under Wayland, of course. People are talking about ydotool as a replacement — but it’s not. It basically just lets you send rawish events to the currently focused window, which is … pretty weird. But I mean, you can switch windows by sending the correct key sequences, but that’s incredibly brittle, and I wonder what the use case for ydotool is at all. And:

Uhm… ok…

So my solution is very specific to my problem: Since I’m using Emacs to get the events, I’m also using Emacs to dispatch the events. That is, if the user hits the q square and we’re in “emacs” mode, we let Emacs handle the q event. If we’re in “mpv” mode, Emacs parses the ~/.config/mpv/input.conf file and sees which command the key maps to, and sends that over to mpv. For instance, for m we have:

(movie-send-mpv-command '((command . ["cycle" "mute"])))

Emacs is already communicating with mpv, of course, so this is pretty easy to add.

But I’m beginning to understand why all these onscreen keyboards are so tightly bound to specific desktop systems: Because the Wayland people haven’t been interested in providing any good protocols for automation and tinkering. (Extensions to the Wayland protocol have been proposed, and finally implemented by some compositors, or course, but not Gnome. But perhaps this will work in a general way in just another decade on Wayland.)

Anyway, let’s see what this looks like now, after tinkering with this for a couple hours…

So here’s the laptop normally…

And with my onscreen keyboard. Hey, that’s not so awful…

So what about when mpv is running? Yeah, that’s fine. I mean, it’s not as if I’m going to type a lot on this — it’s just as an escape hatch when there’s something that’s not available in the normal touch grid.

And I have to find a way to make mpv ignore the touches here, because it pops up some annoying stuff on the screen (title and some controls), but that should hopefully be simple enough.

Oops — Gnome Shell died? Again? That’s the third time that has happened when mpv is closing: Something mpv is doing makes Gnome Shell die. Oh, it’s a known issue, but nobody seems to know why Gnome crashes?

Tra, la, la. Being on Wayland is like being back on X in 1993 — random things you do can take down the entire laptop.

This is surely the Year of Linux on the Laptop!