A New Eval Server For Emacs

Emacs has a mechanism for client/server communication (and remote eval) that’s simultaneously too insecure and too secure at the same time.

Here’s the extremely convenient way to start a server:

(setq server-use-tcp t
      server-host (system-name)
      server-name (concat "foo-" (system-name)))
(server-start)

This will create a file (if called on a machine named “stories”)
called ~/.emacs.d/server/foo-stories with the following content:

192.168.1.53:41929 4021
fH?D+M=u=r@N1O^L-`c"c_GYUj%zj,,hc&9QGF+0}0;c}M3>Evc2SdH_N\`SSV\t

If you then say (on the command line)

$ emacsclient --server-file=foo-stories --eval "(tellstick-switch 0)"

the emacsclient binary will read that file, find the IP address and port number there, connect to that address, output the second line (which is a security cookie), and then the form to be evalled. The server will check that the cookie is what it’s expecting, and then eval whatever it’s asked to eval, and will return the result.

Now, for this to work, the host that you’re running the emacsclient on has to have access to that file, which means that (in practice) the entire thing is based on a shared NFS or sshfs setup, which makes it impossible (or at least extremely awkward) to use in many situations. So it’s too secure.

On the other hand, if you do have that cookie, you can make the server eval anything. And the communication is in clear text, so you can man-in-the-middle as much as you’d like. So it’s really very insecure and can’t be used any other place other than a totally locked-down, trusted network.

So Emacs should have a different way to do an eval server. Specs:

  • Don’t rely on a shared file system
  • Communication should be encrypted
  • The server should be able to limit the number of exposed endpoints

      So, ideally starting the server should look something like:

      (start-eval-server "lights" 8100 '(turn-on-lights turn-off-lights))

      where the first parameter is the name of the service, the second is the port number it should listen to, and the third parameter is a list of functions it will allow the caller to have evaluated.

      The client will typically look like:

      (eval-at "lights" "stories" 8100 '(turn-on-lights kitchen))

      For encryption, we could either do TLS or something home-brewed. And as all crypto experts tell us: Always roll your own crypto, so we’ll go with the latter one. But most of all because we’d really only be doing self-signed certificates, so TLS gives us a lot a problems without giving us security.

      But we could rely on an ssh-like setup with pinned self-signed certificates to give the client security, and client certificates to give the server security, but it’s… difficult to see how that could be made to work without a lot of handholding. And Emacs doesn’t have built-in tools to create certificates, anyway…

      Instead we’ll do symmetric encryption with a shared key. We’ll have ~/.authinfo entries like:

      machine lights port 8100 password s0pers3cret

      So… we’re not relying on a shared file system, but we are relying on pre-shared keys.

      I’ve put the results on github, and in addition to the Emacs client/server bits, I’ve also added a shell script that relies on openssh to do the encryption.

      $ ./eval-client lights stories 8710 '(+ 5 4)'
      9
      $ ./eval-client lights stories 8710 '(+ f 4)'
      Error: Got error message from server:
      (number-or-marker-p f)

      Seems to work!

      The client-server communication is plist-based, so they pass around things like:

      I think AES-256-CBC with PKCS#7 padding is a reasonable cipher to use for this type of thing, but I’m not an encryption expert. Does anybody have any input on that bit?

      The sexp specifies what cipher we’re using, so the protocol should be backwards/forwards compatible if we move to a different cipher in the future. The server should use the same cipher the client used, or if that’s not supportet any more, return an error.

      The error message is also encrypted, of course.

      Another design niggle: I’ve made the eval-at function throw errors when the server says an error has occurred, and if the error occurred when evalling the form, eval-at will throw the same error. Like:

      (eval-at "lights" "stories" 8710 '(+ t 2))
      -> (wrong-type-argument . "(number-or-marker-p t)")

      Does that make sense? I mean, eval-at itself didn’t have a wrong-type argument: It made a network connection, squirted some data over, and got some data back. So it was a success! But…

      What do you think?

      When I find some time (and after using this for a while to see whether it makes sense), I’ll probably propose adding this to Emacs and deprecate the old emacsclient stuff.

Cropping Images in Emacs

I woke up in the middle of the night and started thinking about cropping images in Emacs, as one does. I started wondering how Emacs processed mouse events, and that turns out to be very easy: You just use `read-event’ inside a `track-mouse’ form, and you get all the events and coordinates, offset from the window or the image under point, which is just perfect for my use case here.

 

So after work today, I started typing, and there it is.

Now, cropping an image in Emacs is one thing, but the other is… what do you do with the result? I mean, just displaying the cropped image is nice, but pretty useless. I mean, you can save it, I guess, and that would make sense from an `image-mode’ context. But more generally useful would be from a document composition mode, so I just stuck it into the package for editing WordPress posts.

Behold!

I think it may make sense to factor this out into its own little package so that it can be used elsewhere, but I don’t really have the time at the moment, so I guess it’ll have to wait…

The code is up on Microsoft Github as usual.

Variable Pitch Tables

While using the Emacs WordPress interface on various screens here, it became clear that the sloppy way I was laying out various tables just didn’t really work. I was using the `variable-pitch’ font and sort of eyeballing how much space each column would take and then using `display ‘(space :align-to 100))’ to line stuff up.

But the size of fonts varies wildly from computer to computer, and what looked nice on my HiDPI laptop didn’t line up at all on my LoDPI screen.

So I switched to using fixed-width fonts:

And I just thought, *sigh*.

It just looks so oldz. I spend a lot of time in Emacs in eww which does nice fonts, so I’ve grown less used to the er starkness of tables like this.

Surely there has to be a way to do tables with proportional fonts, and of course there is: eww/shr lays stuff out without a problem, so I just had to take a similar approach here.

And behold:

The package is on Microsoft Github now.

This also allowed me to implement table headings that you can click on to sort the table on the different columns. And I wanted to keep the interface really simple, because I have to say that I hate working with the tabulated-list.el interface, which is a mess of buffer-local variables and magic always happening wherever you’re not looking in the code.

So the interface is one simple function that takes obvious parameters: A list of column names and max widths, and then a list of row contents:

(variable-pitch-table
 '((:name "First" :width 10)
   (:name "Second" :width 5))
 '(("A thing" "Yes")
   ("A wide thing that needs chopping" "And more")
   ("And the last one" "Foo")))

But you’d normally pass in strings that are made with something like

(propertize "At thing" 'face 'variable-pitch)

to get whatever proportional font you want on each element.

Further Fun With the Clipboard

In the previous episode of this blog, I mused about how Emacs should perhaps handle non-text yanking (i.e., HTML and images).

I didn’t really want to write a mode to examine the clipboard and the selections, but I did anyway just now, and I discovered something kinda interesting.

Well I never!

Anyway, this is basically what the mode looks like:

It just lists all the types of selection that exist under the primary/secondary selections, as well as the clipboard. In this instance, I’ve done a “Copy image” in Firefox. So I can inspect the text/html “version” the image…

Which turns out to be the <img;> bit, logically enough.

And the image/png is the image itself, even more logically enough.

But here’s the slightly interesting bit: If you mark some text in Firefox (i.e., put the text into the primary selection in nerd speak), you get these selection types:

And the TEXT bit is as you’d expect…

But what’s that text/x-moz-url-priv?

It’s the URL of the page you marked the text on! This can obviously be used for UX purposes, like providing a function that’d simultaneously quote something and do the link at the same time. Like this:

So something like:

  (when-let ((url (x-get-selection-internal 'PRIMARY 'text/x-moz-url-priv)))
    (insert (format "On <a href="%S">somebody</a> wrote:\n\n"
                    (ewp-decode-text-selection url))))
  (insert "<blockquote>\n")
  (insert (substring-no-properties (current-kill 0)))
  (insert "</blockquote>\n\n")

You can find this stuff on Microsoft Github.

Chrome isn’t as helpful as Firefox here, you just get a very minimal selection list when marking some text there:

(Oh, yeah, I added a command to do screenshots directly from the ewp composition buffer. Otherwise this blog would have taken me HOURS and HOURS to make.)

Of course, how useful this stuff is outside of a writing-for-the-web context is debatable…

PERSON A: It’s not useful!

PERSON B: Oh, OK.

Well, that was a short debate!

The Mysteries of the Selection

After writing the package to edit WordPress articles in Emacs, I started thinking about doing something about non-text X selections in Emacs.

Emacs has all the required low-level support it needs to be able to yank HTML and images into Emacs, but nobody’s mainstreamed more high-level support into Emacs. Mostly because nobody really knew what it would look like, I think. (And because most Emacsens don’t really blog that much.)

Let’s have a look at what’s available here.

Let’s mark a word in an xterm, and then let’s see what Emacs gets:

(gui-get-selection 'PRIMARY 'TARGETS)
[TIMESTAMP TARGETS MULTIPLE UTF8_STRING
COMPOUND_TEXT TEXT STRING text/plain\;charset=utf-8
text/plain]

and compare when marking a word in Firefox:

(gui-get-selection 'PRIMARY 'TARGETS)
[TIMESTAMP TARGETS MULTIPLE text/html
text/_moz_htmlcontext text/_moz_htmlinfo
UTF8_STRING COMPOUND_TEXT TEXT STRING
text/x-moz-url-priv]

See? When marking words in an xterm, you get at text/plain thing you can yank in Emacs (and that’s what normally happens), but you mark something in Firefox, you get a text/html (among other things) in addition).

The same thing is the case in Chrome, although there you have to say “Copy” first to get the text/html part, and it lands on the CLIPBOARD and not in PRIMARY.

But that should be OK, anyway. Let’s look at the output from the text/html selection from Firefox.

Er, OK, that’s… little-endian utf-16. Er, OK… So that’s a thing?

Let’s look at the selection from Chrome, then:

So that’s a UTF-8 string? With… a nul at the end?

*sigh*

Well, OK, it’s easy enough to recognise one or the other forms, as long as there’s only two. *crosses fingers*

That means that the function for yanking HTML ends up being something like:

(defun ewp-yank-html ()
  "Yank the contents of the current X text/html selection, if any."
  (interactive)
  (let ((data (loop for type in '(PRIMARY CLIPBOARD)
                    for data = (x-get-selection-internal type 'text/html)
                    when data
                    return data)))
    (if (not data)
        (message "No text/html data in the current selection")
      (set-mark (point))
      (insert
       (if (and (> (length data) 2)
                (= (aref data 0) 255)
                (= (aref data 1) 254))
           ;; Somehow the selection is UTF-16 when selecting text in
           ;; Firefox.
           (decode-coding-string data 'utf-16-le)
         ;; But some sources add a nul to the end of the data.
         (decode-coding-string
          (replace-regexp-in-string (string 0) "" data)
          'utf-8))))))

Or should it look at the TIMESTAMP from the selections and then yank the newest one? By testing, it seems to be doing the right thing when not doing that…

Yanking images into buffers works along the same line, but there’s differences between where the images originate, of course. If doing a “Copy image” from Firefox, I can yank them as any format (image/png, image/jpeg), but from Chrome doesn’t support transcoding, so you have to loop a bit to find what the format really is. Or… I could look at the output from `gui-get-selection’? Yes, that seems like a better idea.

Let me try to yank a random image from Google…

Yup, that works. So you end up with something like:

(loop for type in '(PRIMARY CLIPBOARD)
      for st = (loop for st across
                     (gui-get-selection type 'TARGETS)
                     when (equal (car (split-string
                                       (symbol-name st) "/"))
                                 "image")
                     return st)
      when st
      return (x-get-selection-internal type st))

But just yanking an image into Emacs isn’t very useful: You have to be able to do something with it. In an HTML-like mode as this Postpress editing mode makes that kinda easy: Just represent it as an <img> element… and stash the entire image in the src=”data:image/jpeg;base64,FKJDSHJF…” bit. Since I’m showing the image as the `display’ bit on top, having a few megabytes of noise in the buffer doesn’t make any difference.

I was worried that this might somehow slow Emacs down, but nope. Displaying very long lines is something that Emacs doesn’t like, but as long as it’s hidden, it doesn’t seem to make any difference.

And it autosaves nicely, too.

OK, making it that easy to insert images into blog posts perhaps isn’t a good idea, but why should all ideas be good?

Anyway, I think something like this is something that Emacs should have, and I think I’ll pursue that further when I get time to do some more real Emacs work (which probably won’t be until like February).

I was originally pondering whether to offer an interface to a detailed view of the selection/clipboard, so that you can journey into all the different selection types, but after using this stuff for a couple of days, I think I’ve reached the conclusion that the only other two yanking types that makes sense (other than plain text, which we have now) are HTML and images. At least in a WYSIWYG-ish context…

Finally! Emacs is going to catch up with 90s editors! At least on this minor point! Amazing!

Editing WordPress Articles in Emacs

I’ve got my blog on WordPress.com, which has some positive aspects (I don’t have to run it) and some negative (I can’t control it fully). But whether I’m running WordPress myself or not, there’s one thing that’s always true: I loathe the browser-based editor.

When composing blog articles, I’ve been using the post-by-mail interface that WordPress.com offers, and it works OK. It’s got severe limitations in what it accepts, though: No <div>s or anything more fancy than a link and an image, really, and even those can’t be combined.

But editing is the main problem: I’ve tried using the Edit with Emacs Firefox plugin, and it doesn’t work with WordPress.com, apparently. So I’ve just… not… edited much, because it’s such a pain in the ass. The “visual” editor doesn’t even have reliable scrolling.

So yesterday evening I pulled myself together and started googling “emacs wordpress”, and found org2blog, which allows you to edit your WordPress blog posts using org mode. So it’s possible!

It turns out that this all is so much easier than I had imagined: WordPress has an old xmlrpc interface that you can just call with your credentials to do all this stuff. (It also has a new interface that’s oauth2-based, i.e., it requires a web browser in the auth flow with is just ick.)

The org2blog people have kindly separated out the low-level stuff into its own library, and I wrote a new library, ewp on top of that. Because even if Org is nice and stuff, I just want to edit my WordPress articles. I want WordPress to have the articles, and don’t want to have a “shadow” set of master texts.

With the help of that library (after fixing a multi/unibyte bug in the xml-rpl.el library (that was merged fast by the nice maintainer)), I finished off the library yesterday, and it’s basic, but does what I need, I think.

It lists the posts on WordPress, allows me to edit them, and to create new posts. (And, of course, since I’m very picture-happy, I also added some stuff to upload embedded media (which was a problem with the post-by-mail interface, because WordPress’ SMTP servers had a size limit of 100MB per email which they refused to increase, even after I asked them (so unreasonable!!!ONE!)).)

Here’s a Youtube demo of me typing away at my very shiny sofa laptop:

This has all been tested very, very lightly, of course, so I’d caution er caution. I’ve tried posting and editing myself on my WordPress.com blog, but who knows what happens on other WordPress installations…

And I’m sure it’ll grow more features as time passes. Things tend that way, don’t they?

Z-Wave and Emacs

I’ve had a 433-MHz-based “home automation” system (i.e., light switches) for quite some time. It works kinda OK. That is, I’m able to switch the lights on and off, which is the main point.

But, man, the range of 433MHz devices sucks, including all Telldus models. I’ve been able to overcome the problems by having transmitters all over the apt, but getting wall-mounted light switches to work with any kind of reliability has proven impossible.

The problem is that the protocol is just inherently unreliable: It just sends commands out into the ether, and doesn’t have any replay logic or ACKs going on.

But there’s newer (but also old) tech available, and 433MHz devices are disappearing from the stores, and the winning protocol is Z-Wave.

So I got a Z-Stick:

It’s a nice device: You plug it in and it shows up as either /tty/USBx or /tty/ACMx (depending on the model, but it makes no difference), and you talk to it by squirting some bytes at it.

I had expected the protocol to be really well-defined and open, but it’s a proprietary protocol that people have been reverse-engineering for years, which led me to believe that there surely would be a nice repository somewhere that describes the protocol in detail, and has, say, an XML file that describes all the different network packages.

Nope.

But after some googling I found this gist that at least let me check whether I can talk to the device…

… and it works!

It turns out that the Z-Wave protocol is kinda nice. Each package has a checksum, and devices retransmit commands a few times unless they get an ACK, and Z-Wave plugged-in devices (like outlets) work as repeaters, so the Z-Wave network works as a mesh. It’s kinda cool.

There’s a ton of software to control these devices, but using something like Open HAB is just so… end userish.

Instead I wanted to just plug it into my existing Emacs-based system so that I don’t have to, like, use software. Software sucks.

If no machine-parseable spec is available, at least there must be some other sensible software out there that I can just crib implementation details from, right? So I binged “z-wave python”.

And just found Python OpenZwave, which turns out to be nothing but a wrapper around OpenZWave, which is a C++ library.

Whyyy.

It’s a simple protocol, really. You just read from a serial device and then squirt some bytes at the device. It’s not like you need to do Fourier realtime transforms on a vast byte stream or anything.

But who am I to criticise people for choosing odd programming languages to implement their free software? Since it’s C++, at least probably they created some kind of over-engineered monster where you have some XML files that define the protocol, and then they create objects from the stream and use a lot of polymorphism that’ll make the control flow impossible to follow (according to the “everything happens somewhere else” object oriented methodology), but that’s good, because I can just use the protocol definition files and ignore the rest.

Right?

Right?!?

No, the main work flow is based on nested if statements with lots of “switch( _data[1] )”. But surely after that they’ll parse the protocol packages into something sensible?

*sigh*

Well, at least they used variable names that’s understandable. But what’s data[4] and data[6]?

*sigh*

I”M SORRY! THIS IS…

Ok, again, I have no business giving a code critique of this library written by, I’m sure, very nice people and put on the intertubes for the world to peruse and use, and despite being written in the “least information density per pixel displayed” style, it’s clear and easy to follow, and has an impressive amount of comments. It’s still all kinds of wrong.

Perhaps it’s just coming from a different culture? It’s Windows-style programming?

I don’t know, but anyway, with the guidance from this excellent piece of, er, software, I was able to make Emacs parse and execute commands when I touch a wall switch.

And Z-Wave works! Where a couple my light switches were a bit hit and miss before, they now work with 100% reliability over the last week.

I don’t have any Z-Wave outlets yet, so I haven’t bothered to implement sending commands to devices, but I’m sure I’ll have to implement that at some point. But as far as I can tell, that should be pretty straight-forward. I foresee a lot of “but what’s _data[7]?” in my future.

Somebody should still create a Z-Wave repo with protocol definitions, especially since it’s now an open-ish standard. But only as PDFs, of course.

[Edit: I should have googled a bit more, because it pretty much looks like everything I wondered about is in the OpenHAB distribution, which is in Java and has more XML protocol definition files than you can shake a stick at. Well done, Java peeps.]