I’m taking a vacation from my programming day job to fix Gnus/Emacs stuff. As well as going to Utah. But that’s finally over, and I’m 
back on a couch in San Francisco.
I think I’ve covered the outstanding bugs now, so it’s time for new features. Mah favourite.
First off: I’ve been meaning to implement “cloud” features for Gnus for yonks, but the right time never seemed to arrive. But now it’s here!
Basically, my vision is that you’d open an Emacs on a totally fresh machine, say `M-x gnus’, answer a couple of questions, and then you have your entire setup on the new machine.
Using magic? No. Using Dropbox? No.
Using IMAP.
Everybody has an IMAP account. Somewhere. Even if it’s only Gmail.
Basically, it should work like this:
There’s a special “*Emacs Cloud*” mailbox. This contains a sequence of data chunks that will bring your Emacs/Gnus setup up-to-date. There are base chunks, and there are incremental hunks.
The Emacs Cloud will cover a subset of the files needed: .gnus.el, News/*.SCORE, .authinfo, and .signature. And perhaps a few more. If you edit these files, they’ll be uploaded (in full) the next time you sync. In addition, Gnus deltas are handled specially. If you read a message in rec.arts.sf.written, a command to propagate that is added to a trace file that’s also uploaded.
The first time you ask Gnus to start clouding it up, it’ll upload all this data, in full, in a “base chunk”. After this, only “incremental chunks” are uploaded. When downloading, Gnus will download enough stuff to recreate the situation on the other side(s). When the incremental chunks (cumulatively) become big enough that they’re bigger than a base chunk would be, a new base chunk is uploaded.
Before uploading anything, the entire chunk is compressed, encrypted (using symmetrical encryption), and then base64-encoded.
So, to return to the opening scenario: You’re on a fresh machine. You say `M-x gnus’. Gnus will then ask you “Do you wish to sync with the Emacs Cloud?” You answer “yes”, and Gnus then asks you what the IMAP server is, what the IMAP user name and password are, and what the Emacs Cloud password is. Gnus then connects to the IMAP server and downloads the data. And there you are.
Later, during normal usage, Gnus will automatically detect updates done in other instances, and download/upload data as required.
So you get a kinda sorta virtual file system over IMAP.
Details to be decided: What kind of format to use for the chunks (something rnews-like?), and what encryption to use. And… er… no, I think that’s it.
And then there’s only typing remaining and we’re done. Unless I get a cold. Then there’s procrastinating before starting to type.
Now, if somebody would just release an Android phone with a physical built-in keyboard, then I could just run Emacs natively on the phone, and read all mailing lists and blogs in Emacs. And with eww, I’d never have to leave Emacs and actually use the horrible Android OS.
In case I have many IMAP accounts (and I do), where will the *Emacs Cloud* group reside? Only one of them containing the read stats of all others too, or on every one containing only the read stats of that account?
That’s one thing I’ll have to look into. I was originally not going to duplicate data from (other) IMAP servers, but I’ve found that the amount of data needed to bring an old .newsrc.eld up-to-date over (normal) IMAP is pretty depressing. It might very well make sense to store the data for all the IMAP servers in the *Emacs Cloud* group. Even for the primary IMAP server.
I think I would want my nnimap groups kept outside of the new mechanism.
It works find for me the way things are, with multiple Gnus installations and other MUAs accessing the same IMAP folders.
“Duplicating” the IMAP info itself shouldn’t affect other MUAs. We’d still be updating IMAP marks the traditional way. This is more like a “fast-forward” feature for Gnus.
Ted Z suggested using dribble files as the file format. At least for marks… I don’t know if the other .newsrc.eld stuff is put there…? And probably not the .gnusrc.el stuff…?
On a side note on the android comment at the end: have you been successful in running native emacs on android? With an Xperia tablet Z and a bluetooth keyboard I ran into issues like “no alt key”, and “Norwegian keys not working”, for which I could google up no easy fix.
The raw dribble format is too verbose. We should be able to coalesce the data a lot better before uploading the chunks. But it’s not all that different from a dribble file.
As for Android, I still don’t have an Android phone, so I’ve never tried. But LG is supposed to release a phone called “Optimus F3Q” with a sliding keyboard next week. I’ll have a look at it and see whether it seems nice or not…
This feels overly complicated, very hard to debug and out of sync with the rest of Emacs. I’d rather go for a solution that’s based on TRAMP or something similar where the user has a wide range of choices where to store his data; if an additional “imap” method is added for the experimentally inclined, okay. But I wouldn’t want that a very useful feature (being able to use Gnus from multiple machines) is hinged on using IMAP :-).
I love the idea of having .newsrc.eld and other metadate in gnus-home-directory synced automatically between gnus MUAs through an IMAP server.
Right now, I sync gnus-home-directory with git (which works fine), but I’d rather have something that more resembles dropbox (where I don’t have to think about syncing).
Also, if one runs an offline IMAP server (dovecot), can one just rsync the additional gnus metadata stored in dovecot’s mail_location to another dovecot instance offline or in the cloud?
I ask this because right now I just rsync my IMAP Maildir between laptops depending on which laptop I want to read my mail on. Mail from the internet is actually pulled in from fetch mail.
Ideally this new Emacs mail cloud might resembler git-annex, http://git-annex.branchable.com, where IMAP “repositories” could be synced with the cloud or to a local IMAP instance or both.
I like it! I currently sync with unison; it works, but it’s not great (sometimes things get a bit out of sync and I have to re-add marks to messages). Something built-in that’s meant for the task sounds awesome. And everyone has IMAP, yes.
I don’t think that syncinc just one applications configuration over imap is worth the time. I use vcsh[1] to sync all my dotfiles.
Syncinc just the feed data might be interesting. On the other hand one can easily setup an Feed to mail[2] workflow and sync feeds with imap.
[1] https://github.com/RichiH/vcsh
[2] http://blackwinter.github.io/rss2mail
re: android phone: get a motorola droid 3 – the keyboard will send meta and control keys correctly. Have had mine for 2 years and absolutely love it after figuring out how to set up SSH to my remote server.
been reading the mailing list posts, and I would prefer if I can do my own syncing of my ~/.authinfo.gpg – I already have that file in my private dotfiles git repo that I keep up-to-date on my laptop. I don’t need it doubly synced (and especially not to my email providers – fine and upstanding people tho they may be, I don’t want to burden them with that responsibility). Other than that, really looking forward to it 🙂
Always thanks, Lars!
I like everything for Gnus!!!