I fucked up.
Short version: If you’re reading mailing lists with an NNTP news reader via news.gmane.org, you should update your news reader to point to news.gmane.io instead.
Over the past few years, people have asked me what happened to Gmane, and I’ve mostly clasped my hands over my ears and gone “la la la can’t hear you”, because there’s nothing about the story I’m now finally going to tell that I don’t find highly embarrassing. I had hoped I could just continue that way until I die, but perhaps it would be more constructive to actually tell people what’s going on instead of doing an ostrich impression.
So here’s the long, boring and stupid story.
But first, just some background, because there’s no reason you should know what I’m blathering on about: In 2002 I started a mailing list archive called Gmane. It was fun for many years: I got to write a whole bunch of software for the web site, and it was a useful and satisfying hobby (with some quirks, like not daring to visit India). After a decade or so I started getting burned out, and that’s when my problems started.
At that point I should have looked for somebody to take over Gmane in an orderly fashion, but there were (at least) two reasons I didn’t: I felt a childish attachment to the entire project (I think the feeling can best expressed summed up as “NO! MINE!”), and while I could easily see that somebody would want to take over the web part, the NNTP part (which was the one I used personally) seemed too obscure for anybody to be interested in.
So things limped along, me not having any fun at all at this point, and whenever anybody approached me with feelers towards taking over, I would mostly not reply, because when formulating a reason to say no, I didn’t really have a good reason.
And then the DDoS happened, taking out both Gmane and my kind employer, and I lost my shit. I can’t properly express how little I want to be typing these paragraphs, because I’m so embarrassed by my inadequate handling of the situation. But at least here was an opportunity to hand over Gmane to somebody who could continue running it responsibly.
I got a whole bunch of nice offers from people, among them from somebody working at C, a company with a good reputation and huge reach.
I thought it over, and I said yes to the offer from Y. Because I was thinking “those C. people are nice…” Somehow it had gotten into my mind that Y was C. I have no explanation or excuse: I looked at two totally different company names, and I thought in my head that I was saying yes to one while sending email to the other.
My proposal was that I would still own the gmane.org domain, but Y said they had to own it, so I just signed it over, sent them an SSD with the spool, and continued running the NNTP news server (now hosted in a slightly different place).
By that time I had realised that I’m in touch not with the company I was intending to give Gmane to, but at that point I thought it would be churlish to yank Gmane away from them, saying “oops! wrong company!”
Yes, everything here is embarrassing.
It took Y half a year to get the web site up again, and finally all those dead links stopped screaming. However, the site went down frequently (I didn’t get much communication from Y, but apparently there were more DDoS attacks), and it’s been down completely for at least the last year, I think?
Which brings us to last year, when the company I’d been a co-founder of back in 1997 and worked for since, entered into a sales process. (The other co-founder and majority owner got cancer and suddenly died some time earlier.) The new owner would probably not let me host the news.gmane.org server in the server room, so I got in touch with my contact M at Y to arrange a DNS change for the news.gmane.org server.
And I heard nothing back.
So I Cc’d everybody I’d ever known at Y, including the corporate counsel S, and then I got a response from M, CC’d to S, that started with:
I already responded to this on 22nd January.
So apparently M’s email can’t reach me unless he’s Cc’in S? I guess they must have extremely high security email at Y or something, because I can’t think of any other reason for M’s emails not reaching me. (I grepped the exim logs for the previous year: No contact from Y’s MTAs.)
But I thought, well, now I have contact at least, so I can get the DNS updated! Then S responded:
You have contact with M, please follow through with him directly.
I don’t need copying on any of this.
And then… crickets: I didn’t get any further communication with Y, and no DNS update for the NNTP server.
So at this point, I definitely should have done… something… but instead the sale of my employer was progressing, and then it was done, and then I took a long holiday, and then…
I’m an expert at procrastinating, especially when whatever I have to do involves some degree of confrontation.
Time passes, until… two weeks ago, when I got a message that I really have to have my server removed within a couple of weeks, so I sent off new emails to the Y people, even Cc-ing the guy I wasn’t supposed to Cc, and I heard nothing back, at all. So I have to assume the gmane.org domain can’t be updated any more and I have to do something.
I have set up a new server, and I have mirrored all the data to it, but the problem is the domain, of course. The name of the server is one thing: Update your .gnus files to point to news.gmane.io now; the old name will stop working in a couple of weeks (I got an extension, apparently).
The bigger problem is all the mailing lists: When the server goes away, 15K mailing lists will start to bounce.
So the question is: What should I do about that? There’s several options:
1) Let them bounce: It’s not a major disaster, but I can imagine list admins at places like vger being annoyed (Gmane subscribes to hundreds of vger lists).
2) Unsubscribe them all and let news.gmane.org die a final death: The unsubscription has to happen before the server goes away, because the unsubscription emails have to come from the IP address mentioned in the SPF record for gmane.org.
3) Unsubscribe and resubscribe under a different domain: I have registered a new domain gmane-mx.org and done some experimentation. Of twenty lists un/resubscribed (I’ve scripted this bit so it’s not a lot of work to do a few hundred lists in a batch), two were successfully resubscribed. It seems the reason for this is that sourceforge lists can no longer be subscribed without creating a sourceforge account? And many of the other lists just bounced, despite being active, so it means that they’ve moved without the gmane.conf being updated, but the Gmane address is still subscribed. Fixing this will take manual intervention. But do I want to do this? Asking mailing lists admins to trust me again, with a new domain after the previous debacle, is a bit too much, isn’t it?
4) Somebody at Y could wake up and do the fucking DNS update: Make blaine.gmane.org point to 159.69.161.202 (that’s all you have to do) and we can at least have time to do whatever transition we’re doing in an orderly fashion.
I don’t know what’s going to happen, but list admins: If gmane.org addresses start to bounce, and you want to fix things manually, just replace “gmane.org” with “gmane-mx.org” in all the email addresses. This works already.
OK, this blog post turned out to be even more meandering and unstructured than I had anticipated, but that’s probably because I don’t even know what I want to do here, or what people want to have happen.
- Should I just let news.gmane.org die? Is it even useful for anybody any more?
- Should I resubscribe all the lists as gmane-mx.org to continue having a “full” archive, and then try to make somebody more responsible and responsive take over it all again? I mean, there could be a new web interface with similar links as the old one?
- Is the above point even relevant any more in these days of GDPR? What’s the upside for anybody running such a service?
In many ways, I do want Gmane to go away and not have to think about it any more, because it’s all tied up in feelings of inadequacy. I hope I’m not fishing for sympathy here; this has been really awkward to write, and the only conclusion anybody can draw from reading this is that I’m a fuck-up, and I fucked this up.
But I guess I’m asking for some feedback as to what should happen next. I think I have about a week until the old server goes away now, so whatever happens has to happen fast.
Hi Lars,
I use nntp gmane (gwene, too) daily. I don’t have any particular advice on what to do, just want to thank you for these services.
Another daily user gmane NNTP user here, gmane is how I get my mailing list fix – much <3 for the years of service. Thanks for the transparency about the situation, hope you can figure out the best way forward!
news.gmane.org is incredibly useful to me, at least, and has been for many years. If I dare offer myself as a representative sample of “people”, what people would like to happen is for news.gmane. to continue working, and ideally with a resurrected web interface as well. Of course we want that. It’s useful! It’s free!
You shouldn’t care. You have no obligation to us.
Our selfish wishes should in no way prevent you from walking away from the whole thing if you don’t want to deal with it anymore, and it certainly sounds like you don’t.
For both legal (GDPR, India) and personal reasons, it sounds like you need to let it go.
I strongly disagree with the notion that you fucked up, though. Having human emotions, be they of attachment or inadequacy, is not a failure. With 20/20 hindsight, of course it’s possible to find things that could have been done differently.
Still, you (more or less) singlehandedly provided an incredibly useful service to the world at large, for free, for many years. From an end-user viewpoint, it was a huge success. That it eventually outgrew what a single person could manage as a side project is not a failure, it just became a victim of its own success.
That the people you tried to hand it over to failed to keep it alive is on them, not you. To me it speaks volumes about what a heroic and mostly thankless task you must have been doing all this while.
Don’t worry, we’ll manage without. If we want it bad enough, we could always pick up the pieces from your github and run it ourselves. Having seen the toll it has taken on you, though, I’m not volunteering (selfish bastard that I am).
PS. If you’re feeling charitable some day, maybe collect the different pieces into one repo on github called “gmane” with a readme explaining the different bits? There used to be a web page on gmane.org listing the different components, but it’s not there anymore, and figuring out which of your repositories are part of gmane is not trivial.
pok, that has got to be that best reply ever. I have never used gmane, but I agree with you about what Lars should/could do. Kudos, my friend, you are a hero.
Hear Hear!
gmane has been a wonderful thing and to single handedly provide such a service for so long is incredible. I haven’t known any other expert person who even knew how to build and run such a thing let alone possess the drive or commitment to do it.
Hi. I used gmane for a long time until the ddos happened. So a big thank you for your service. You should not feel bad about this whole story. You have done more for “the internet“ than most of your users and you did it for free. So if you have still fun, continue, if not, let it go and don’t feel sorry. You owe to nobody. Thanks again for your time.
R.
Hi Lars,
another daily user of gmane here. It’s part of my daily work and life routine.
I can’t thank you enough for gmane and I would be very sad if gmane would disappear.
Thanks again for all your efforts and work!
regards,
-mika-
Hi Lars,
thanks for everything, I use gmane daily since 2005, subscribed to +- 50 lists, I hope you will pull through, whatever you do, and sorry to not be of any help.
I’ve loved and have been using gmane since nearly the beginning. You don’t owe us anything, but it would be awesome to have the service continue.
first I have to say *thank you* for the excellent service provided for many years!
second I would advise to automatically resubscribe as many lists as possible to the new domain. all other list owners will simply change the address as described or remove the subscription if they are stupid.
last you can think about hand over to a trustworthy party.
there is no reason to be ashamed because of been burn out, trust me, I was in the same situation ignoring everything that bothered me. I takes time and help to recover to regular live.
Kay
I’m yet another regular long time user of the NNTP service. Thank you so much for doing it over the years. Do what right for you and don’t make yourself feel guilty about anything. You’ve provided a useful service for a long time for free.
Thank you for the sustained effort!
You helped a lot of people for a long period.
Please don’t feel bad about any of that,
whatever happened or didn’t happen is insignificant compared to the service provided.
You made a change for good.
First of all, let me join the others thanking you for the incredibly useful service you have provided for many years. This took a lot of personal energy and is not something that should just be taken for granted. So thank you!
If you decide you want to continue trying to get the DNS record fixed, have you tried contacting Y by any other means? They are a company, so I assume they have a phone number and a (postal) address? Maybe via the registrant info for their domain, or a NIC handle for their IP block? Or is the state of affairs so terrible it’s impossible to contact a real person by these means?
Whatever you decide to do, I wish you all the best, and may you be able to spend at least as much time and energy on things you enjoy!
Gmane’s a miracle born of work and volunteering. A happy world without Gmane is always possible, so what? Its role and your role are beyond discussion. The hope you’ve spread is all around us.
Personal growth is always a miracle. Who cares about the downtime. Putting yourself down is counterproductive and does not bring justice to history and to yourself. If anything, this post shows that whatever you do in life, you bet yourself on it, because that’s how you do things.
Gmane or not, if anyone dares not to believe in you, let time do the natural selection. “When You Believe” from The Prince of Egypt may be an appropriate listening.
I’ve been using the NNTP service for a long time. Really handy way of accessing mailing lists of FOSS projects, especially if you like to dip in and out of many projects. The web version was also great, as it had a much better interface than type mailing list archives. I’d be very happy if they both survived, but even if they don’t thanks for your years of work on the project.
Thank you for running the service for all these years. The snafu sounds very frustrating. I hope you get some peace around it.
The big deal in this story is that you gave away ‘gmane.org’ so it is no longer your project, a bit like when the MySQL people sold it to Oracle.
Now ‘gmane.io’ and ‘gmane-mx.org’ are effectively a fork of ‘gmane.org’, like MariaDB is a fork of MySQL. Quite a common occurrence in the world of geek projects (and not just). Ideally get Y to give you back the domain, and as other commenters have said, try to get them on the phone; but not essential.
As to bouncing, mailing list packages are designed to cope with it because most people don’t unsubscribe invalid mailing list addresses (e.g. previous employers), so don’t worry.
Then please register yourself on something like Ko-Fi.org or LiberaPay or Flattr or Gumroad (or even Patreon) so there is a record of a community of supporters.
Just wanted to join in and say thank you for running Gmane all these years. It has enabled me to follow more mailing lists than I probably should have. The sheer amount of work you’ve done and the annoyances you’ve faced and handled speak highly of you, in a way that recent events in no way detract from.
As for the future, I would selfishly wish for something I can point an NNTP reader towards to continue to exist, but please do whatever is conducive to your health and happiness.
Can feel your pain. Gmane used to be quite handy and reliable.
Something you created for a pet project for free with all the hours invested in tweaking and tinkering, watching it grow into the big project it used to be for so long.
Don’t feel obligated to us, the community, for your way forward.
Especially, if you already felt burned out. That’s not something you should ignore or stay at. That’s a serious condition and a warning signal you should take seriously.
The way I see it, is that you did your part and whatever happened is nothing to be ashamed or embarrassed about. I could go on, all I want you to consider is: do what is right for you, the courage you had to inform the public speaks for itself.
Thank you. Good luck whatever you decide to do and stay healthy after all.
Thank you a lot for doing all this. I just received the resubscription notification for a list I admin and I was very happy. I feel your pain too and I don’t blame you at all: I was disappointed by the new owner but in the past three years I only wished I could have helped more before your burnout.
I use the NNTP gateway since essentially forever to read Debian and kernel mailing lists. It would be fanstastic if this service were kept alive, if some old Sourceforge lists can’t be resubscribed so be it, but I’m sure a number of important lists will care to get re-subscribed
Also, if you want/need support financially for the new host, I’d be happy to subscribe for a Patreon or similar.
I would also like to thank you for Gmane and for Gnus. I’ve been using Gnus since 1998 and Gmane since it got launched. I tried out some “modern” alternatives to Gnus but only to realize how much I’m in symbiosis with this awesome monstrosity (proper threading & quoting, scoring, washing, the extensibility etc.).
With Gmane (both nntp and http), being able to chime in to ongoing mailing list discussions has been tremendously useful for me. Luckily, this service (replying to old messages with the proper In-Reply-To) is still provided by mail-archive.org. Does anyone know other alternatives?
Lars, you are one of my heroes of free computing, and definitely the one with the best humor!
Even if Gmane has to go away (I would hope that it doesn’t, but not at the price of personal suffering), I hope that the Gnus saga will continue.
Thanks Lars, ALWAYS^^^
Thank you very much for GMane. I used it extensively via NNTP interface (with KNode, when it was there, then Thunderbird, then Gnus). I have also used web interface for search.
Nowadays I switched to public-inbox.org, though I guess it has only a subset of mailing lists.
My email list just got an unsubscribe of the old m.gmane.org and a new subscription request for m.gmane-mx.org. Is there still a gmane website? I didn’t use the nntp server so much a the website. It had feeds and stats and all kinds of nice features for email lists.
There’s currently no Gmane web site, and whether there will be in the future is a bit up in the air. But probably not. But the news server will continue living on (as gmane.news.io) for the time being, and all the subscriptions have been moved to m.gmane-mx.org.
Lars, thank you for the the honest blog post and that you’ve finally overcome the procrastination on that issue!
I understand that there will not likely be a HTTP interface again soon, but a simple homepage explaining what gmane is, it’s current state and how to access it by NNTP would be very much appreciated.
I’m writing a web site for doing sign-ups of new lists etc, and there’ll be instructions and stuff too.
Just an echo of what people have already said: you did me, personally, a massive favour for well over a decade. I can’t count the hours I saved by having all those mailing lists spiderable and searched. For my money, you need feel nothing but a deep, quiet sense of pride for having been The Person Behind One Of Those Really Useful Bits Of Infrastructure for so long. The public explanation of what’s got things to where they are is appreciated, of course, but the real gift was running the service when you did, and that in no way places any obligations on you, going forwards. Thanks a ton.
Who would have guessed that free server admins have emotions and sometimes make tricky decisions !?!?!
🙂
Thanks a lot for all the good times, I really enjoyed using it when it was up.
A quite long mea culpa while it should not. Just a big thanks for having made gmane, whatever it has been, is and will be!
Even if you may have made some sub-optimal decisions, they are far outweighed by all the efforts that you put into providing this service for free to everyone for so long. Thank you very much for that! I think it would be great if the NNTP service could live on – no matter if it’s on gmane.io or gmane.org.
Thank you for providing this service for so long, for everyone, for free! I do not think you have to worry at all about not every decision having always been perfect. It would be great if the NNTP service could continue to live on, in whatever form and on whatever domain is easiest.
I used the website extensively and the NNTP occasionally until it changed hands. I’m just another person who wanted to say thank you for the time you put into it.
news.gmane.org has always been extremely useful to me (and a big thank you for providing it in the first place). I therefore throw all my weight of one single atom behind the wish that it continues!
Lars, long ago one of the Hamlib project admins subscribed our development list to Gmane. I long pointed to the Gmane archive rather than the one at Source Forge as it is just horrible.
Now, because of GDR and other such silliness, mail list admins of lists hosted by Source Forge are prohibited from accessing the list subscribers. I do not know if there is a means to maintain our list feed to your new service.
Regardless, thanks for everything over the years.
Hi, a lot of thanks from a user since years ! You didn’t fucked up it’s all of us we fucked up to didn’t give you help.
Many, many thanks for the service. Have used gmane NNTP for years and have found nothing else that is as easy and quick to use. Hope there is a way to keep the lights on and for you to not burn out over it all.
I almost never spontaneously leave comments on blogs, maybe twice in the last ten years. But like many others I have been using Gmane NNTP for many, many years, and it has been an invaluable service. No matter what happens going forward, THANK YOU.
Just moved over to news.gmane.io, love your work Lars!
Hi, I use Gmane via NNTP to follow a few mailing lists, at least Emacs ORG-mode, zsh, Python. I just edited my groups to point to news.gmane.io and things work fine. It’s great and Gnus is great so thanks.
As for the gmane.org domain, well, I think you’ve done enough. You’ve documented what people need to do if the domain goes away and that’s enough.
Hi Lars,
thanks, it seems to work well, however the list
nntp+news.gmane.io:gmane.comp.tex.german
seems not to get updated.
I sent several mails yesterday, which I can see in their webpage
https://www.listserv.dfn.de/sympa/arc/tex-d-l
but which are not found in gmane.
Any idea what that could be?
Uwe Brauer
I’ve now resubscribed it.
Lars, if you made any mistake, it was being too generous, carrying more than your fair share in maintaining Gmane–a service for which we should be grateful.
The Web UI was a treasure. If you ever have the resources to consider restoring it, please post a prominent call for help, especially in Emacs-related places, so that we may offer to share in the work.
I’m happy to hear that the NNTP service will continue. It makes following FOSS mailing lists a pleasure using Gnus. Thank you for continuing that service, and for all of your work on Gnus as well.
thank you for gmane. please keep it alive, if you can.
btw, I don’t think `gmane.org.unix-heritage.general` newsgroup is being updated.
“Your subscription request has been forwarded to the list administrator
at tuhs-owner@minnie.tuhs.org for review.”
So I asked TUHS’ owner why does it take so long to approve the request & he said “I only accept subscription requests if they are sent to this [wkt at tuhs.org] email address. Can you get Lars to email me with the address he would like subscribed to the TUHS list?”
Many thanks for news.gmane.io to keep the NNTP gateway alive. I hope it can continue for long time. Some colleagues find it absolutely essential for their work.
A minor nitpick – one still sees “Approved: news@gmane.org” in headers of messages already handled by gmane.io. (e.g. in messages on gmane.comp.mathematics.sage.devel).
Yeah. there’s still gmane.org all over the place. It’ll take a while to fix, because there’s a bunch of scripts that assumes that the headers have “gmane.org” in them.
I never (seriously, never) comment on blog posts (or any other posts for that matter).
But for this one I just need to.
Thank you from the bottom of my heart for providing the NNTP backend for so long.
news.gmane.org started to refuse connections some 20 hours ago and I just found your blog post and updated to news.game.io. Now all is well and I get my daily dose of U-Boot and various kernel mailinglists in a neat searchable fashion again.
Please keep it up! Your service made my life sooo much simpler. Many thanks for doing this and don’t be worried about fucking up every once in a while, we are all just human.
Thanks for the work!
I use nntp gmane because its much more efficient than all the web based things.
Here I have the pure information and no adds useless graphics and so on…
Its a pleasure to read news in this old minimalistic way 🙂
(I switched to news.game.io but it doesnt work unfortunately, at
least trac or pyhton announce are gone)
Whatever happens thanks a lot.
Anton
Just wanted to express my gratitude for this great resource you provided for so many years (and for free!)
It looks like news.gmane.io is fine.
Thanks for your blog and your instructions about new news.gmane.io server
Lars, may you check GMANE subscription for the my Google group gmane.comp.systems.buster ?
I don’t see new e-mails over NNTP now. And I do not see GMANE in user list of this group.
Is it possible to re-subscribe?
It was (and I forgot how) configured 10+ years ago and was broken in last few days of December 2019.
No web interface now at gmane.io, no other contacts just this blog.
Thank you!
I get the following response:
“Your message to ru-buster+subscribe@googlegroups.com has been
blocked. See technical details below for more information.
The response was:
The group ru-buster@googlegroups.com does not allow posting through email. “
OK. Retry. Permission is on.
I assume this means “welcome”? “Ваш запрос о подписке на группу ” ru-buster ” находится на рассмотрении.”
I see it. Approved. Thanks.
Thanks for all your hard work… I also don’t have any advice, sorry….
Another grateful gmane/gwene user, mostly via NNTP. Thank you Lars.
Hi Lars,
Recently, mailing lists for ERC have been retired: gmane.emacs.erc.announce, gmane.emacs.erc.general. Instead of them, a new list, https://lists.gnu.org/mailman/listinfo/emacs-erc , has been created.
Could you maybe add it to Gmane?
Thank you in advance!
Editing/resubscribing/etc lists can be done at admin.gmane.io.
hi Lars, I’m coming late to this thread, but I got here because permalinks to gmane.org in discussion forums (in this case one related to doxymacs) point to permalink.gmane.org — I tried rewriting in a few ways with gmane.io but it did not work. can this be recovered?
No, there is no web interface to Gmane any more.