Gwene Is Up Again, Too

After a month-long hiatus after the troubles, Gmane’s sister web site, Gwene, is back up again, too.

Gwene allows you to sign up RSS feeds via the web site, and then you can read those feeds by pointing your news reader at news.gwene.org.

rememberGwene also used to have a web-based interface to browse the contents, and I may resurrect that at some point, but it didn’t really seem very… useful.  Everything on Gwene is already on the web, and replicating that just seems rather churlish.  But writing it in Node was kinda fun.

Gmane Alive!

A few weeks back, a DDoS was the final straw that broke this Gmane camel’s back, and I took my marbles (as they were) and went home.

I mean, I shut down the web interface of gmane.org.

But now it’s back, and under new management: Yomura Corporation.  Not all things are up yet, but it’s getting there, and looks rather nice.  I mean, it looks kinda like the old site, and nothing’s better than that!  Surely!  Here’s an example article.

Thanks very very much to Martin and Mark for setting this up, and I’m sure they’ll can respond to things on their blog.  And also thanks to everybody else who offered to take over Gmane.

Gmane has been archiving mailing lists as usual during the web interface holiday, and those articles aren’t available on the new web interface yet, but that will be fixed after a while.

And while I have all yourn attention, I want to write an apology:

While the DDoS was the trigger here for me to abandon Gmane, it’s been brewing for a while, and I could have handled the whole situation better. I started Gmane in 2002, and I spent a lot of time on it the first few years. But the last five years or so just haven’t been fun. I’ve done virtually no new programming on the site in that period, and that is, after all, the fun part.

In addition to the death threats and the people who want to sue me, etc, it’s also been a constant low-level source of anxiety. Oh, there goes a server… let’s build a new one… oh, now the RAID array is growing full, let’s buy more disks…

So I should have gotten rid of Gmane years ago. I mean: Found someone else to take it over in an orderly fashion, instead of this panic attack.

The reason I didn’t is probably partly because I didn’t think anybody would (and especially not the NNTP bits, but I was completely wrong), but also because I just didn’t want to let go. I still have ideas for what I wanted Gmane to be, but I didn’t have the time… or really, I always have time. I mean, it’s all choices. I could have written a new Gmane web interface instead of watching a few movies, couldn’t I?

And there was always some fun bits about Gmane. I could regale people in bars with drunken stories about being sued in India, for instance. Not many people I know can!

And the thing is, most people in the world are great people, and I wanted to provide this service to these people. I know I kvetch a lot, but less than one in a hundred Gmane-related interactions were in any way negative.

The reaction after I announced these problems are typical, really. Lots and lots of nice people:

One asshole:

And here I am giving that one asshole almost as much attention as I do all those lovely people, but that is unfortunately how my brain works at this point. I think I’ve been somewhat worn down over the years…

But before I end this announcement, I forgot to mention the people who’ve helped run bits of Gmane. While have been doing all the sysadmining and the programming on the main site, in the beginning there were quite a few who helped out with approving subscriptions and handling spam and stuff, and I can’t mention everybody.

But I’ll give a shout out to the people who’ve done stuff until the very end: Olly Betts, who ran the search engine that Gmane used after the one I wrote proved not to scale well enough. Adam Sjøgren, who did the day-to-day mailing list approval thing almost single-handedly for years and years. And Steinar Bang, who handled spam reports for many years.

Thank you, thank you.

The End of Gmane?

In 2002, I grew annoyed with not finding the obscure technical information I was looking for, so I started Gmane, the mailing list archive. All technical discussion took place on mailing lists those days, and archiving those were, at best, spotty and with horrible web interfaces.

The past few weeks, the Gmane machines (and more importantly, the company I work for, who are graciously hosting the servers) have been the target of a number of distributed denial of service attacks. Our upstream have been good about helping us filter out the DDoS traffic, but it’s meant serious downtime where we’ve been completely off the Internet.

Of course, there are ways to try to mitigate all this: I’m moving the Gmane servers off of my employer’s net, and I’m putting Cloudflare in front of the Gmane web servers.

But I ask myself: Is this fun any more?

Running a mailing list archive means, of course, that people want stuff removed from the archive, so I’ve apparently been sued in India (along with Google and Yahoo) (and I’m never going there: I might be sentenced. I don’t know). And I’m the Internet Help Desk, which is nice, but confusing. And all the threats of “legal action” are, well, something.

And now the DDoS stuff, which I have no idea why is happening, but I can only assume that somebody is angry about something.

Probably me being a wise ass.

So… it’s been 14 years… I’m old now. I almost threw up earlier tonight because I’m so stressed about the situation. I should retire and read comic books and watch films. Oh, and the day job. Work, work, work. Oh, and Gnus.

I’m thinking about ending Gmane, at least as a web site. Perhaps continue running the SMTP-to-NNTP bridge? Perhaps not? I don’t want to make 20-30K mailing lists start having bouncing addresses, but I could just funnel all incoming mail to /dev/null, I guess…

The nice thing about a mailing list archive (with NNTP and HTTP interfaces) is that it enables software maintainers to say (whenever somebody suggests using Spiffy Collaboration Tool of the Month instead of yucky mailing lists) is “well, just read the stuff on Gmane, then”. I feel like I’m letting down a generation here. And despite what I rambled about in that paragraph up there, I’ve had many fun interactions with people because of Gmane. And lots and lots and lots of appreciative feedback over the years.

*sigh*

But there’s The Mail Archive. Those guys are doing a good job. If The Mail Archive had been as good in 2002 as it is now, I probably wouldn’t have started Gmane.

I’m open to ideas here. If somebody else wants to take over the concept, I can FedEx you a disk containing the archive (as an NNTP spool). I’ve written a lot of software for Gmane, but it’s all quite site specific and un-documented. And the web interface was written in, like, 2004, so it’s way way way un-Web 2.0-ey and shiny. You’re probably better off implementing this stuff from scratch.

Oh, and along with the spool you’ll get the gmane.conf file which has the mailing list->NNTP mapping.

I can’t really recommend the job, though. It sucks.

[Update: See this comment.]

Further Legal Developments

As part of running the mailing list archive Gmane, I’m asked to remove messages from the archive from time to time, which I do.  (Although not extremely promptly, since I’m lazy.)  That’s fine, but some people seem to think that having a lawyer send the message to me is more effective than just sending an email directly.  And it never is, because the lawyers never say exactly what it is that they want to have removed.  There are several hundred million messages archived on Gmane, so…

It’s been a while since the last time, but yesterday I got a letter from a lawyer again.  This time, it’s Italian, and seems to involve…  Berlusconi?

Your guess is as good as mine.

 DSC01415

DSC01416

DSC01417

DSC01419

DSC01420

DSC01421

As far as I can tell, they want me to remove some message from (for instance) a Mozilla group, but they don’t list specific URLs they want to have removed, either…

Anybody want to ferret out what URLs they mean and guess what the case is really about?

 

New Gmane SSDs

The Gmane news spool is 97% full, so I either had to delete some Gwene stuff, or buy more SSDs. Image

I bought more SSDs.  The current setup is 5x 512GB Samsungs in RAID5.  I bought 5x 1TB while in the US, so that gives us 2x the current size in RAID5, which should be enough for the next uhm five years? or so?

But the problem is how to do the switchover.  The last time it was pretty seamless.  I set up a new, spiffy machine with a spiffy hardware RAID controller. (See my secret diary for details.) Then synced all the articles over, and swapped some IP addresses at the end.

I really don’t want to buy another spiffy RAID controller this time.  But if I’m reusing the current hardware, there’s going to be downtime.

Here’s the plan:

1) Sync the spool over to an external SATA disk.

2) Take the server down, swap in the new SSDs, set up the RAID.

3) Rsync the articles from the external SATA disk to the RAID.

5) Profit!

2-5 will realistically take a day or so, with Gmane being totally dead while this is happening.  I think.

Hm…  or I could point the spool to the external disk while doing 3.  In read only mode.  Then there should only be a few hours downtime while I’m doing the RAID rebuild.  Hm.  Yes, I think that sounds doable…

So a few hours complete deadness, and a day or so in read-only mode.

But it’s a bit scary doing it this way.  If I were doing a completely separate new server, there would always be an easy way to roll back if things don’t work…