Gwene Is Up Again, Too

After a month-long hiatus after the troubles, Gmane’s sister web site, Gwene, is back up again, too.

Gwene allows you to sign up RSS feeds via the web site, and then you can read those feeds by pointing your news reader at news.gwene.org.

rememberGwene also used to have a web-based interface to browse the contents, and I may resurrect that at some point, but it didn’t really seem very… useful.  Everything on Gwene is already on the web, and replicating that just seems rather churlish.  But writing it in Node was kinda fun.

Gmane Alive!

A few weeks back, a DDoS was the final straw that broke this Gmane camel’s back, and I took my marbles (as they were) and went home.

I mean, I shut down the web interface of gmane.org.

But now it’s back, and under new management: Yomura Corporation.  Not all things are up yet, but it’s getting there, and looks rather nice.  I mean, it looks kinda like the old site, and nothing’s better than that!  Surely!  Here’s an example article.

Thanks very very much to Martin and Mark for setting this up, and I’m sure they’ll can respond to things on their blog.  And also thanks to everybody else who offered to take over Gmane.

Gmane has been archiving mailing lists as usual during the web interface holiday, and those articles aren’t available on the new web interface yet, but that will be fixed after a while.

And while I have all yourn attention, I want to write an apology:

While the DDoS was the trigger here for me to abandon Gmane, it’s been brewing for a while, and I could have handled the whole situation better. I started Gmane in 2002, and I spent a lot of time on it the first few years. But the last five years or so just haven’t been fun. I’ve done virtually no new programming on the site in that period, and that is, after all, the fun part.

In addition to the death threats and the people who want to sue me, etc, it’s also been a constant low-level source of anxiety. Oh, there goes a server… let’s build a new one… oh, now the RAID array is growing full, let’s buy more disks…

So I should have gotten rid of Gmane years ago. I mean: Found someone else to take it over in an orderly fashion, instead of this panic attack.

The reason I didn’t is probably partly because I didn’t think anybody would (and especially not the NNTP bits, but I was completely wrong), but also because I just didn’t want to let go. I still have ideas for what I wanted Gmane to be, but I didn’t have the time… or really, I always have time. I mean, it’s all choices. I could have written a new Gmane web interface instead of watching a few movies, couldn’t I?

And there was always some fun bits about Gmane. I could regale people in bars with drunken stories about being sued in India, for instance. Not many people I know can!

And the thing is, most people in the world are great people, and I wanted to provide this service to these people. I know I kvetch a lot, but less than one in a hundred Gmane-related interactions were in any way negative.

The reaction after I announced these problems are typical, really. Lots and lots of nice people:

One asshole:

And here I am giving that one asshole almost as much attention as I do all those lovely people, but that is unfortunately how my brain works at this point. I think I’ve been somewhat worn down over the years…

But before I end this announcement, I forgot to mention the people who’ve helped run bits of Gmane. While have been doing all the sysadmining and the programming on the main site, in the beginning there were quite a few who helped out with approving subscriptions and handling spam and stuff, and I can’t mention everybody.

But I’ll give a shout out to the people who’ve done stuff until the very end: Olly Betts, who ran the search engine that Gmane used after the one I wrote proved not to scale well enough. Adam Sjøgren, who did the day-to-day mailing list approval thing almost single-handedly for years and years. And Steinar Bang, who handled spam reports for many years.

Thank you, thank you.

The End of Gmane?

In 2002, I grew annoyed with not finding the obscure technical information I was looking for, so I started Gmane, the mailing list archive. All technical discussion took place on mailing lists those days, and archiving those were, at best, spotty and with horrible web interfaces.

The past few weeks, the Gmane machines (and more importantly, the company I work for, who are graciously hosting the servers) have been the target of a number of distributed denial of service attacks. Our upstream have been good about helping us filter out the DDoS traffic, but it’s meant serious downtime where we’ve been completely off the Internet.

Of course, there are ways to try to mitigate all this: I’m moving the Gmane servers off of my employer’s net, and I’m putting Cloudflare in front of the Gmane web servers.

But I ask myself: Is this fun any more?

Running a mailing list archive means, of course, that people want stuff removed from the archive, so I’ve apparently been sued in India (along with Google and Yahoo) (and I’m never going there: I might be sentenced. I don’t know). And I’m the Internet Help Desk, which is nice, but confusing. And all the threats of “legal action” are, well, something.

And now the DDoS stuff, which I have no idea why is happening, but I can only assume that somebody is angry about something.

Probably me being a wise ass.

So… it’s been 14 years… I’m old now. I almost threw up earlier tonight because I’m so stressed about the situation. I should retire and read comic books and watch films. Oh, and the day job. Work, work, work. Oh, and Gnus.

I’m thinking about ending Gmane, at least as a web site. Perhaps continue running the SMTP-to-NNTP bridge? Perhaps not? I don’t want to make 20-30K mailing lists start having bouncing addresses, but I could just funnel all incoming mail to /dev/null, I guess…

The nice thing about a mailing list archive (with NNTP and HTTP interfaces) is that it enables software maintainers to say (whenever somebody suggests using Spiffy Collaboration Tool of the Month instead of yucky mailing lists) is “well, just read the stuff on Gmane, then”. I feel like I’m letting down a generation here. And despite what I rambled about in that paragraph up there, I’ve had many fun interactions with people because of Gmane. And lots and lots and lots of appreciative feedback over the years.

*sigh*

But there’s The Mail Archive. Those guys are doing a good job. If The Mail Archive had been as good in 2002 as it is now, I probably wouldn’t have started Gmane.

I’m open to ideas here. If somebody else wants to take over the concept, I can FedEx you a disk containing the archive (as an NNTP spool). I’ve written a lot of software for Gmane, but it’s all quite site specific and un-documented. And the web interface was written in, like, 2004, so it’s way way way un-Web 2.0-ey and shiny. You’re probably better off implementing this stuff from scratch.

Oh, and along with the spool you’ll get the gmane.conf file which has the mailing list->NNTP mapping.

I can’t really recommend the job, though. It sucks.

[Update: See this comment.]

Further Legal Developments

As part of running the mailing list archive Gmane, I’m asked to remove messages from the archive from time to time, which I do.  (Although not extremely promptly, since I’m lazy.)  That’s fine, but some people seem to think that having a lawyer send the message to me is more effective than just sending an email directly.  And it never is, because the lawyers never say exactly what it is that they want to have removed.  There are several hundred million messages archived on Gmane, so…

It’s been a while since the last time, but yesterday I got a letter from a lawyer again.  This time, it’s Italian, and seems to involve…  Berlusconi?

Your guess is as good as mine.

 DSC01415

DSC01416

DSC01417

DSC01419

DSC01420

DSC01421

As far as I can tell, they want me to remove some message from (for instance) a Mozilla group, but they don’t list specific URLs they want to have removed, either…

Anybody want to ferret out what URLs they mean and guess what the case is really about?

 

New Gmane SSDs

The Gmane news spool is 97% full, so I either had to delete some Gwene stuff, or buy more SSDs. Image

I bought more SSDs.  The current setup is 5x 512GB Samsungs in RAID5.  I bought 5x 1TB while in the US, so that gives us 2x the current size in RAID5, which should be enough for the next uhm five years? or so?

But the problem is how to do the switchover.  The last time it was pretty seamless.  I set up a new, spiffy machine with a spiffy hardware RAID controller. (See my secret diary for details.) Then synced all the articles over, and swapped some IP addresses at the end.

I really don’t want to buy another spiffy RAID controller this time.  But if I’m reusing the current hardware, there’s going to be downtime.

Here’s the plan:

1) Sync the spool over to an external SATA disk.

2) Take the server down, swap in the new SSDs, set up the RAID.

3) Rsync the articles from the external SATA disk to the RAID.

5) Profit!

2-5 will realistically take a day or so, with Gmane being totally dead while this is happening.  I think.

Hm…  or I could point the spool to the external disk while doing 3.  In read only mode.  Then there should only be a few hours downtime while I’m doing the RAID rebuild.  Hm.  Yes, I think that sounds doable…

So a few hours complete deadness, and a day or so in read-only mode.

But it’s a bit scary doing it this way.  If I were doing a completely separate new server, there would always be an easy way to roll back if things don’t work…

 

The Internet Help Desk

 About once a month, I get really drunk and answer the Gmane email.  90% is “please remove my

message”, which I do.  If it’s from the person who asks to have it removed.

The other 10% are kinda … assorted.  I think this is of interest to you.  All of you.  So here I present a [redacted] extraction of those 10%s from tonight.  I don’t mean to be mean.  But c’mon.

Prepare to be puzzled.

—————

The below companies are using zeromq in the field of “financial
services”, “game devolpment”, “embedded systems”, ” academic research”,
“aeronatics”.

Microsoft
Zynga
Barclays

But we require the information about for what purpose these all above
companies ar using zeromq.[for example we are planing to use as a
message broker for sending messages between two clients] can you please
provide these informations as soon as possible. I need very early. I ll
be waiting for your reply.”

———–

Search title gets encoded as UTF-8/ISO-8859-1.  You want to wonder if
this is a bug.

———–

I’m building an AngelList profile, and need your help.  Can you do me
a favor and follow me by clicking here:

———–

I’m trying to get in touch with [redacted]. Do you have an email
address or his current contact information? Best Regards,
[redacted] Investigator

———–      

Ich war mir über die Veröffentlichung dieser Email nicht bewusst und
bitte Sie daher meine Personenbezogenen Daten aus dem Beitrag zu
löschen

———–

There’s a message with subject “Google”.  The body says “Cam on”.

———–

We have learned that unauthorized individual(s) secured a link
promoting our website [redacted].com on one or more of your web
properties.

———–

it got returned with this message.  my message is attached here for
you to  see. why was it returned?  i need for aall to know about
[redacted], and its pro capabilities,  and for those who use
[redacted], knocking it down,  saying its not production ready,are
mistaken.

———–

I got your contact in this post
http://permalink.gmane.org/gmane.comp.handhelds.phonegap/%5Bredacted%5D I wonder
if you could create a new wifi network, entering the ssid and password
in the app and creating the network automatically.

———–

I need your assistance if you can. I accidentally sent a message to
the [redacted] email list last night with a live password in
it. The password is hardcoded in to dozens of machines, so changing it
would be… inconvenient, to say the least. I don’t yet see the message
on your archive page, so I can’t provide the URL link, but I wanted to
see if you could remove the message before it is processed and posted.

———–

This week, due to your GNAME postings, I just suffered the loss of an
important job opportunity, and contract because an individual obtained
the private communications I had not intended for publication by
[redacted]. After the list error I wrote several people on that list
ridiculed me.

As I informed Mr. [redacted], there are treatises between the Nations of
the United States and yours, I believe, Norway, which mandate
compliance with the California State and Federal privacy laws
respectively.

I have sent him as I will probably have to send you a Cease and Desist
Notice, in lieu of a possible Federal Civil Suit for Injunctive Relief,
and damages. If any internet content states that a person in the United
States does not want his personal email posted on the internet, you
must abide by the meaning of the content.

———-

Though the organization [redacted] expressly prohibits the publication
of any archived communications, my personal and private communication
was published on the internet, via GMANE.

Though Mr. [redacted] alleges that he has no liability, the origin of
the privacy violation originated from his negligent management of his
list.  I have notified you of the ANTI SPAM ACT of the US Government
which can be enforced due to the treaties between the US Government and
Norway. You have failed to respond in kind.

The penalties that can be adjudged against you due to your reckless
and knowing violation of the aforementioned act, and my privacy rights
under the California Constitution, the highest law in the land for any
one person living in the State of California, are very high.

The posting cost me a $76K job opportunity.

Monetising

Ever since Google Adwords dumped Gmane for reasons, Gmane hasn’t had any income.  It doesn’t really matter that much, but I find it annoying.

The Gmane web site has some traffic.

About half a million page views per day.  Surely there’s $$$ in that.  Not that I like ads.  I think ads are yucky.

I signed up for Google Webmaster Tools to see if it had anything interesting to say.

I’m going to digress a bit.  Google sucks.  Has anybody noticed?  They make a bunch of things that are barely good enough, and then they don’t improve, in general.  Take the Webmaster Tools signup process.  Please.  Gmane has a bunch of different sub domains, and they all have to be signed up separately.  Geez.

So I signed up them all, which involves putting a special file in a special file on all the domains, which was kinda complicated, since none of the domains actually map unto actual files.

But I got it done, and then removed the hacks to make it work.

Uh-oh.  If the files aren’t there, then they’re “unverified” after a while, so I stop getting stats.

*sigh*

Anyway, I left the file in place for blog.gmane.org (by accident), so I’ve got some stats there:

So, er…  There are 15K queries.  Which are…  uhm…  distinct things that people have asked and which has led to Gmane?  Perhaps?  And there are 4M “impressions”.  “Displaying 60K”.  Which means…  Ok, I had no idea, so I binged.

After fifteen minutes, I’m still not sure, but some people seem to say that “impressions” means that Gmane was included in the search result on some page or other.  The page displayed, perhaps?  And “displaying 60K” is a mystery.  It might be the things included in the list below, but I don’t know why they’re only displaying 60K, or what the criteria is.

Then there’s “Clicks 200K”.  So 200K people clicked on a link to Gmane?  Again with the mysterious “Displaying”.

Classic Google information quality:

I love the Y axis here.  Choosing 70K as the scale here instead of the more traditional 50K is a daring, avant garde choice.  Or perhaps they just don’t care.

Anyway, interesting as all this may be, it doesn’t really give me any money.  So I signed up with the Amazon ad network. 

They have a wonderful product called “omakase”, where you just put the ad JS into your page, and then they spider your page and put relevant stuff on your page.

Money will soon ensue!

*a month passes*

So let’s look at the relevant ads on gmane.linux.kernel now:

Amazing.  I didn’t know that so many Linux kernel hackers were into Jimmy Buffet and Jay-Z!

But looking at the logs, I can’t really see any activity from the Amazon omakase spide, so perhaps it’s not so surprising after all.

*sigh*

Anyway, it’s not that this is that important, but it’s…  annoying.