Random Thoughts

This blog has been hosted on WordPress.com for many a year. It has, all in all, been a very pleasant experience: It feels like the uptime has been at least 110%, and most everything just works.

The problems with using that solution is that it’s very restrictive. There’s so many little things you just can’t do, like adding Javascript code (for which I’m sure many people are grateful), or customising the CSS in a convenient way.

I’ve worked around the shortcomings of the platform, but the small annoyances have piled up, and this weekend I finally took the plunge.

The reason for doing it now instead of later was that WordPress.com seemed to experience a hickup a couple of days ago, and I thought that instead of bugging support with the problem, I’d just take it as an opportunity to get moving. The problem was that the admin pages suddenly started taking 15 seconds to load. I checked it out in the browser debugger, and it was the initial “GET /” thing that took 15.something seconds, but only if I was logged in. So they obviously had an auth component that was timing out, and falling back to a backup thing (and it’s been fixed now).

But I clicked “export”, created a new VM at DigitalOcean, and got importing.

And… it failed. It got a bit further every time, downloading all the media from the old blog, but then failed with “There has been a critical error on your website. Please check your site admin email inbox for instructions.”.

After doing that for about ten times (and no email), I checked the export XML file, and what did I find?

*sigh*

So I got a new export file (after waiting 15 seconds), and ran the import again… and it failed again the same way. So that wasn’t the problem after all?

I blew the VM away, started from scratch again, and this time skipped doing the import of the media, and that worked perfectly.

To do the media, I just scripted something do download all the images, and then I rsynced it over to the new instance. Seems to work fine, even if the images aren’t in the “media library” of WordPress, but I never cared about that anyway…

It’s even possible to copy over subscribers and stats from the old WordPress.com instance, but that requires help from the Automattic support people. And I’m flabbergasted at how efficient they are: I had two requests, and each time it took them less than five minutes to do the request and get a response. I’ve never seen customer support, I mean Happiness Engineering, that efficient before; ever. It almost made me regret doing the entire move to self-hosted blogging…

Anyway. This is a test! If this post is posted, the new WordPress instance works.

Allegedly, 30% of all web pages are now WordPress. I’m guessing most of these WordPress sites aren’t typical blog sites, but there sure are many of them out there.

Which makes it so puzzling why Google and WordPress don’t really play together very well.

Lemme just use on of my own stupid hobby sites, Totally Epic, as an example:

OK, the first hit is nice, because it’s the front page. The rest of page one in the search results is all “page 14”, “category” pages and the like, none of which are pages that anybody searching for results are interested in.

The worst of these are the “page 14” links: WordPress, by default, does pagination by starting at the most recent article, and then counts backwards. So if you have a page length of five articles, the five most recent articles will be on the first page, then the next five articles are on “page 2”, and so on.

You know the problem with actually referring to these pages after the fact: What was once the final article on “page 2” will become the first article on “page 3” when the blog bloviator writes a new article: It pushes everything downwards.

So when you’re googling for whatever, and the answer is on a “page 14” link, it usually turns out not to be there, anyway. Instead it’s on “page 16”. Or “page 47”. Who knows?

Who can we blame for this sorry state of affairs? WordPress, sure; it’s sad that they don’t use some kind of permanent link structure for “pages”. Instead of https://totally-epic.kwakk.info/page/5/, the link could have been https://totally-epic.kwakk.info/articles/53-49/; i.e., the post numbers, or https://totally-epic.kwakk.info/date/20110424T042353-20110520T030245/ (a publication time range), or whatever. (This would mean that the pages could increase or shrink in size if the bloviator deletes or adds articles with a “fake” time stamp later, but whatevs?)

Can we also blame Google? Please? Can we?

Sure. There’s a gazillion blogs out there, and they basically all have this problem, and Google could have special-cased it for WordPress (remember that 30% thing? OK, it’s a dubious number) to rank these overview pages lower, and rank the individual articles higher. Because it’s those individual pages we’re interested in.

This brings us to a related thing we can blame Google for: They’re just not indexing obscure blogs as well as they used to. Many’s the time I’m looking for something I’m sure I’ve seen somewhere, and it doesn’t turn up anywhere on Google (not even on the Dark Web; i.e., page 2 of the search results). Here’s a case study.

But that’s an orthogonal issue: Is there something us blog bleeple can do to help with the situation, when both Google and WordPress are so uniquely useless in the area?

Uneducated as I am, I imagined that putting this in my robots.txt would help keep the useless results out of Google:

User-agent: *
Disallow: /author/
Disallow: /page/
Disallow: /category/

Instead this just made my Google Search Console give me an alert:

Er, OK. I blocked it, but you indexed it anyway, and that’s something you’re asking me to fix?

You go, Google.

Granted, adding the robots.txt does seem to help with the ranking a bit: If you actually search for something now, you do get “real” pages on the first page of results:

The very first link is one of the “denied” pages, though, so… it’s not… very confidence-inducing.

Googling (!) around shows that Google is mostly using the robots.txt as a sort of hand-wavy hint as to what it should do because the Calironia DMV added a robots.txt file in 2006.

It … makes … some kind of sense? I mean, for Google.

Instead the edict from Google seems to be that we should use a robots.txt file that allows everything to be indexed, but include a

directive in the HTML to tell Google not to index the pages insead.

Fortunately, there’s a plugin for that. But googling for that isn’t easy, because whenever you’re googling for stuff like this you get a gazillion SEO pages about how to get more of your pages on Google, not less. Oh, and this plugin seems even better (that is, it allows you to control what pages to noindex more pretty well).

So I added this to that WordPress site on March 5th, and I wonder how long it’ll take for the pages in question to disappear from Google (if ever). I’ll update when/if that happens.

Still, this future is pretty sad. Instead of flying cars we have the “Robots “noindex,follow” meta tag” WordPress plugin.

[Edit one week later: No changes in the Google index so far.]

[Edit four weeks later: All the pagination pages now no longer show up in Google if I search for something (like “site:totally-epic.kwakk.info epic”), so that’s definitely progress. If I just search for “site:totally-epic.kwakk.info” without any query items, then they’ll show up anyway, but I guess that doesn’t really matter much, because nobody does that.]