Random Thoughts

I was blathering on yesterday about how hard it is to screenshot a web page these days. I mean programmatically, because my use case is to make links on a blog have the same life span as the blog itself — taking screenshots in your browser manually is usually pretty easy.

But if you use, say, shot-scraper from a non-US IP address you usually get something like the above. Which sucks.

Today, though, I though — there are things like the Ublock “annoyances” list. For instance here we have some nice lists made to remove annoying things like cookie banners (and other modals). Why can’t shot-scraper use those lists, huh?

Why not indeed:

So what I did was I forked shot-scraper to add a syntax to load a Javascript file:

def _evaluate_js(page, javascript):
    try:
        if javascript.startswith("file "):
            path = javascript[5:].strip()  # everything after "file "
            return page.add_script_tag(path=path)
        else:
            return page.evaluate(javascript)
    except Error as error:
        raise click.ClickException(error.message)

And then I just generated a JS file that uses all those selectors to remove elements, and there you go. Kinda hacky, but…

The code lives in the Emacs WordPress library. But somebody should take this idea and integrate it with shot-scraper proper — a switch like --remove-annoyances that just downloads those block lists and uses them would be ideal.

It’s not like this is a panacea, though, because there’s so many other stumbling blocks. Like:

You’d have to work harder to get around those… And:

Some sites have so obfuscated HTML that it’s nigh impossible to just remove the offending elements.

But still! While this doesn’t work on all sites, it works on a whole lot of them, so that’s some progress, at least.

Hack off:

Hack on:

See? Better.

I’ve talked about this before, but to recap: As someone who does quite a bit of research into somewhat obscure topics on the web, there’s nothing as annoying as when you read an old web page that says something like “and you can read that really interesting interview on this page“, and then you follow that link, and discover that that site disappeared a decade ago.

And the Wayback Machine didn’t archive it.

So, ideally, whenever you link to something, a copy of what you’re linking to should be stored on your own site — so what you’re writing and what you’re linking to has the same lifespan. That’s kinda difficult to do, though — lots of issues with “safely” mirroring a site in a useful manner. But what’s trivial is to do is to screenshot what you’re linking to.

It’s a 90% solution: No, it’s not ideal to read a screenshot of a page instead of the page itself, but it’s a lot better than nothing:

But… Actually taking a screenshot of a web page and then manually uploading it to your blog site would be an insane amount of work. But computers are pretty good at automating stuff, so my Emacs-based WordPress interface does this automatically… as well as it can, because even screenshotting things from your own machine is getting to be pretty hard.

Because not only are there cookie banners and various other blockers, but even “nice” sites like the above somehow feel the need to plaster some modal over the page contents. *sigh* And that’s not the worst, really — there’s so many “anti scraper” tools that trigger for even the most innocent of automatic usages like the above that you may end up being permanently banned if you try to use anything other than the newest of the newest actual real browsers to visit a web site.

It’s not that I blame them — it’s an arms race against out-of-control AI scrapers, but the use cases that are most affected by all of this are use cases like this — the AI scrapers have infinite resources and use residential VPNs and heavy automation to seem like real people, and don’t care one whit one way or the other. Well, I’m guessing that playwright (which is what I’m using for this) will come with an LLM extension soon to click through all the modals, right?

[Slight digression: While typing this blog post, it occurred to me that Cloudflare had announced APIs for doing stuff like screenshots, so I wondered whether they’d come up with something fun in this area. So I pointed that API at an imdb page and viola:

A big fat nothing, because imdb uses the Big Amazon Firewall to block everything from data center IPs and browsers that don’t pass a human-like check.]

So I don’t really have a solution here for all of that. I just wanted to mention that I’ve cleaned up the code to actually display the linked screenshots and made it into a WordPress plugin. (Hover over that Microsoft Github link to see the plugin in action. And possibly click on that thumbnail you get when hovering, too.)

(Note that this isn’t one of those annoying “preview” things that some web sites put on URLs — I find that to be the most annoying thing ever, and totally useless. What you’re seeing here is a screen capture of the linked site taken the same date I posted this post — so you’re seeing exactly what I linked to when I linked to it.)

Unfortunately, there is no way to do automatic screenshots from the server — Cloudflare blocks/challenges all access from known data center IPs, so that’s just not feasible. So if you want to do something like this, you have to find your own way to get the screenshots of what you’re linking to.