70% Of Drivers Really Are Above Average

You see it all the time — “70% of drivers think they’re above average” used as an example at how stupid people are. Often mixed with some confused Dunning-Krueger references.

But the thing is: Most people observe the other drivers around them and come to the conclusion that they drive better than the drivers around them. That they are better drivers. And they’re right.

The apparent contradiction here is that there is no universal norm for what makes a “good driver”.

We all have this one acquaintance who’s really into cars. She parks the car doing a j-turn, and she knows how to do an ollie, a lazy eight and a double choctaw. But the thing is, she’s constantly involved in collisions with other cars, and it’s always the other guy’s fault. She can prove it! She’s been in so many court cases and has never been found guilty!

And we all have that other acquaintance who’s not into cars, and she’s never hurt anybody. She’s not even had a little fender bender. But the thing is, you’ve been a passenger in her car, and it’s harrowing. She always goes 10km/h under the speed limit, and you can tell that all the other drivers are mad at her. She can’t even do parallel parking!

Both of these people think they drive better than the average driver… and they’re right. But they use completely different metrics for “what’s a good driver”.

If you’re now muttering “both of those drivers are insane! they shouldn’t be allowed on the street! now, on the other hand, I drive really well and I never…”

Then stop that train of thought and be enlightened.

Even more deduplication on kwakk.info

As I’ve nattered on about before, I started including text pages from comics in the comics ‘zine search engine, and this has some unique problems. I’m just dumping hundreds of thousands of comics into the grinder, picking out the text pages, and then OCR-ing them. But many comics have been scanned several times, and many include editorial pages that are more or less identical across several titles.

So I’m now running the pages through a text-based deduplicator… but it used a “quick” (FSVO) OCR, Tesseract, which behaves horribly on non-standard pages like the above, and there are now five copies of that page in the search engine. Which is just so annoying when trying to actually find stuff — you have to wade through duplicates.

Which means that I had to do something more, and that’s now in production: After doing the real OCR, with Surya, which is great even with 30 degree text and badly scanned pages on colourful backgrounds, I’m running an extra step and deduplicating again based on the output from that.

*phew*

The pipeline is basically:

1) First get rid of byte-identical CBX files. This gets rid of about 20% of the comics.

2) Identify text pages and delete all other pages. This gets rid of about 98% of pages.

c) Identify byte-identical pages and delete them. This gets rid of 15% of the text pages.

e) Run Tesseract OCR over all the pages and keep only the first “instance” of each duplicate page. This gets rid of a further 20% of the pages.

ii) Remove credits pages and the like — i.e., every page that has less than a hundred words. This gets rid of about 15% of the remaining pages.

II) Finally, do the new Surya-based OCR deduplication, which gets rid of about 5% more of the pages.

Of course, all of these numbers are trending upwards as more and more comics are added. And most of these steps may have false positives — it’s mostly probabilistic, but I’ve tried to be on the lookout for false positives. *crosses fingers*

There are also so many complete runs of classic comics in the index now that I think there’s probably a complete coverage of classic “hype” pages now, for those interested in that. For instance, if you want to do research on Wally Wood mentions on the Bullpen Bulletin pages, that should be possible.

Annoyingly enough, Surya isn’t able to parse that “On the Ledge” logo… Vertigo is so edgy…

Anyway. I think I’ve futzed around with this thing enough for a while.

How come translation apps aren’t getting better?

I’ve been learning French for three years now. When I’m reading French comics, I always have an Ipad Mini with Google Translate going, so that I can just wave it in front of the page when I encounter a word I don’t know (and can’t surmise from the context).

(I’ve got a strap attached to the back of the Ipad so I just keep a hand in there, ready for waving…)

Back when I started using this, er, methodology, I was pretty impressed by how good Google Translate was… but three years later, it’s still just… that. It’s pretty good! But it has absolutely not gotten any better, and that’s just surprising. The evolution in LLM-assisted computing has been incredible, and you’d think this would be something that they’d funnel into translation, right?

But it really doesn’t seem like it — nothing seems to have changed over the past three years.

Oh, the guy isn’t talking about her condom, but the top of her convertible. And what’s coming is the rain. See, it’s pluvieux? (Or is that pluvieuse? I forget.)

(Yes, “capote” also means condom, so it’s not a wrong translation per se… it’s just… eh…)

But I thought I’d just quickly run through some other programs and see whether they’re better or not, these days.

Apple Translate is, as the last time I tried it, really really bad.

DeepL is on par with Google Translate, but takes 5x as much time — which makes it unusable for my use case, where I just want to wave the Ipad around to get at what a word means. If it’s too slow, it gets in the way of reading.

Translate Now is risibly bad — the UX is the worst I’ve encountered and the translations are worse than Google Translate.

So… the apps aren’t very good, but what about just asking Claude?

Claude is perfect, if given the image. I wonder whether it’d say the same with just the text? (Presumably that’s all these other platforms are getting…)

Yup, it got it with just the text.

See? Using an LLM here would be so nice. That’s what Translate Now says they offer, but the app is so bad that… who knows?

Using an LLM here is probably 1000x more expensive than whatever the traditional apps do, so I do realise there’s some restrictions here, but…

It does seems like there’s a market opportunity here. C’mon people. Make it happen! Make the app! C’mooooon!

June Music

Music I’ve bought in June.

Let’s see… I somehow got quite a lot of stuff in June without really trying.

Faenskap for alltid

The album from Rosa faenskap is really good. I’ve seen them live a couple times, and it’s great hearing that they’ve managed to capture that explosiveness on tape.

CxBxT-Hoshi o Atsumete

I got two new albums from an old favourite — Tujiko Noriko. She’s a lot more ambient these days than in the old and wild days, so there’s a lot of tunes that sort of just are… there… but in a pleasant way. But the album under the CxBxT name is more crunchy.

Astrïd and Sylvain Chauveau Cover Songs Originally Sung by Women Singers / Video Games

Astrïd and Sylvain Chauveau Cover Songs Originally Sung by Women Singers is very nice, but perhaps a bit one note? I’ve always liked Sylvain Chauveau, but I’m unfamiliar with Astrïd — I think I’ll give them a go…

Horse Lords - Eureka 378-B / Brain of the Firm [Official Video]

The new Horse Lords album is even stranger than the earlier albums. They’re sampling people doing what sounds like shape note singing? It’s cool.

Richard Dawson & Circle - Lily (Official Video)

I’m slowly buying my way through Richard Dawson’s back catalogue — I love his stuff, but I’m really trying to pace myself. I remember when I got into Can a long time ago, and I bought, like, ten albums from them all at once. With the result that they just sat there like an undigested block, and I never listen to those albums. So when I discover somebody with a big backlist, I try to never buy more than one album per six months or something.

Seems to be a methodology that works for me, but it requires some self control, which I just don’t have a lot of.

Anyway, this album (with Finnish band Circle) is great, too.

Speaking of old stuff, I got another album by Wang Chung, because I loved the To Live and Die in L A soundtrack.

And it’s good, but it’s not as good.

Joan As Police Woman - I Defy (Real Life Anniversary Session)

And another artist I’ve been buying all albums from — but now I’m all caught up. So then she goes ahead and releases a re-recorded version of her first album. It’s pretty good. A couple versions are perhaps better than the first version around? But many aren’t. In any case, it’s nice.

Tara Clerkin Trio - 'Lazy Daisy' (Official Audio)

Tara Clerkin Trio has gotten a lot of attention lately, so I bought the album like two days ago. I haven’t gotten the chance to listen to it more than a couple times, but it seems pretty nice? But a bit surprising that it’s getting so much attention?

Charli XCX - What I Like [Official Video]

And yet another artist I’m buying my way through their backlist, but slowly: Charli XCX! But I have to say that I think the True Romance album is a bit… eh… not my thing.

And… that’s all I had to say about the June albums.

Text Pages From 100,000 Comics

I did a test run of adding text pages from comics to the comics magazine search engine a couple weeks ago, but I wasn’t quite sure whether it was a good idea. Or whether my pipeline was good enough.

And after getting some feedback, it seems like the answers are “yes” and “not quite”.

So I’ve fixed some problems: The detection of duplicate pages was really bad, but it’s now better. There are still duplicate pages in there, but getting it totally right just takes a lot of processing. The current algo uses Tesseract to do a “quick” OCR, and then it compares the texts to find the distances, and then excludes the one that are too similar.

The problem here is that Tesseract, while being a good OCR as far as traditional OCRs go, it does get a goodly percentage of the words wrong, so if the scans aren’t pristine, you’ll get “different words” even if it’s the same page that’s been scanned twice.

For the search engine on kwakk.info, I’m using the Surya OCR engine, which is much, much more precise, but is also much, much slower. I could then feed the data from that back into the duplicate detection thing, but hiudhfiudsahgiushguishfiudsahf. Not worth it.

Other improvements: I’m removing pages that have less than a hundred words, because those are usually just credits pages and the like.

One thing I don’t have a fix for: I want editorials and letters pages and perhaps text-filled ads, but I don’t really want text pages that are part of the comics — they happen sometimes because some comics writers are smartypants (like the classic Howard the Duck issues by Steve Gerber). But the only way to weed those out would be to use an LLM classifier, and that just seems like it would be too slow. And too much work to set up.

So the upshot here is that I think the data set now is pretty usable, so I’m including it in the magazine and fanzine and “everything” searches.

Oh, and while the vast majority of the comics included so far seem to be mainstream books, it looks like a collection of Cerebus comics has snuck in, at least.

You’re welcome I’m sure.

Oh, and about that number… 100K comics. I wondered — is that a lot or is it a little? How many comic books have been published in the US, anyway? (The data set is 90% American comics.) After googling a bit, it seems that nobody knows, of course, but a ballpark number may be between 1M and 2M books… which means that 100K is much more than I thought! It’s between 5 and 10%! Huh.

Possibly.

Anybody have a better ballpark number?