I fired up the machinery to snarf more comics to find more text pages (letters pages, editorials, hype pages etc) for kwakk.info. I had pretty much exhausted my previous sources, so I, er, went to other, even more … diverse sources.
The good news is that the index isn’t 99% super-heroes now. The bad news is that these new sources were even more disorganised than the previous ones, so there may be some junky junk in there now.
There’s now almost 200K comics with text pages indexed. If I understand correctly, that’s between 10 and 20% of all US comics ever published.
Which is nice — there’s complete runs of all the big series, which means that there’s probably complete runs of company-wide editorial pages (like the Bullpen Bulletins etc).
Not everybody who scans comics include all non-comics pages, so there may well be some missing pages like that, but there’s so many complete runs now that I think it’s likely that there’s full coverage anyway.
So while it possible that all those kinds of things are in there, it’s not easy to confirm, since everything is just dumped in there. And some things, like the Vertigo “On The Ledge” pages, can unfortunately not be searched for, since the OCR doesn’t recognise that logo as text… I guess that’s why you use text like that to confuse robots…
Anyway, there’s so many comics that quality control is impossible, but I did find (randomly) that about 2K porn comics had snuck in, so I deleted those, because kwakk.info is a family oriented site. Or something.
There’s also several thousands of non-English language comics in there now, and I’m not sure whether to run some classifier over the texts and remove those… or move them to language-specific sections.
But I did do some quality control, and I found that letters pages like the above were not included because the classifier took that to be a comics page and not a letters page.
The classifier has been tweaked, but this means that I have to re-run the 200K comics through the machinery again, so that’s going to take… a month or so, I guess? But after that, if you have URLs that point to specific pages, they may well change.
So that’s where we’re at.