Ten years ago, I got fed up with various ad hoc approaches for looking at the previews for American comics books, so I wrote a web site called Goshenite. (Because that’s an emerald, like diamond. Get it? Get it?)
(I’d previously used an Emacs-based mode that was based on scraping the Mile High Comics data, but that didn’t work reliably for various reasons.)
So that’s all well and good, but then Diamond Comics had a lil ol bankruptcy, and I don’t think anybody knows what’s going to happen exactly. In any case, most of the bigger publishers have already fled to Lunar and Penguin Random House, so that means that I had to update Goshenite now.
Which I’ve done.
Scraping Lunar was quite easy, but those Penguins have a very Random House. I don’t know what their thought processes are exactly — they’re a book distribution company, so you’d have thought they’d be interested in getting the information about those books spread as widely as possible, right? So that people can buy them?
So there’s this helpful-looking menu where you can download an Excel file with all the books for the month. Perfect! Brilliant! So that’s just a link, right?
No, that’s some Javascript monstrosity that apparently uses encryption to create the POST you have to make (note “load_nonce” there) to get the Excel file, and then it takes ten seconds to apparently create the Excel file. And it’s got replay protection, so you can only download “it” once. Very dynamic, very modern.
Again, I don’t know the thought process behind this — perhaps they just have a CMS that does all this stuff automatically, and this was the easiest way for them to create a link to an XML file? Or perhaps they have some insane person working for them who’s heard the phrase “information security” and then we get what we see?
In any case, if a human can see something, a computer can download it, so a shot Selenium script later, I’ve got an automatic solution for updating Goshenite with the PRH data too, so…
Enjoy? Ou pas.
There may be some repeated publishers (that are available from several distributors), and the publishers may not have exactly the same names, but whatchagonnado.
Here’s the link to Goshenite again, just in case you missed it.