Fast Music, or: USB Is Weird

I have my music on an USB3 RAID5 consisting of three external disks connected to one of these, which isn’t a bad little computer: It’s has a 1.7GHz i7-3517UE (Ivy Bridge) CPU, so it’s small, but not horribly slow.

But then one of the disks went AWOL and I thought that perhaps it was time to upgrade the disk array. It’s about seven years old, after all, and surely things have gotten better in the meantime.

My main use case, is, well, it’s a file server that I play music off of:

That’s not a very strenuous task: It basically has to feed out FLAC files over NFS faster than my stereo machine (pictured above) can play them, and basically no machine made after 1987 is too slow for that task.

But my Emacs-based music player doesn’t do any caching of metadata, so if I ask it “show me all the 8K records I have in chronological order”, it has to read eight thousand files, and that takes a while if the disk is slow. This is a problem that has grown year by year, of course, so it’s another reason to explore faster disks.

(I mean, I could add a caching system to my music system, but to quote what Leonard Nimoy said in The Empire Strikes Back: “Meh.”)

So I got a couple of USB3 SSDs. Splurge! I connected them up to the Intense PC and started copying things over. I wondered how slow the original RAID was, and it turned out to be 50MB/s, which is very slow, indeed. With the new disks, I should get like, er, more! MORE!

Copying finished, I did indeed get higher speed. 100MB/s. Which is pitiful. The native speed of the SSD should be 500MB/s, but given USB3, it should be slower, but not 20% of the speed.

So after much head-scratching, I noticed that the CPU was pegged to 100% whenever I read intensively from the disks. Is it possible that USB is such a crappy system that a 1.7GHz Xeon CPU from some years ago would be the bottleneck here?

So I extended a USB3 cable to the other server I had in the same closet, which I had bought a month earlier to do the RAID for my film collection:

It has a i5-7260U CPU @ 2.20GHz, so not much difference in Hurtzes, but it’s a 7th gen Intel CPU, and the other machine has a 3rd gen.

And… Wow! 320MB/s reading speed! 3x faster than the older machine, with the same SSDs, USB3 hub and everything.

I quickly rejuggled my setup and made that machine do the /music array, too, and sighed a breath of relief.

Now I can play music six times faster than before! Whoho!

But then!

The RAID went AWOL, always with the same messages about “tag#0 FAILED”, “USB disconnect” and “I/O error” on various /dev/sdx-es.

I first suspected the USB3 hub, so I got a new one… A couple of days later, the same thing. Tried a different USB3 cable (it’s always the cable!); same thing.

Of course, after each time this happens I have to rebuild the RAID, things get inconsistent and stuff.

Finally, I move the USB from the port on the left there to the right…

And two weeks later, still haven’t had a single disk brown-out.

So: The takeaway here is: 1) USB is a janky thing. It’s not quite like SCSI in olden days (no goat sacrifices needed), but it’s janky. 2) If your USB is slow, get a faster CPU.

The good thing about USB setups like this is that, in my experience, once you get them going satisfactorily, they’re pretty stable. Unless you do something crazy like insert a new USB device. Then all bets are off.

Of course, having a machine with room for plenty of SATA disks internally would be better, but I’ve never seen one that’s a) small and IV) allows easy access to disks that have failed and have to be replaced.

But look!

I can now display all the albums from 1975 by the snap of your fingers! If your fingers snap really slowly. But still!

And since the /dvd disks spin down automatically, my computer setup is now 100% without anything mechanical moving around normally, and I can walk past that closet without hearing any humming sounds.

Well, beyond my tinnitus, that is.

The Ever-Shifting Sands of udev.rules

I use Telldus Tellstick to do home automation *cough* I mean control the lights:

It’s an unassuming USB stick that implements a serial interface so that you can talk to it by just sending some strings to it and read the response. Ideal for Linux! Yes!

But.

You do want the device to show up at a default space so that you can find it, and if you have more than one serial USB interface thing, which one of /dev/ttyUSB0, /dev/ttyUSB1 og /dev/ttyUSB2 is it?

udev to the rescue!

And here your woes begin, because every fucking time you upgrade Linux, the fucking udev people change the fucking syntax vaguely and how you fucking refer to fucking devices. Fucking.

In the really olden days you said

KERNEL=="ttyUSB*", SYSFS{idVendor}=="1781", SYSFS{idProduct}=="0c31", 
  MODE="0666", NAME="tellstick"

But then they changed SYSFS to ATTR in the slightly less olden days:

KERNEL=="ttyUSB*", ATTR{idVendor}=="1781", ATTR{idProduct}=="0c31", 
  MODE="0666", NAME="tellstick"

Then they disallowed renaming devices, and instead you add symbolic links to the device:

KERNEL=="ttyUSB*", ATTR{idVendor}=="1781", ATTR{idProduct}=="0c31", 
  MODE="0666", SYMLINK+="tellstick"

Then they changed ATTR to ATTRS:

ATTRS{idVendor}=="1781", ATTRS{idProduct}=="0c31", MODE="0666",
  SYMLINK+="tellstick"

Then they changed what the attributes refer to, so instead of getting this:

[larsi@stories ~]$ ls -l /dev/tellstick 
blrwxrwxrwx 1 root root 7 Feb 25 16:12 /dev/tellstick -> ttyUSB1

you get this:

root@stories:/home/larsi# ls -l /dev/tellstick 
lrwxrwxrwx 1 root root 15 Feb 25 16:27 /dev/tellstick -> bus/usb/001/014

And that doesn’t work, because that’s not a serial USB interface, but a raw USB interface of some kind which can’t be opened like a serial interface.

And, remember, you can’t refer to /dev/ttyUSB*, because that’s the problem you’re trying to solve.

The solution to these problems is this following command:

 # udevadm info -a -n /dev/ttyUSB1

You’ll get output like

 looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/ttyUSB1/tty/ttyUSB1':
    KERNEL=="ttyUSB1"
    SUBSYSTEM=="tty"
    DRIVER==""

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/ttyUSB1':
    KERNELS=="ttyUSB1"
    SUBSYSTEMS=="usb-serial"
    DRIVERS=="ftdi_sio"
    ATTRS{latency_timer}=="16"
    ATTRS{port_number}=="0"

Well, nothing there to distinguish the Tellstick from anything else, to continue down the output…

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0':
    KERNELS=="1-10:1.0"
    SUBSYSTEMS=="usb"
    DRIVERS=="ftdi_sio"
    ATTRS{authorized}=="1"
    ATTRS{bAlternateSetting}==" 0"
    ATTRS{bInterfaceClass}=="ff"
    ATTRS{bInterfaceNumber}=="00"
    ATTRS{bInterfaceProtocol}=="ff"
    ATTRS{bInterfaceSubClass}=="ff"
    ATTRS{bNumEndpoints}=="02"
    ATTRS{interface}=="TellStick"
    ATTRS{supports_autosuspend}=="1"

Yes! The ATTRS{interface}==”TellStick” thing looks like it’s something we can use to distinguish between Tellsticks and other devices, and it’s not so far down in the device hierarchy that we’re not talking about serial interfaces any more, and presto!

[larsi@stories ~]$ ls -l /dev/tellstick 
lrwxrwxrwx 1 root root 7 Feb 25 16:33 /dev/tellstick -> ttyUSB1

Here’s the magical setup file, for reference if anybody wants to write a udev rule for Tellstick devices that works on February 26th 2018, but probably not the week after:

$ cat /etc/udev/rules.d/10-tellstick.rules 
ATTRS{interface}=="TellStick", MODE="0666", SYMLINK+="tellstick"

You can also refer to the parent attributes by saying SUBSYSTEMS at a strategic point, so the following also works today:

KERNEL=="ttyUSB[0-9]*", SUBSYSTEM=="tty", 
  SUBSYSTEMS=="usb", ATTRS{idVendor}=="1781", 
  ATTRS{idProduct}=="0c30", 
  MODE="0666", SYMLINK+="tellstick"

Basically, when upgrading a Linux machine, most everything just works; all the peripherals and all the software. But I always end up twiddling the udev setup for half an hour.

Useful Consumer Review

I’ve had the vast majority of the lights in my apt. controlled remotely (from Emacs, of course) for like a decade. It’s a flexible system built on Telldus Telstick receivers and transmitters, and Nexa wall sockets.

But… Look at the un-pretty:

Yes, those outlets are fugly. Fortunately quite a few of them are hidden behind furniture, but I’ve been on the lookout for prettier solutions. Remote-controlled light bulbs are nice, but most of them use proprietary control systems that seem fiddly and not easy to integrate into my setup. Ikea have released bulbs that show some promise, but the form factors are still very limited, so I’ve been biding my time for years waiting for somebody to make something… better.

Look!

Telldus apparently started making these outlets a couple of years ago. They use the same 433MHz signalling system as the Nexa outlets (and the Telldus transmitters), and they look so much prettier. Of course, Telldus seems to be phasing them out already because they want to sell a completely new product line based on ZWave, so these were on half price sale and I snapped up a stack of them.

Sometimes it pays to be oblivious.

So last night I sat down to start changing the outlets, and the first thing to do is to program them. (I give each outlet an individual code, a room code, and an apt. code, so that I can switch the entire room/apt on/off with one signal.)

But they behaved very strangely. One of the outlets I could give code 40, but not 202. But I could give it code 56. Hm. 7 bit problems? No, I could give it code 153… WTF?

So I started dumping the output from the included remote controls:

And it dawned on me that the second-to-last hex digit was always 9 on the included remote controls, but not in the codes I transmitted.

What I had was

(when (> unit 16) 
  (setq house (+ house (/ unit 16)) 
        unit (mod unit 16)))

That is, I did modulo 16 on the unit code (because that’s a 4 bit value) and then went to the next house code. And that worked fine for the Nexa outlets, but with these Telldus outlets, not all house codes are valid. I didn’t investigate in depth what patterns were allowed, but every 16 was definitely allowed, so I changed the above code to:

(setq house (+ house (* (/ unit 16) 16)) 
      unit (mod unit 16)))

So I skip to the next sixteenth house code instead of the next one.

And then it works! Of course, that means that I had to reprogram all of the outlets, even the old Nexa ones that I didn’t swap out. Oh, the tangled twisty turns of life.

Look teh pretties:

They seem to have pretty good reception, too. That is, at least not any worse than the Nexa outlets. But as you can see, they’re not perfect: They’re juuust a smidgen too wide to fit two side-by-side into a standard outlet, and they may even be problematic with a fat non-remote neighbour.

*sigh*

But prettier!  And more convenient.  These Telldus outlets have a manual on/off button, so you can get the lamps to switch on/off even if the control system is down, for some reason or other.

Now I just have to decide what to do with the old outlets… Probably not much use for them, but perhaps I should hold on to them to see whether the Telldus outlets continue for function well.

Useful Consumer Review

You know when you’re measuring out things for baking? So you put a mixing bowl on the kitchen scale and then measure out 300g of sugar, and then you tare it back to zero, and then you’re going to pour 500g flour into the bowl, but after pouring some flour, the flour bag is empty, so you start looking through the cupboard for more flour, and just when you’re about to start pouring the rest of the flour, and then the scale auto-switches itself off? And then you cry and cry because your life is ruined forever and ever?

Nothing like that has happened to me, but here’s the solution:

The Soehnle Page Profi.

It does have auto-off functionality, but according to my rigorous testing, it waits until ten minutes of inactivity before doing that. Which should be enough to look through several kitchens’ worth of cupboards for that bag of flour.

And it takes AAA batteries, which is convenient, and it has a max range of 15kg, which is 3x more than most kitchen scales.

On the less positive side, the viewing angle for the LCD display is a bit on the low side, and it’s very shiny and it looks like a grease stain catastrophe after a few seconds. And the touch tare/on-off buttons are way to easy to touch (heh heh) accidentally.

Death to all touch buttons!

Watch Repair Guy

In my 20s, I bought a bunch of cheap but fun watches. While tidying up the other month, I came across the watch cache, and I thought it might be fun to start wearing them again.

The batteries had all expired decades ago, of course, and taking them all to the watchmaker sounded kinda silly, because changing the batteries would cost most than the watches themselves. Well. Almost.

And it turns out the batteries by themselves cost virtually nothing, so I bought all the required types and a watch repair set.

And a rum and coke.

Popping open the back cover of the watches is trivial if you have the right tool (seen to the left there): It’s a kinda blunt knife. And then you need a teensy screwdriver to push stuff around inside the watch to make the battery pop out.

If you have a watch with a screw-on back, you need the tool above. It’s pretty obvious if the watch is of that type, because the back has notches going around the circumference. It’s, however, not completely obvious to use this tool, so search Youtube for a how-to. It’s really easy when you understand the concept.

Look! Watches! Telling approximately the correct time! Now I don’t have to wear the same one more than twice a month!

There’s apparently four different battery types that cover the gamut of watch types.

So there you go: Swapping watch batteries is very easy. You just need a) batteries, and b) a watchmaker tool set (there’s a bunch of cheap ones out there), 4) rum and coke, and xvii) very good lighting.

Useful Consumer Review

I’ve got most of the rooms in the apt. wired up for sound, but I’ve never managed to work up enough stamina to get the bathroom wired.

So I’ve been using this Creative wireless blaster thingie for years and years, and it works OK. It sounds fine and it usually works. But it’s… kinda a lot of stuff.

It’s got an external power supply, and there’s all those wires back and forth… I mean, I can’t stand untidy wires.

Just look at what spills out whenever I open the cupboard door next to my main computer.

Aaanyway.

The amplifier in the bathroom died some months back, and I got a new one, and this one has bluetooth built in. So I thought: Hey! An opportunity to slim down the chaos on top of that cupboard in the bathroom, at least. I should be able to get rid of basically all of that by switching to internal bluetooth.

Order! Cleanliness! Goodness!

So I bought this bluetooth transmitter.

It’s a Homespot Dual Stream Bluetooth Audio Transmitter, and I got it because it talked about low latency bluetooth (in addition to aptX and all the other modern bluetooth goodness).

Getting the devices to pair was slightly tricky, because the amplifier will pair with anything, anywhere, and the transmitter will pair with the first willing device when it’s switched on. Which was confusing, because the first three times I switched it on, it paired with something, but not the amplifier. I got it to work on the fourth try by holding it two centimetres from the amplifier when I switched it on.

Heaven knows what it’s streaming music to in addition to the amplifier now…

But remember that “low latency” thing? Well:

If you pump up the volume you’ll hear a lovely Machinedrum ditty coming from the office (wired sound), and then I turn the amplifier up in the bathroom. Listen to that loooow latency.

*sigh*

Well, it’s not like I usually have the music on the bathroom switched on. I mostly just use it while showering, and in that case I can’t hear any music coming from other rooms.

But that’s really annoying. The Creative wireless thing has a much, much lower latency: You mostly experience the effect as a sort of thickening of sound, not the stumbling effect you get from the bluetooth latency.

So I dunno… I think I may just switch back to the old setup, even though it’s so… messy…

VHS, Linux, Problems

I’ve been trying to tidy up the storage locker in the loft this autumn, getting rid of old junk (so that I can put more, slightly newer junk up there). I happened unto this box:

A nice stack of VHS tapes. If I remember correctly, the reason I kept these was that during the 80s and 90s I recorded quite a lot of music (videos and live stuff) from the TV, and I had always thought about some day going through them, picking out the good bits and uploading them to Youtube or something.

Perhaps this year is finally the year?

I have an old, old DV conversion thingie somewhere, but it’s so old that it’s a Firewire dongle. And I have no Firewire equipment, so I thought it might just be easier to buy something new and shiny. Surely?

I think you can already guess where this tale is going.

This is a Diamond VC500, and it has both composite in and S-Video in, and it seems to be a popular choice.

My old VHS player (a Panasonic DMS-ES35V) has S-Video out, so I hoped that it would, like, just kinda work, and it does. With composite, but not S-Video. That is, after I discovered how to switch the input, something worked:

$ v4l2-ctl -d /dev/video0 -i 1 -s 5

(That’s S-Video input and the PAL standard.)

But the video was in black-and-white, and after googling for a few days, it turns out that this is a known problem: S-Video on the VC500 doesn’t work, at least not in Linux. And S-Video is a much better signal than composite, so that’s a bummer.

So I got another one of these dongles at random, and this is from StarCom and presents itself as

 Bus 001 Device 012: ID eb1a:5051 eMPIA Technology, Inc.

And S-Video works! However, sound doesn’t, and googling for another two days show that this is a common problem with the “StarTech.com S-Video / Composite to USB Video Capture Cable Adapter”, at least under Linux.

But audio is easy: I could just hook it up to the internal sound card, and after fiddling with alsacontrol for a few hours, I managed to get some sound out of it.

All set! Recording time, here we go!

Unfortunately, I had lost the remote control to my VHS player and adjusting the tracking without it is impossible.

Ebay to the rescue!

It’s amazing that things like VHS head cleaners are still easily obtainable, so I got one of those, too.

Now, surely, I’m ready to record.

Except finding some acceptable settings for ffmpeg or memcoder to process the data and output in a format that’s good for editing turned out to require several more days worth of googling. Here’s what I ended up with:

$ ffmpeg -f v4l2 -t 04:00:00 -thread_queue_size 1024 -i /dev/video0 -f alsa -i hw:0 -aspect 4:3 -c:v prores_ks -profile:v 3 -q:v 4 -vf yadif=0:-1:0,crop=iw:ih-6:0:0 -c:a pcm_s16le -af aresample=async=1000 /video/vhs/v10.mov

So the codec here is ProRes. I tried a dozen different codecs, but either they were too slow (leading to drops from ffmpeg) or they were in formats that crashed Lightworks (the video editor I use). huffyuv, raw, h.264, whatever…

But this one works for me, at least.

Oh, and the “crop” thing is because the 6 final lines output from the StarTech dongle are bright green.  And I’m deinterlacing since VHS is interlaced.

An action shot of the entire machinery. So techy.

Now the only problem is that I had to upgrade the kernel to Linux 4.12 and the machine to the latest Debian, since the support for the StarTech USB thingie first appeared in that kernel. And after upgrading LightWorks didn’t want to start. So I upgraded to Lightworks 14, which starts, but doesn’t accept the old license, so I had to buy a new license, and now the license server is … wrong (“There are no unactivated licenses available”).

Anyway, I can use the free version for this, since 720p should be enough.  I created a new account on Youtube for this stuff, since it’s, er, best to keep it separated from the more important free jazz vids.

Hm.  Or perhaps I should just use HTML <video> things and host the .mp4 files privately?  That would avoid any Youtube hassles…  Hm…  But the discoverability on Youtube is nice.  Perhaps both?

I’ve done the first four hour tape now, and there was about 20 minutes of things that are possibly interesting.  For instance, this Kristin Hersh piece where she does four songs live in the 120 Minutes studio:

Oh, it’s apparently not possible to do simple <video> embeds on a WordPress.com site?

Oh, it is possible now?

Yes!  All those pages that said that it wasn’t allowed were outdated.  Apparently WordPress.com started allowing this earlier this year?

Anyway.