Wednesday, September 11, 2013

obligatory six-month update

hello, little blog! how have you been? things have been pretty good for me... well, except for the one incident...

back in july, virago & i went to milwaukee to see four legends of midwest breakcore: stunt rock, abelcain, doormouse, and anonymous. (doormouse, you may recall, headlined the after-party at our wedding.)

the show was great, with three of the performers effectively coming out of retirement for the show. the music was great, we got to chat and catch up with doormouse and stunt rock, and made some new friends. and we did other fun milwaukee stuff like dining out (milwaukee loves friday fish fries and something called "pulled ham", which is like pulled pork, except it's ham), record shopping, and the obligatory brewery tour.

then on the way home, we were cruising the chicago freeway when another driver "fell asleep" and crashed into my rear bumper at full speed. (i don't know how you fall asleep on the freeway at 3 in the afternoon without drugs, alcohol, or a medical condition, but that was her story.) we were then pushed down the freeway an unknown distance, hit the curb, and spun around 360 degrees, ending up in the left lane with a broken front axle.

somehow, despite the fact that the chicago freeway is pretty busy on a weekday mid-afternoon, we didn't hit the median or any other vehicles. we could easily have been killed, but escaped with only minor injuries (virago had some back pain for a few days, but that was it). but of course my car was totaled, and the six-pack of new glarus beer i had picked up to bring home was in the trunk right at the point of impact, sending beer and broken glass all over the trunk, and damaging a few of the records i'd brought with me for trade.

so that sucked, but at the same time it became an opportunity to get a newer car. after getting our insurance check, we headed out to terry lee honda (chosen because it was the best-reviewed honda dealer in the metro area, and after going there it's clear why) and now we have a 2011 civic—the same model as my previous car, only 10 years newer, in black, and a touch more blinged out.

so, near death experience aside, things have been good. i've been keeping busy. i finished a new release called taco meat, coming hopefully soon on a german label called betonblume. it's a bit glitchy, with some wack cylinders-style sampling but this time with beats—with actual drum programming (and not just sequencing percussion sounds sampled from somewhere else). i'm really pleased with it.

i've been making tons of art as well, and getting work in art shows. this summer i was in a show called run, computer, run (show is over but you can still view my piece on the website). i also have work in a show called _ģ̶Ł̶1̶ɫ̶C̶ʮ̶_Δ┌┼---\/\/\/\/\/\/!┬---jonCates_(AKA "Glitch Art with jonCates"), which opens tomorrow in saskatchewan as well as online at then later this month i'll have a piece in a show called money as error. and i'm currently working on something pretty cool for the wrong - new digital art biennale, which opens november 1. all that, and i post new work daily at, and a couple other things in the works i probably shouldn't mention yet. and did i mention i'm still doing my weekly net radio show Active Listening / The Act of Listening on

so yeah, i've been productive. stay tuned for future updates (though following me on facebook or tumblr will probably yield better results than relying solely on this blog).

Tuesday, March 12, 2013

just how bad is soundcloud's "automatic content protection system"?

as many of you know, one of my music processes is is a form of sonification (or audification) where i trick my computer into opening various data files as if they were sound files, and then i make music out of the resulting sounds. (i wrote a tutorial about it here.)

a while back i posted an old song called "denial of service" to soundcloud because i was thinking about including it on my recently-released album on the DLL. the song is what you might call "pure" data sound, in that every sound in the song is made of this sonified data. so imagine my surprise when i got an email from soundcloud that they'd taken the song down because they thought i had sampled some house song:

i repeat: every sound in my song is taken from sonified data. no drum machines, drum samples, or any other instruments were used. but somehow my song still triggered an automatic takedown for uncleared sampling. i didn't even run any audio effects on the samples—they are all clean.

want to hear for yourself? here is my song (warning: fairly abrasive). and here is the song they think i sampled (warning: fairly mediocre). no human who listened to these two recordings would think they had anything in common.

as a glitch artist/musician, i find this fascinating. what could this algorithm be hearing in my song to make it think i sampled some house track? in terms of tone, timbre, tempo, and meter, these two recordings could hardly be more dissimilar. but to a bot, they apparently sound alike. somewhere in there, indiscernible to mere human hearing, the two must have some tonal sweep or polyrhythm in common. or some checksum came up with the same result by chance. it's hard to even speculate about how this may have happened without knowing more about how their bot works.

but it's also frustrating that my song has been pulled down based on such a laughably false accusation. and although i'm certain i will win the dispute, it's annoying to have to send soundcloud my name, address, and phone number, as well as tick seven checkboxes threatening to sue my ass off if i'm lying:

of course, there's no way for me to take soundcloud to court for wasting my time or for issuing bogus threats. all this because of a poorly programmed bot. in a way i'm lucky that my song obviously doesn't contain any "musical" samples of any sort. what would happen if i had made some housey track that just happened to use the same drum hits, or just sounded similar?

update: within a couple hours of filing a dispute, i received the following message:

thank you for providing feedback in regards of the upload:
stAllio! - denial of service

This notification is to inform you that your upload has been released to your account.
The SoundCloud team

so although it's reassuring that these disputes are reviewed quickly, it's alternately fascinating and horrifying that such a thing could happen in the first place. and i must wonder what would've happened if my case hadn't been so open-and-shut.

Monday, January 07, 2013

summoning ghost frames

this is a write-up of the ghost frames workshop i delivered at GLI.TC/H 2112

ghost frames is a term i use to describe video frames which do not actually exist in a video file but which are sometimes revealed when using damaged data and/or glitch processes. for the sake of discussion, i'll also use "ghost frames" to refer to the process i use to invoke these frames. my process involves a bug in certain versions of avidemux, though there are likely other ways you could do something similar. (nick extracrispy tells me he can do something similar in quicktime, for example.)

specifically, i use avidemux version 2.5.4. the process is likely to work in other 2.5 and earlier versions, though i haven't tested them. it definitely does not work in the current version, 2.6. also, during the workshop, some mac users reported that they couldn't get avidemux to install. for os x lion/mtn lion, check these instructions for installing avidemux.

i initially discovered ghost frames thanks to another glitchy behavior in avidemux. my desktop computer runs windows 7 and has a video capture card for recording tv signals off a coax connection. the default tv capture program in windows 7 is called windows media center, which records programs in a format called WTV. WTV is proprietary (closed) format, but it's close enough to MPEG that when i try to open a WTV file in avidemux 2.5.4, it asks me "This looks like an MPEG. Do you want to index it?"

answering yes to this question and no to any follow-up questions results in a video that is recognizable but thoroughly mangled. here are some typical examples:


occasionally you'll get weirder, unpredictable glitches like this:


i'd been playing around with this wtv transcoding glitch for months before i saw my first ghost frame, which i discovered by accident. to invoke ghost frames in avidemux, scroll through a damaged video backwards. just skip to the end (or any point in the middle, if you prefer), and use your left arrow key to reverse through the file. if your file is damaged in the "right" way (say, via WTV transcoding), then you'll start to see frames that aren't really there:


what seems to be happening here is that avidemux is getting confused and applying incremental changes in the wrong direction. to save space, MPEG files only occasionally send full frames (keyframes or i-frames) and fill the rest of the video with p- and b- frames that only describe which pixels have changed since the previous frame. reversing through the file seems to trick avidemux into applying these changes wrong, thus revealing glitched frames that aren't actually in the file. (and i love the fact that you do this by going in reverse; it really makes it seem like some magic ritual.)

you can even do something i call dancing between the keyframes, inching back and forth through a glitchy area of the file as the artifacts become increasingly pronounced:


of course, it's one thing to see a ghost, and something else altogether to capture a ghost. ghost frames disappear just as suddenly as they appear. skip so much as one frame too far and those ghostly glitches you've been cultivating could vanish. you can't just hold down the left arrow key and storm through the place: you have to inch along. you have to creep.

so that's how you summon ghost frames from WTV files. it's all well and good if you're running windows 7 with a tv capture card, and have video content you can pipe through a coax connection, but that's a fairly narrow demographic. also, WTV transcoding introduces some artifacts i'd prefer to be able to turn off, like the banding at the top of the screen. so when preparing for the ghost frames workshop, i spent a few hours digging around for non-WTV files that would work with this process. most files i tried wouldn't. but by chance, i found one file that did. further investigation revealed that the teaser videos on NSFW adult website could be used to summon ghost frames—but only the videos from the past several months. (i don't have a subscription to the site so i don't know if the full-length subscriber videos glitch in the same way, though it seems likely.)

fun glitches! but because i found these files in the wild and they seem to be normal working video files, i don't know what about them makes them glitch like this. more about that below.

areas to explore

  1. crack the mystery of what makes the ellinude videos glitch. is it something in the encoding settings? something in the camera that shot them? something in the actual frame content? a combination, or other factors? the obvious goal is to reproduce these effects on files of our own choosing.
    my analysis suggests these files were encoded using ffmpeg, but i haven't quite been able to reverse-engineer the encoding settings. (and don't know if it would do much good even if i could.) some metadata from some of the video files is available here (thanks to antonio roberts for his help compiling this metadata).
  2. if we can summon ghost frames from transcoded WTV files and from ellinude's MP4s, then they could likely be found in other types of mangled/miscoded video. damaged/databent files, deliberately poor encoding, hacked codecs, etc are possible avenues for source video.
  3. this process has a lot of similarities with datamoshing. the two processes could likely be combined to interesting effect.
  4. version/system sensitivity testing—will this process work in other versions of avidemux, and if so which ones? will the same file produce the same ghost frames using different versions of avidemux or different operating systems?
  5. identify other tools/processes for summoning ghost frames.