YouTube storyboards; to use or not to use?

File this one under “musings on software development”…

One of the issues that occasionally crops up when coding against established online services, such as Twitter or YouTube, is what to do about the undocumented features those sites offer.

Background, for non-programmers: Many popular web sites, from social sites like Facebook and Twitter, to personal repositories like DropBox and Flickr, offer a programming API.  The API (application programming interface) is a means of working with the service from within software; it means third party applications can interact with the service just like a web visitor can (in some ways better than a web visitor can).  Instead of a search box for the query and a web page to display the results, APIs use recognised data transfer protocols, like REST, to query a service’s database and accept the results.

So sites can be accessed by software as well as human visitors.  Great, what’s the problem?

Thumbnails displayed as scrubber is dragged along video timeline.

Thumbnails displayed as the scrubber control is dragged along a video timeline.  These preview thumbnails are delivered to the video player as a ‘sheet’ consisting of a grid of images.

The problem is that sometimes sites have undocumented APIs that are intended for their internal use only, and are not publicised for general third party use.  Of course programmers quickly discover where they are, because calls to them show up in web logs.  Such is the case with YouTube’s storyboard feature.  Visitors to the popular video hosting site will have noticed that on many videos hovering the mouse over the long horizontal drag control (called the scrubber) causes a thumbnail to appear related to that part of the video.  And dragging the control causes a line of them to appear, showing previews of the video immediately before and after the current position.  YouTube achieves this by taking scaled down screen shots of each video at regular intervals, typically every 5 seconds for short videos and 10 seconds for longer ones, throughout the video’s duration.  Then it stitches them together into a big image, comprising of a grid of all the shots neatly laid out in order.

(Actually, so speed up loading times, YouTube creates several grid images, each dealing with a different segment of the video.)

The standard programming YouTube API exposes a lot of data about each video on the site: its license, tags, the comments people have added, etc.  But crucially it does not reveal the location of the storyboard grid image.  This location is one of dozens of very low level details that can be obtained by calling an undocumented YouTube script on their web site.  The script accepts the video ID as a parameter, and spits out details on such things as bit transfer rates and available video formats in return.  One of those details is a cryptic URL and accompanying data parameters which, when assembled in the right way, point towards various sets of images (of varying scales) that can be used as storyboard grids.

For the work we do with semantic video, being able to access thumbnails for arbitrary points in the video would be incredibly useful.  For example, it means semantic links into a video could be accompanied by a thumbnail showing the action close to that specific point, rather than the single static thumbnail YouTube generates to represent the whole video in search listings etc.  They would also be useful as previews in AutoKitty, when resizing and dragging items on a video’s timeline.

So… the dilemma is: to use or not to use?

I (Simon) have spent a little time experimenting with the storyboard images.  I’m not the only one: a few other coders have been looking at making use of them, as can be witnessed with a quick Google search.  But much of the code ‘out there’ to decipher the cryptic URL and parameters is very basic, and some of it doesn’t appear to work.  So I poked around and experimented to find out what each parameter meant (it wasn’t too hard) and wrote a nifty little JavaScript (jQuery) extension to fetch the storyboard images and display them correctly — cell by cell — on a page.  I think it may be the most comprehensive code out there for dealing with these images.

Because the API call is undocumented, it cannot be relied upon in the same way as the official APIs.  But it would be a shame not to use the storyboards, now that we know how to get access to them — the thing is not to use them in a mission critical way!

Here’s hoping YouTube adds them to their official API data.  (But don’t hold your breath!)



Comments are closed.

Previous Posts

%d bloggers like this: