[sf-lug] YouTube downloading

Wed May 18 02:42:14 PDT 2011

(Copying over a conversation from CABAL's mailing list:)

Date: Tue, 17 May 2011 19:49:43 -0700
From: Rick Moen <rick at linuxmafia.com>
To: conspire at linuxmafia.com
Subject: Re: [conspire] get back to me first
Organization: If you lived here, you'd be $HOME already.

Quoting Bruce Coston (jane_ikari at yahoo.com):

> I need to get a distro on one of my many small-ish laptop partitions
> that will play back YouTube videos I suck at the library after I
> suspend to disk and go home during our many summer Internet outages.
> My elive install suffers real problems with this, after I take it off
> disk suspend, back home.

Bruce --

I'm going to make a longer post on this subject that aims to be more
comprehensive, but here's the short version:

You're relying on local cache state of the Adobe Flash interpreter being
useful and complete after suspend to disk. Bad strategy. Instead,
install scripts or browser extensions to download the clips to local
disk.

Slight elaboration: I'm assuming Flash (Sorenson H.263 encoding in FLV
container) for the sake of discussion, though Google's YouTube service
also offers video in MPEG Licensing Authority's patent-problematic H.264
encoding in either FLV or MP4 (MPEG-4 Part 14) containers (often
inaccurately called 'HTML5' video), in Google's (ex-On2) VP8 encoding in
WebM containers (which along with Vorbis encoding in a Matroshka
container is part of the HTML5 draft), and in MPEG-4 Visual encoding in
3GP containers (a format primarily intended for mobile 'phones). It's
pretty much the same problem regardless.

Why a problem? Your media player plugin (Adobe Flash, for the sake of
discussion) is streaming-in the inbound A/V bitstream and dumping it
into ~/.mozilla/firefox/[hash]/Cache on an ongoing basis, but that
cached copy isn't guaranteed complete and useful at any given moment.
Also, it's not the least bit surprising to hear that Adobe Flash acts
stunned and stupid if it suddenly wakes up after being suspended to
disk. So, no, you're actually lucky if that ever works, because you can
be sure that the buggy spaghetti code in question isn't designed to do
it.

So, let's say you look at the page HTML on a YouTube page to try to find
the .flv direct link so you can pull a copy down using wget or curl.
Surprise, you can't find it. Why not? Because you are not Google's
customer; you are their product -- and they are far, far more eager to
make Our Lords in Hollywood happy than they are you. So, they cause
YouTube to wrap the .flv link in inscrutable JavaScript, in order to
make it difficult to puzzle out. Moreover, they gratuitously change that
JavaScript obfuscation occasionally, to screw people up who've
de-obscured the latest version and are letting others know how.

(They do that because Our Lords in Hollywood don't want users to have
local copies that cannot be retroactively withdrawn from availability.)

Nonetheless, there are folks who gamely keep up with the changes, and
you can & should benefit from their work:

clive (Perl script)
cclive (newer C++ rewrite of clive)
youtube-dl (Python script)
Flash Video Downloader (Firefox extension)
Easy YouTube Downloader (Firefox extension)
Get 'em all.

There are distro packages for clive, cclive, youtube-dl (judging by
Debian). You may find the distro packages' frequency of update adequate
to keep up with JavaScript mischief at YouTube, Photobucket,
Dailymotion, Metacafe, Facebook, Yahoo, depositfiles.com, etc. -- or
not. If not, you might need to have them as locally installed and
updated software.

ALSO:

There's a persistent misconception, spread about by the ill-informed,
that the open source Flash interpreters (Gnash, Swfdec, and GPLFlash)
are inadequate to handle the Flash served by YouTube. No, wrong. It's
really just the obfuscatory JavaScript.[1] Which, if you use a
downloader such as those listed above, and it works, suddenly becomes a
non-problem. So, in addition to the recommendations above, please
consider losing the Adobe Flash proprietary garbage, or at least using
Gnash in preference. (For one thing, the Gnash developers don't spy on
users, about which please see http://lwn.net/Articles/129729/.)

[1] Adobe consistently withheld Flash's technical specifications from
the open source community, making selected details available only under
restrictive agreements. Starting June 2009, in the face of mounting
competition from alternatives (H.264 in MP4, VP8 in WebM, Vorbis in
Matroshka, and especially Microsoft Silverlight), they finally coughed
up the SWF file format specification without restriction, but omitted
anything about the key Real Time Messaging Protocol (RTMP) and Sorenson
Spark codec. Moreover, the open source community had already
independently reverse-engineered everything Adobe did disclose.

Date: Tue, 17 May 2011 19:55:15 -0700
From: Rick Moen <rick @linuxmafia.com>
To: conspire at linuxmafia.com
Subject: [conspire] YouTube downloading software (was: get back to me
first)
Organization: If you lived here, you'd be $HOME already.

I should also acknowledge Micah Lee's really useful posting on this
subject. Thanks, Micah!

This is my compendium of links relative to my suggestion to Bruce
Coston, just posted
(http://linuxmafia.com/pipermail/conspire/2011-May/006225.html) that he
deal with his video streaming problem with YouTube and similar sites
through use of downloaders, which in turn can also permit, if he wishes,
losing that dreadful Adobe Flash plugin software and substituting
open-source compatibles (Gnash, Swfdec, GPLFlash).

Here's the laundry list, which I hope will prove useful. Remember, you
should usually favour distro packages where available, rather than going
to upstream links (such as those cited). All codebases mentioned are
genuine open source (except for Google's Chrome Web browser).

clive is a Flash video extraction tool in Perl for user-uploaded video
hosts such as Youtube, Google Video, Dailymotion, Guba, and Metacafe. It
can be chained with 3rd party tools for subsequent video re-encoding and
playing and playing. 
http://home.gna.org/clive/

clive-utils is a collection of add-on tools to parse RSS feeds, parse
video pages, and manage login passwords related to video sites. 
http://code.google.com/p/clive-utils/

cclive is a rewrite of clive in C++ for lower system load. 
http://cclive.sourceforge.net/

abby is a Qt-based graphical front end in C++ for clive and cclive. 
http://code.google.com/p/abby/

youtube-dl is a (command-line) Python script for downloading Flash or
WebM (vp8 + Vorbis) video clips for YouTube.com, Google Video,
Photobucket, Yahoo Video, Dailymotion, DepositFiles, and some other
sites. You provide the clip's URL, and youtube-dl saves the clip as a
local file with file extension .flv. 
http://rg3.github.com/youtube-dl/

For maximum compatibility with even the more difficult (DRM-obscured)
YouTube clips, you should also install the rtmpdump program to decrypt
files served using Adobe's proprietary 'RTMP' protocol. See: 
http://rg3.github.com/youtube-dl/documentation.html

YouTube Downloader GUI is a Qt/kdelibs front-end for youtube-dl: 
http://linux.softpedia.com/get/Multimedia/Video/Youtube-Downloader-GUI-28348.shtml

Qttube is a similar Qt front-end for youtube-dl: 
https://wiki.ubuntu.com/QtTube

Flash Video Downloader is a Firefox extension to download Flash video
clips from YouTube, Dailymotion, Break.com, and many other sites. 
http://www.flashvideodownloader.org/

Easy YouTube Downloader is a Firefox (and Chromium / Chrome) extension
to download YouTube videos in FLV, 3GP, MP3, MP4 and 720p HD and 1080p
Full-HD qualities. 
http://www.yourvideofile.com/ 
https://addons.mozilla.org/en-us/firefox/addon/easy-youtube-video-downl-10137/

quvi is a command-line tool in C for parsing Flash video links out of
YouTube and similar sites. 
http://quvi.sourceforge.net/

PyTube is a Python app that searches and downloads YouTube video clips
and encodes them using FFMpeg: 
http://pytube.sourceforge.net/ 
(gone from: http://bashterritory.com, http://www.getdeb.net/app/PyTube,
http://www.marcosrodriguez.me/pytube/)

pytube (different codebase) is a Python script for YouTube and MetaCafe
that de-obscures the video clip's URL from its obscuring JavaScript, so
that you can download it. 
http://code.google.com/p/pytube/

Note: There was for a long time a Python / gtk+2 graphical application
also called Pytube, but unfortunately everyone relied on someone else to
host it, and it it appears to have vanished from the Internet.

Utube Ripper (formerly Youtube Downloader for Linux) is an application
written in Gambas that works exclusively on Linux. It's useful to
download and convert Youtube videos in a simple and efficient way. 
http://sourceforge.net/projects/utube/

(g)vdown is an application that can download videos from video sharing
websites like YouTube, Google Video, MyVideo and so on. It is written in
Python. It has a command-line interface (vdown) and a PyGTK interface
(gvdown). 
http://code.google.com/p/vdown/

Get YouTube Video is a KDE service menu usable with Konqueror to 
download YouTube Flash video clips. 
http://www.kde-apps.org/content/show.php/Get+YouTube+Video+(improved)?content=41456

gnetvideoplayer is a C program for GNOME that serves as a player and
downloader for YouTube and Stage6 videos, and depends on wget, MPlayer,
and GtkMediaPlayer. 
https://forxa.mancomun.org/frs/?group_id=108

youshell is a Python script to search and download Flash video clips
from YouTube and load them into Mplayer. 
http://code.google.com/p/youshell/

Elltube is a graphical Python / Qt4 YouTube Flash downloader and
converter. 
http://elltube.sourceforge.net/

MP3 Rocket is a Java program to download YouTube and other clips 
http://www.mp3rocket.com/

Articles: 
http://www.linux.com/learn/tutorials/442454-youtube-tools-for-the-linux-desktop 
http://arun.wordpress.com/2007/01/29/download-youtube-videos-the-linux-way/ 
http://www.junauza.com/2010/01/download-youtube-videos-from-linux.html 
http://maketecheasier.com/download-youtube-videos-for-offline-viewing-on-linux/2009/02/09/ 
http://linux.byexamples.com/archives/302/how-to-wget-flv-from-youtube/ 
http://www.freesoftwaremagazine.com/columns/youtube_and_gnu_linux_download_and_convert_videos_easy_way