Date: Tue, 17 May 2011 19:49:43 -0700
From: Rick Moen <rick@linuxmafia.com>
To: conspire@linuxmafia.com
Subject: Re: [conspire] get back to me first
Organization: If you lived here, you'd be $HOME already.

Quoting Bruce Coston (jane_ikari@yahoo.com):

> I need to get a distro on one of my many small-ish laptop partitions
> that will play back YouTube videos I suck at the library after I
> suspend to disk and go home during our many summer Internet outages.
> My elive install suffers real problems with this, after I take it off
> disk suspend, back home.

Bruce --

I'm going to make a longer post on this subject that aims to be more comprehensive, but here's the short version:

You're relying on local cache state of the Adobe Flash interpreter being useful and complete after suspend to disk. Bad strategy. Instead, install scripts or browser extensions to download the clips to local disk.

Slight elaboration: I'm assuming Flash (Sorenson H.263 encoding in FLV container) for the sake of discussion, though Google's YouTube service also offers video in MPEG Licensing Authority's patent-problematic H.264 encoding in either FLV or MP4 (MPEG-4 Part 14) containers (often inaccurately called 'HTML5' video), in Google's (ex-On2) VP8 encoding in WebM containers (which along with Vorbis encoding in a Matroshka container is part of the HTML5 draft), and in MPEG-4 Visual encoding in 3GP containers (a format primarily intended for mobile 'phones). It's pretty much the same problem regardless.

Why a problem? Your media player plugin (Adobe Flash, for the sake of discussion) is streaming-in the inbound A/V bitstream and dumping it into ~/.mozilla/firefox/[hash]/Cache on an ongoing basis, but that cached copy isn't guaranteed complete and useful at any given moment. Also, it's not the least bit surprising to hear that Adobe Flash acts stunned and stupid if it suddenly wakes up after being suspended to disk. So, no, you're actually lucky if that ever works, because you can be sure that the buggy spaghetti code in question isn't designed to do it.

So, let's say you look at the page HTML on a YouTube page to try to find the .flv direct link so you can pull a copy down using wget or curl. Surprise, you can't find it. Why not? Because you are not Google's customer; you are their product -- and they are far, far more eager to make Our Lords in Hollywood happy than they are you. So, they cause YouTube to wrap the .flv link in inscrutable JavaScript, in order to make it difficult to puzzle out. Moreover, they gratuitously change that JavaScript obfuscation occasionally, to screw people up who've de-obscured the latest version and are letting others know how.

(They do that because Our Lords in Hollywood don't want users to have local copies that cannot be retroactively withdrawn from availability.)

Nonetheless, there are folks who gamely keep up with the changes, and you can & should benefit from their work:

Get 'em all.

There are distro packages for clive, cclive, youtube-dl (judging by Debian). You may find the distro packages' frequency of update adequate to keep up with JavaScript mischief at YouTube, Photobucket, Dailymotion, Metacafe, Facebook, Yahoo, depositfiles.com, etc. -- or not. If not, you might need to have them as locally installed and updated software.

ALSO:

There's a persistent misconception, spread about by the ill-informed, that the open source Flash interpreters (Gnash, Swfdec, and GPLFlash) are inadequate to handle the Flash served by YouTube. No, wrong. It's really just the obfuscatory JavaScript.[1] Which, if you use a downloader such as those listed above, and it works, suddenly becomes a non-problem. So, in addition to the recommendations above, please consider losing the Adobe Flash proprietary garbage, or at least using Gnash in preference. (For one thing, the Gnash developers don't spy on users, about which please see http://lwn.net/Articles/129729/.)


[1] Adobe consistently withheld Flash's technical specifications from the open source community, making selected details available only under restrictive agreements. Starting June 2009, in the face of mounting competition from alternatives (H.264 in MP4, VP8 in WebM, Vorbis in Matroshka, and especially Microsoft Silverlight), they finally coughed up the SWF file format specification without restriction, but omitted anything about the key Real Time Messaging Protocol (RTMP) and Sorenson Spark codec. Moreover, the open source community had already independently reverse-engineered everything Adobe did disclose.






Date: Tue, 17 May 2011 19:55:15 -0700
From: Rick Moen <rick @linuxmafia.com>
To: conspire@linuxmafia.com
Subject: [conspire] YouTube downloading software (was: get back to me first)
Organization: If you lived here, you'd be $HOME already.

I should also acknowledge Micah Lee's really useful posting on this subject. Thanks, Micah!

This is my compendium of links relative to my suggestion to Bruce Coston, just posted (http://linuxmafia.com/pipermail/conspire/2011-May/006225.html) that he deal with his video streaming problem with YouTube and similar sites through use of downloaders, which in turn can also permit, if he wishes, losing that dreadful Adobe Flash plugin software and substituting open-source compatibles (Gnash, Swfdec, GPLFlash).

Here's the laundry list, which I hope will prove useful. Remember, you should usually favour distro packages where available, rather than going to upstream links (such as those cited). All codebases mentioned are genuine open source (except for Google's Chrome Web browser).


clive is a Flash video extraction tool in Perl for user-uploaded video hosts such as Youtube, Google Video, Dailymotion, Guba, and Metacafe. It can be chained with 3rd party tools for subsequent video re-encoding and playing and playing.
http://home.gna.org/clive/

clive-utils is a collection of add-on tools to parse RSS feeds, parse video pages, and manage login passwords related to video sites.
http://code.google.com/p/clive-utils/

cclive is a rewrite of clive in C++ for lower system load.
http://cclive.sourceforge.net/

abby is a Qt-based graphical front end in C++ for clive and cclive.
http://code.google.com/p/abby/

youtube-dl is a (command-line) Python script for downloading Flash or WebM (vp8 + Vorbis) video clips for YouTube.com, Google Video, Photobucket, Yahoo Video, Dailymotion, DepositFiles, and some other sites. You provide the clip's URL, and youtube-dl saves the clip as a local file with file extension .flv.
http://rg3.github.com/youtube-dl/

For maximum compatibility with even the more difficult (DRM-obscured) YouTube clips, you should also install the rtmpdump program to decrypt files served using Adobe's proprietary 'RTMP' protocol. See:
http://rg3.github.com/youtube-dl/documentation.html

YouTube Downloader GUI is a Qt/kdelibs front-end for youtube-dl:
http://linux.softpedia.com/get/Multimedia/Video/Youtube-Downloader-GUI-28348.shtml

Qttube is a similar Qt front-end for youtube-dl:
https://wiki.ubuntu.com/QtTube

Flash Video Downloader is a Firefox extension to download Flash video clips from YouTube, Dailymotion, Break.com, and many other sites.
http://www.flashvideodownloader.org/

Easy YouTube Downloader is a Firefox (and Chromium / Chrome) extension to download YouTube videos in FLV, 3GP, MP3, MP4 and 720p HD and 1080p Full-HD qualities.
http://www.yourvideofile.com/
https://addons.mozilla.org/en-us/firefox/addon/easy-youtube-video-downl-10137/

quvi is a command-line tool in C for parsing Flash video links out of YouTube and similar sites.
http://quvi.sourceforge.net/

PyTube is a Python app that searches and downloads YouTube video clips and encodes them using FFMpeg:
http://pytube.sourceforge.net/
(gone from: http://bashterritory.com, http://www.getdeb.net/app/PyTube, http://www.marcosrodriguez.me/pytube/)

pytube (different codebase) is a Python script for YouTube and MetaCafe that de-obscures the video clip's URL from its obscuring JavaScript, so that you can download it.
http://code.google.com/p/pytube/

Note: There was for a long time a Python / gtk+2 graphical application also called Pytube, but unfortunately everyone relied on someone else to host it, and it it appears to have vanished from the Internet.

Utube Ripper (formerly Youtube Downloader for Linux) is an application written in Gambas that works exclusively on Linux. It's useful to download and convert Youtube videos in a simple and efficient way.
http://sourceforge.net/projects/utube/

(g)vdown is an application that can download videos from video sharing websites like YouTube, Google Video, MyVideo and so on. It is written in Python. It has a command-line interface (vdown) and a PyGTK interface (gvdown).
http://code.google.com/p/vdown/

Get YouTube Video is a KDE service menu usable with Konqueror to
download YouTube Flash video clips.
http://www.kde-apps.org/content/show.php/Get+YouTube+Video+(improved)?content=41456

gnetvideoplayer is a C program for GNOME that serves as a player and downloader for YouTube and Stage6 videos, and depends on wget, MPlayer, and GtkMediaPlayer.
https://forxa.mancomun.org/frs/?group_id=108

youshell is a Python script to search and download Flash video clips from YouTube and load them into Mplayer.
http://code.google.com/p/youshell/

Elltube is a graphical Python / Qt4 YouTube Flash downloader and converter.
http://elltube.sourceforge.net/

MP3 Rocket is a Java program to download YouTube and other clips
http://www.mp3rocket.com/


Articles:
http://www.linux.com/learn/tutorials/442454-youtube-tools-for-the-linux-desktop
http://arun.wordpress.com/2007/01/29/download-youtube-videos-the-linux-way/
http://www.junauza.com/2010/01/download-youtube-videos-from-linux.html
http://maketecheasier.com/download-youtube-videos-for-offline-viewing-on-linux/2009/02/09/
http://linux.byexamples.com/archives/302/how-to-wget-flv-from-youtube/
http://www.freesoftwaremagazine.com/columns/youtube_and_gnu_linux_download_and_convert_videos_easy_way