[sf-lug] (no subject)

[Jonathan Grindstaff]

Before i get started, i'll just take a minute to outline some points.

1. digital video loses quality really fast when it is re-rendered.
This was recently likened to re-faxing something over and over. At
first this might seem odd because, of course, "digitally copying a
file a million times won't change it". And that is true, copying the
video does not change it.
But when working with video we are not always just copying . . .
sometimes we want to blend two clips together for a nice fade effect
etc. This means that the two video clips have to be merged together to
form a new image. . . . the reason dv looks worse and worse when you
work with it is that almost all digital video is _compressed_ in some
way. The deterioration of visual quality then is an issue of
re-compression. When video gets re-compressed, it changes . . . most
ppl think it's for the worse . . . some times i think it looks funky
and cool in an "avant garde" sort of way . . . . but that is way, way
beside the point.

2. the "raw" captured dv footage is about 1gb for every 4 minutes and
30 seconds.

3. for today's available bandwidth, it is impractical to be sending
that amount of data over our puny networks.

4. digital video projects are often done by editors all over the wold
working on the same projects.

5. they can do this a few different ways: a) sneakernet stuff b) work
off of low resolution copies of the real source material, and then
swap it out for the original high quality source materials at
"render/compile" time.

So for our project:

One of the ideas for editing the film is that ppl can use cinelerraCV
or some application to do editing work on low resolution versions of
footage which can be swapped out for higher resolution footage later
on.

We can do this because the project files saved from most video editing
suites can be saved as xml. And swapping out references to files is a
piece of cake.

We have high-res versions that match our lower-res "frame for frame".
All the high-res files have file names that are paired with the
low-res versions for easy swapping.

What would be really nice, from my perspective is to be able to say to
our editors "hey go on the internet archive and download you favorite
clips from the vast DTP collection and remix them. When you are done
upload your xml project file to us and we will re-render our high-res
footage according to your edits. Then we will post it back online for
you to watch / download."

I would like to make this feature available asap so our video editors
can use the internet archive DTP collection as their resource for
footage. This would allow our project to scale considerably. Scaling
would happen in terms of the number of editors available and the
amount of footage available to each editor. Resulting in an
exponential productivity increase.

Our front end is essentially set up. The internet archive is hosting
our clips.  The clips are available in several formats and sizes. The
search feature on the archive allows editors to find clips that are of
interest quickly and download a video in a format/size that makes
sense for them. The video editing software is easy to get and install.
XML project files are usually no larger than a few kb and can be sent
easily.

To achieve the goal of scaling in the way i described we would need:
- storage space of 5 terabytes
- a 1.8 MHz computer with 1gb of RAM for rendering.

This would be enough space to hold all of our source video. We need
this storage space so that when an editor sends in a project file we
can reach into our archive and pull out the clips needed to render the
editor's project. Giving editors the ability to edit footage using
lower-resolution files and the ability to submit edits via sending a
small xml file, is a goal that can not be achieved unless all of our
source is available for compiling those projects.

As of this moment we are backing up our edits on many external hdds.
these drives hold from 250-500gb each. Because of this we are unable
to render projects the way that we would like to. We are keeping very
good track of what is stored where, but at this point it seems we need
to move beyond our current scheme in order to leverage the potential
of all of our editors.

-j

On 10/25/07, Christian Einfeldt <einfeldt at gmail.com> wrote:
> hi
>
> On 10/24/07, jim stockford <jim at well.com> wrote:
> >
> > christian,
> >     would you describe, briefly, what you need for
> > storage for Digital Tipping Point? Mainly, can you
> > figure out how to work with a few hundred Gigabytes
> > of some storage that's internet accessible?
>
> Jim, thanks for asking.  I don't have the details to answer your question.
> That question is going to have to be answered by Jonathan Grindstaff, who
> wrote Jonathan's Video Glue (JVG), the name that we give to the BASH scripts
> that we use for automating certain processes for capturing, compressing,
> exporting and uploading video to the Internet Archive's Digital Tipping
> Point Video Collection (IA DTP VC).  You can see the IA DTP VC here:
>
>  http://archive.org/details/digitaltippingpoint
>
> You can see a flow chart here for Jonathan's tools:
>
> http://digitaltippingpoint.com/wiki/index.php?title=From_Tape_to_Archive
>
> Jonathan is the DTP point person on the kinds of questions that you are
> asking.  I have also cc'd Adam Doxtater (founder of Mad Penguin) and
> Margaret Aranyosi (co-founder of Bold Everything.com), who are our two sys
> admins for the DTP.  The stuff that you are talking about are not directly
> relevant to what Adam and Margaret are doing, since they admin the server
> that is running digitaltippingpoint.com, but they have been gracious enough
> to help us brainstorm in the past when we have had some deep sys admin type
> of issues, so I am just keeping them in the loop.  But the DTP raid storage
> solution that Jonathan will outline for you will be running locally
> somewhere, maybe even at the hacker hang-out that Kristian has been talking
> about.
>
> The idea is that in order for our DTP project to scale up, we are going to
> need to have more than just Christian Einfeldt having access to the
> rough-edited DV that is our "source code".  We are going to be emailing and
> FTPing around xml-type files with the edits for the "source code" and it
> will be useful for a few people to have commit privileges for that stuff,
> and also to access the actual raid that houses the data.
>
>
> > For
> > example, maybe upload the stuff you're currently
> > working on and/or the positively to-be-used stuff
> > and keep not-yet-edited and not-to-be-used stuff
> > elsewhere.
> >     just out of curiosity, where is your stuff stored now?
> > on tapes?
>
> Right now, the holy of holies (to use a maybe not so apt phrase) is the 450
> miniDV tapes that contain the interviews and b-roll that is our film.  Many
> of the interviewees spoke with us on the condition that portions of the
> footage not be publicized anywhere.  So we are having to cut out certain
> segments before we can load the footage onto the IA DTP VC for the world to
> play with.  Our eventual goal is to let the video kiddies play with our
> footage on Jumpcut and YouTube and Eyespot.  People will be grabbing our
> footage from the IA DTP VC and our Digital Tipping Point channel on YouTube
> and playing with it.  That will create interest in our film (hopefully) so
> that people will want to see the high res footage.
>
> We envision the high res footage eventually being released cinematically and
> on DVD.  I believe that in only a few years, the movie industry is going to
> switch to a business model that ensures high booking rates in the same way
> that the airline industry tries to keep its planes full.
>
> Currently, there is a crisis in the entire content industry.  Everything
> from books to music to movies to software is feeling the massive wave that
> we call FOSS.  YouTube and other similar content providers are putting up so
> much content on the Internet that movie studios and movie theaters can't
> keep their movies sold or their theaters full.  Even big bands are not
> selling CDs or legal downloads like they want.  Books are not being
> purchased.  In each of these industries, there will be one or two or three
> big hits per publishing house (studio, record label) per year, and those
> hits have to carry the rest of the publishing house's costs.
>
> So to keep the theaters full, some genius is going to come up with a
> combination of Digg.com and Fandango.com.  Both of those sites have a
> function of aggregating consumer interest, and Fandango sells tickets.  So
> the movie studios are going to fix it such that people can share video clips
> with their friends about interesting vids, and then coordinate with their
> friends to see a film.  As the interest in a film builds, it will be like
> interest in a flight building.  People will book tickets, except that rather
> than having to book a seat on a plane weeks or months in advance, people
> will book a seat in small little theaters that seat maybe 30 to 90 people
> for a screening.  The theaters will cut down on their losses by assuring a
> full house, and they will cut down on their costs by viral marketing to push
> the film and by showing big screen screenings of small films by indie
> producers.  They will, essentially, be big-screening YouTube!!
>
> As part of the marketing hype for both big studio productions like The
> Matrix and smaller things like The Blair Witch Project, the theaters will
> also allow consumers to vote for their favorite pre-roll trailers, both
> commercial trailers and trailers produced by YouTubers like Caitlin Hill,
> who is a popular teenager on YouTube.  So in the same way that now there are
> some major diggers who other diggers use as guide posts for what they are
> going to digg, in each major movie market there will be a few major diggers
> who gain prominence and will be able to swing the digg-fandango vote as to
> what indie and small video shorts will be on the pre-feature trailers.  In
> many cases, the theater-goers for any individual showing will be people who
> have very close demographic interests, and in many cases, they will largely
> know each other and be friends or members of the same club, like the Sierra
> club, etc.
>
> So, for example, you will have Sierra club members digging Al Gore's film
> "An Inconvenient Truth" (or something like it) and so on any given showing,
> the theater will enjoy a full house for say a maximum of three or four
> showings of "Inconvenient Truth" and then they will be done.  Since they are
> only going to be screening stuff that is available on the Internet anyway,
> their costs will be small, and their audiences will be small, but at least
> the theater will be full.
>
> We are already seeing theaters chop up their space into
> demographically-discreet segments, so that they can cater to adults, for
> example, who might want to be able to drink alcohol during a screening.
> (I'm not talking about XXX-rated films, just films oriented to mature
> audiences, as opposed to the teenager date films).
>
> Okay, so I digressed there, but the point is that we will have several
> layers to our film.  We will have the high-res footage that we will try to
> push through the above-described Digg-Fandango phenomenon; and we will have
> our low-res clips that will be circulating on YouTube and the IA DTP VC to
> push the finished products; and we will have multiple module directors who
> are putting together FOSS-oriented content for people who want to see small
> films about FOSS and the cool things that people are doing with FOSS.
>
> But as to your discreet question about our needs for raids, Jonathan is
> going to have to answer that question.
>