[sf-lug] (no subject)

[jim stockford]

thank you. i'd like to help, but at this point i can
offer only a small fraction of the 5TB you need.
i'll keep trying, maybe i can come up with
something.

On Oct 25, 2007, at 9:35 PM, Jonathan Grindstaff wrote:

> Before i get started, i'll just take a minute to outline some points.
>
> 1. digital video loses quality really fast when it is re-rendered.
> This was recently likened to re-faxing something over and over. At
> first this might seem odd because, of course, "digitally copying a
> file a million times won't change it". And that is true, copying the
> video does not change it.
> But when working with video we are not always just copying . . .
> sometimes we want to blend two clips together for a nice fade effect
> etc. This means that the two video clips have to be merged together to
> form a new image. . . . the reason dv looks worse and worse when you
> work with it is that almost all digital video is _compressed_ in some
> way. The deterioration of visual quality then is an issue of
> re-compression. When video gets re-compressed, it changes . . . most
> ppl think it's for the worse . . . some times i think it looks funky
> and cool in an "avant garde" sort of way . . . . but that is way, way
> beside the point.
>
> 2. the "raw" captured dv footage is about 1gb for every 4 minutes and
> 30 seconds.
>
> 3. for today's available bandwidth, it is impractical to be sending
> that amount of data over our puny networks.
>
> 4. digital video projects are often done by editors all over the wold
> working on the same projects.
>
> 5. they can do this a few different ways: a) sneakernet stuff b) work
> off of low resolution copies of the real source material, and then
> swap it out for the original high quality source materials at
> "render/compile" time.
>
> So for our project:
>
> One of the ideas for editing the film is that ppl can use cinelerraCV
> or some application to do editing work on low resolution versions of
> footage which can be swapped out for higher resolution footage later
> on.
>
> We can do this because the project files saved from most video editing
> suites can be saved as xml. And swapping out references to files is a
> piece of cake.
>
> We have high-res versions that match our lower-res "frame for frame".
> All the high-res files have file names that are paired with the
> low-res versions for easy swapping.
>
> What would be really nice, from my perspective is to be able to say to
> our editors "hey go on the internet archive and download you favorite
> clips from the vast DTP collection and remix them. When you are done
> upload your xml project file to us and we will re-render our high-res
> footage according to your edits. Then we will post it back online for
> you to watch / download."
>
> I would like to make this feature available asap so our video editors
> can use the internet archive DTP collection as their resource for
> footage. This would allow our project to scale considerably. Scaling
> would happen in terms of the number of editors available and the
> amount of footage available to each editor. Resulting in an
> exponential productivity increase.
>
> Our front end is essentially set up. The internet archive is hosting
> our clips.  The clips are available in several formats and sizes. The
> search feature on the archive allows editors to find clips that are of
> interest quickly and download a video in a format/size that makes
> sense for them. The video editing software is easy to get and install.
> XML project files are usually no larger than a few kb and can be sent
> easily.
>
> To achieve the goal of scaling in the way i described we would need:
> - storage space of 5 terabytes
> - a 1.8 MHz computer with 1gb of RAM for rendering.
>
> This would be enough space to hold all of our source video. We need
> this storage space so that when an editor sends in a project file we
> can reach into our archive and pull out the clips needed to render the
> editor's project. Giving editors the ability to edit footage using
> lower-resolution files and the ability to submit edits via sending a
> small xml file, is a goal that can not be achieved unless all of our
> source is available for compiling those projects.
>
> As of this moment we are backing up our edits on many external hdds.
> these drives hold from 250-500gb each. Because of this we are unable
> to render projects the way that we would like to. We are keeping very
> good track of what is stored where, but at this point it seems we need
> to move beyond our current scheme in order to leverage the potential
> of all of our editors.
>
> -j
>
> On 10/25/07, Christian Einfeldt <einfeldt at gmail.com> wrote:
>> hi
>>
>> On 10/24/07, jim stockford <jim at well.com> wrote:
>>>
>>> christian,
>>>     would you describe, briefly, what you need for
>>> storage for Digital Tipping Point? Mainly, can you
>>> figure out how to work with a few hundred Gigabytes
>>> of some storage that's internet accessible?
>>
>> Jim, thanks for asking.  I don't have the details to answer your  
>> question.
>> That question is going to have to be answered by Jonathan Grindstaff,  
>> who
>> wrote Jonathan's Video Glue (JVG), the name that we give to the BASH  
>> scripts
>> that we use for automating certain processes for capturing,  
>> compressing,
>> exporting and uploading video to the Internet Archive's Digital  
>> Tipping
>> Point Video Collection (IA DTP VC).  You can see the IA DTP VC here:
>>
>>  http://archive.org/details/digitaltippingpoint
>>
>> You can see a flow chart here for Jonathan's tools:
>>
>> http://digitaltippingpoint.com/wiki/index.php? 
>> title=From_Tape_to_Archive
>>
>> Jonathan is the DTP point person on the kinds of questions that you  
>> are
>> asking.  I have also cc'd Adam Doxtater (founder of Mad Penguin) and
>> Margaret Aranyosi (co-founder of Bold Everything.com), who are our  
>> two sys
>> admins for the DTP.  The stuff that you are talking about are not  
>> directly
>> relevant to what Adam and Margaret are doing, since they admin the  
>> server
>> that is running digitaltippingpoint.com, but they have been gracious  
>> enough
>> to help us brainstorm in the past when we have had some deep sys  
>> admin type
>> of issues, so I am just keeping them in the loop.  But the DTP raid  
>> storage
>> solution that Jonathan will outline for you will be running locally
>> somewhere, maybe even at the hacker hang-out that Kristian has been  
>> talking
>> about.
>>
>> The idea is that in order for our DTP project to scale up, we are  
>> going to
>> need to have more than just Christian Einfeldt having access to the
>> rough-edited DV that is our "source code".  We are going to be  
>> emailing and
>> FTPing around xml-type files with the edits for the "source code" and  
>> it
>> will be useful for a few people to have commit privileges for that  
>> stuff,
>> and also to access the actual raid that houses the data.
>>
>>
>>> For
>>> example, maybe upload the stuff you're currently
>>> working on and/or the positively to-be-used stuff
>>> and keep not-yet-edited and not-to-be-used stuff
>>> elsewhere.
>>>     just out of curiosity, where is your stuff stored now?
>>> on tapes?
>>
>> Right now, the holy of holies (to use a maybe not so apt phrase) is  
>> the 450
>> miniDV tapes that contain the interviews and b-roll that is our film.  
>>  Many
>> of the interviewees spoke with us on the condition that portions of  
>> the
>> footage not be publicized anywhere.  So we are having to cut out  
>> certain
>> segments before we can load the footage onto the IA DTP VC for the  
>> world to
>> play with.  Our eventual goal is to let the video kiddies play with  
>> our
>> footage on Jumpcut and YouTube and Eyespot.  People will be grabbing  
>> our
>> footage from the IA DTP VC and our Digital Tipping Point channel on  
>> YouTube
>> and playing with it.  That will create interest in our film  
>> (hopefully) so
>> that people will want to see the high res footage.
>>
>> We envision the high res footage eventually being released  
>> cinematically and
>> on DVD.  I believe that in only a few years, the movie industry is  
>> going to
>> switch to a business model that ensures high booking rates in the  
>> same way
>> that the airline industry tries to keep its planes full.
>>
>> Currently, there is a crisis in the entire content industry.   
>> Everything
>> from books to music to movies to software is feeling the massive wave  
>> that
>> we call FOSS.  YouTube and other similar content providers are  
>> putting up so
>> much content on the Internet that movie studios and movie theaters  
>> can't
>> keep their movies sold or their theaters full.  Even big bands are not
>> selling CDs or legal downloads like they want.  Books are not being
>> purchased.  In each of these industries, there will be one or two or  
>> three
>> big hits per publishing house (studio, record label) per year, and  
>> those
>> hits have to carry the rest of the publishing house's costs.
>>
>> So to keep the theaters full, some genius is going to come up with a
>> combination of Digg.com and Fandango.com.  Both of those sites have a
>> function of aggregating consumer interest, and Fandango sells  
>> tickets.  So
>> the movie studios are going to fix it such that people can share  
>> video clips
>> with their friends about interesting vids, and then coordinate with  
>> their
>> friends to see a film.  As the interest in a film builds, it will be  
>> like
>> interest in a flight building.  People will book tickets, except that  
>> rather
>> than having to book a seat on a plane weeks or months in advance,  
>> people
>> will book a seat in small little theaters that seat maybe 30 to 90  
>> people
>> for a screening.  The theaters will cut down on their losses by  
>> assuring a
>> full house, and they will cut down on their costs by viral marketing  
>> to push
>> the film and by showing big screen screenings of small films by indie
>> producers.  They will, essentially, be big-screening YouTube!!
>>
>> As part of the marketing hype for both big studio productions like The
>> Matrix and smaller things like The Blair Witch Project, the theaters  
>> will
>> also allow consumers to vote for their favorite pre-roll trailers,  
>> both
>> commercial trailers and trailers produced by YouTubers like Caitlin  
>> Hill,
>> who is a popular teenager on YouTube.  So in the same way that now  
>> there are
>> some major diggers who other diggers use as guide posts for what they  
>> are
>> going to digg, in each major movie market there will be a few major  
>> diggers
>> who gain prominence and will be able to swing the digg-fandango vote  
>> as to
>> what indie and small video shorts will be on the pre-feature  
>> trailers.  In
>> many cases, the theater-goers for any individual showing will be  
>> people who
>> have very close demographic interests, and in many cases, they will  
>> largely
>> know each other and be friends or members of the same club, like the  
>> Sierra
>> club, etc.
>>
>> So, for example, you will have Sierra club members digging Al Gore's  
>> film
>> "An Inconvenient Truth" (or something like it) and so on any given  
>> showing,
>> the theater will enjoy a full house for say a maximum of three or four
>> showings of "Inconvenient Truth" and then they will be done.  Since  
>> they are
>> only going to be screening stuff that is available on the Internet  
>> anyway,
>> their costs will be small, and their audiences will be small, but at  
>> least
>> the theater will be full.
>>
>> We are already seeing theaters chop up their space into
>> demographically-discreet segments, so that they can cater to adults,  
>> for
>> example, who might want to be able to drink alcohol during a  
>> screening.
>> (I'm not talking about XXX-rated films, just films oriented to mature
>> audiences, as opposed to the teenager date films).
>>
>> Okay, so I digressed there, but the point is that we will have several
>> layers to our film.  We will have the high-res footage that we will  
>> try to
>> push through the above-described Digg-Fandango phenomenon; and we  
>> will have
>> our low-res clips that will be circulating on YouTube and the IA DTP  
>> VC to
>> push the finished products; and we will have multiple module  
>> directors who
>> are putting together FOSS-oriented content for people who want to see  
>> small
>> films about FOSS and the cool things that people are doing with FOSS.
>>
>> But as to your discreet question about our needs for raids, Jonathan  
>> is
>> going to have to answer that question.
>>
>