[sf-lug] (no subject)
jim at well.com
Thu Oct 25 22:47:25 PDT 2007
thank you. i'd like to help, but at this point i can
offer only a small fraction of the 5TB you need.
i'll keep trying, maybe i can come up with
On Oct 25, 2007, at 9:35 PM, Jonathan Grindstaff wrote:
> Before i get started, i'll just take a minute to outline some points.
> 1. digital video loses quality really fast when it is re-rendered.
> This was recently likened to re-faxing something over and over. At
> first this might seem odd because, of course, "digitally copying a
> file a million times won't change it". And that is true, copying the
> video does not change it.
> But when working with video we are not always just copying . . .
> sometimes we want to blend two clips together for a nice fade effect
> etc. This means that the two video clips have to be merged together to
> form a new image. . . . the reason dv looks worse and worse when you
> work with it is that almost all digital video is _compressed_ in some
> way. The deterioration of visual quality then is an issue of
> re-compression. When video gets re-compressed, it changes . . . most
> ppl think it's for the worse . . . some times i think it looks funky
> and cool in an "avant garde" sort of way . . . . but that is way, way
> beside the point.
> 2. the "raw" captured dv footage is about 1gb for every 4 minutes and
> 30 seconds.
> 3. for today's available bandwidth, it is impractical to be sending
> that amount of data over our puny networks.
> 4. digital video projects are often done by editors all over the wold
> working on the same projects.
> 5. they can do this a few different ways: a) sneakernet stuff b) work
> off of low resolution copies of the real source material, and then
> swap it out for the original high quality source materials at
> "render/compile" time.
> So for our project:
> One of the ideas for editing the film is that ppl can use cinelerraCV
> or some application to do editing work on low resolution versions of
> footage which can be swapped out for higher resolution footage later
> We can do this because the project files saved from most video editing
> suites can be saved as xml. And swapping out references to files is a
> piece of cake.
> We have high-res versions that match our lower-res "frame for frame".
> All the high-res files have file names that are paired with the
> low-res versions for easy swapping.
> What would be really nice, from my perspective is to be able to say to
> our editors "hey go on the internet archive and download you favorite
> clips from the vast DTP collection and remix them. When you are done
> upload your xml project file to us and we will re-render our high-res
> footage according to your edits. Then we will post it back online for
> you to watch / download."
> I would like to make this feature available asap so our video editors
> can use the internet archive DTP collection as their resource for
> footage. This would allow our project to scale considerably. Scaling
> would happen in terms of the number of editors available and the
> amount of footage available to each editor. Resulting in an
> exponential productivity increase.
> Our front end is essentially set up. The internet archive is hosting
> our clips. The clips are available in several formats and sizes. The
> search feature on the archive allows editors to find clips that are of
> interest quickly and download a video in a format/size that makes
> sense for them. The video editing software is easy to get and install.
> XML project files are usually no larger than a few kb and can be sent
> To achieve the goal of scaling in the way i described we would need:
> - storage space of 5 terabytes
> - a 1.8 MHz computer with 1gb of RAM for rendering.
> This would be enough space to hold all of our source video. We need
> this storage space so that when an editor sends in a project file we
> can reach into our archive and pull out the clips needed to render the
> editor's project. Giving editors the ability to edit footage using
> lower-resolution files and the ability to submit edits via sending a
> small xml file, is a goal that can not be achieved unless all of our
> source is available for compiling those projects.
> As of this moment we are backing up our edits on many external hdds.
> these drives hold from 250-500gb each. Because of this we are unable
> to render projects the way that we would like to. We are keeping very
> good track of what is stored where, but at this point it seems we need
> to move beyond our current scheme in order to leverage the potential
> of all of our editors.
> On 10/25/07, Christian Einfeldt <einfeldt at gmail.com> wrote:
>> On 10/24/07, jim stockford <jim at well.com> wrote:
>>> would you describe, briefly, what you need for
>>> storage for Digital Tipping Point? Mainly, can you
>>> figure out how to work with a few hundred Gigabytes
>>> of some storage that's internet accessible?
>> Jim, thanks for asking. I don't have the details to answer your
>> That question is going to have to be answered by Jonathan Grindstaff,
>> wrote Jonathan's Video Glue (JVG), the name that we give to the BASH
>> that we use for automating certain processes for capturing,
>> exporting and uploading video to the Internet Archive's Digital
>> Point Video Collection (IA DTP VC). You can see the IA DTP VC here:
>> You can see a flow chart here for Jonathan's tools:
>> Jonathan is the DTP point person on the kinds of questions that you
>> asking. I have also cc'd Adam Doxtater (founder of Mad Penguin) and
>> Margaret Aranyosi (co-founder of Bold Everything.com), who are our
>> two sys
>> admins for the DTP. The stuff that you are talking about are not
>> relevant to what Adam and Margaret are doing, since they admin the
>> that is running digitaltippingpoint.com, but they have been gracious
>> to help us brainstorm in the past when we have had some deep sys
>> admin type
>> of issues, so I am just keeping them in the loop. But the DTP raid
>> solution that Jonathan will outline for you will be running locally
>> somewhere, maybe even at the hacker hang-out that Kristian has been
>> The idea is that in order for our DTP project to scale up, we are
>> going to
>> need to have more than just Christian Einfeldt having access to the
>> rough-edited DV that is our "source code". We are going to be
>> emailing and
>> FTPing around xml-type files with the edits for the "source code" and
>> will be useful for a few people to have commit privileges for that
>> and also to access the actual raid that houses the data.
>>> example, maybe upload the stuff you're currently
>>> working on and/or the positively to-be-used stuff
>>> and keep not-yet-edited and not-to-be-used stuff
>>> just out of curiosity, where is your stuff stored now?
>>> on tapes?
>> Right now, the holy of holies (to use a maybe not so apt phrase) is
>> the 450
>> miniDV tapes that contain the interviews and b-roll that is our film.
>> of the interviewees spoke with us on the condition that portions of
>> footage not be publicized anywhere. So we are having to cut out
>> segments before we can load the footage onto the IA DTP VC for the
>> world to
>> play with. Our eventual goal is to let the video kiddies play with
>> footage on Jumpcut and YouTube and Eyespot. People will be grabbing
>> footage from the IA DTP VC and our Digital Tipping Point channel on
>> and playing with it. That will create interest in our film
>> (hopefully) so
>> that people will want to see the high res footage.
>> We envision the high res footage eventually being released
>> cinematically and
>> on DVD. I believe that in only a few years, the movie industry is
>> going to
>> switch to a business model that ensures high booking rates in the
>> same way
>> that the airline industry tries to keep its planes full.
>> Currently, there is a crisis in the entire content industry.
>> from books to music to movies to software is feeling the massive wave
>> we call FOSS. YouTube and other similar content providers are
>> putting up so
>> much content on the Internet that movie studios and movie theaters
>> keep their movies sold or their theaters full. Even big bands are not
>> selling CDs or legal downloads like they want. Books are not being
>> purchased. In each of these industries, there will be one or two or
>> big hits per publishing house (studio, record label) per year, and
>> hits have to carry the rest of the publishing house's costs.
>> So to keep the theaters full, some genius is going to come up with a
>> combination of Digg.com and Fandango.com. Both of those sites have a
>> function of aggregating consumer interest, and Fandango sells
>> tickets. So
>> the movie studios are going to fix it such that people can share
>> video clips
>> with their friends about interesting vids, and then coordinate with
>> friends to see a film. As the interest in a film builds, it will be
>> interest in a flight building. People will book tickets, except that
>> than having to book a seat on a plane weeks or months in advance,
>> will book a seat in small little theaters that seat maybe 30 to 90
>> for a screening. The theaters will cut down on their losses by
>> assuring a
>> full house, and they will cut down on their costs by viral marketing
>> to push
>> the film and by showing big screen screenings of small films by indie
>> producers. They will, essentially, be big-screening YouTube!!
>> As part of the marketing hype for both big studio productions like The
>> Matrix and smaller things like The Blair Witch Project, the theaters
>> also allow consumers to vote for their favorite pre-roll trailers,
>> commercial trailers and trailers produced by YouTubers like Caitlin
>> who is a popular teenager on YouTube. So in the same way that now
>> there are
>> some major diggers who other diggers use as guide posts for what they
>> going to digg, in each major movie market there will be a few major
>> who gain prominence and will be able to swing the digg-fandango vote
>> as to
>> what indie and small video shorts will be on the pre-feature
>> trailers. In
>> many cases, the theater-goers for any individual showing will be
>> people who
>> have very close demographic interests, and in many cases, they will
>> know each other and be friends or members of the same club, like the
>> club, etc.
>> So, for example, you will have Sierra club members digging Al Gore's
>> "An Inconvenient Truth" (or something like it) and so on any given
>> the theater will enjoy a full house for say a maximum of three or four
>> showings of "Inconvenient Truth" and then they will be done. Since
>> they are
>> only going to be screening stuff that is available on the Internet
>> their costs will be small, and their audiences will be small, but at
>> the theater will be full.
>> We are already seeing theaters chop up their space into
>> demographically-discreet segments, so that they can cater to adults,
>> example, who might want to be able to drink alcohol during a
>> (I'm not talking about XXX-rated films, just films oriented to mature
>> audiences, as opposed to the teenager date films).
>> Okay, so I digressed there, but the point is that we will have several
>> layers to our film. We will have the high-res footage that we will
>> try to
>> push through the above-described Digg-Fandango phenomenon; and we
>> will have
>> our low-res clips that will be circulating on YouTube and the IA DTP
>> VC to
>> push the finished products; and we will have multiple module
>> directors who
>> are putting together FOSS-oriented content for people who want to see
>> films about FOSS and the cool things that people are doing with FOSS.
>> But as to your discreet question about our needs for raids, Jonathan
>> going to have to answer that question.
More information about the sf-lug