[sf-lug] File transfer technologies

[Tom Haddon]

On Fri, 2007-09-21 at 09:37 -0700, Jeff Bragg wrote:
> Does anyone have any hard data about the relative speeds file transfer
> technologies?  I'm particularly interested in empirical data relating
> to wget vs rsync vs scp.  This would be applied to an internal
> network, so latency isn't really an issue.  The goal is to determine
> the best way to transfer largish tarballs (size varies, but is
> typically in the 10s of GBs) in the most efficient way in terms of
> both time and machine resources.  At the moment we're using wget, and
> in my (anecdotal) experience wget performed at least as well as rsync,
> typically, at least in terms of speed, and probably better than rsync
> in terms of machine resources (that initial calculation of the job
> size, etc can be a little intense when you're moving 100-1000+ GB).
> My experience with scp in this sphere is more limited.  I really only
> use it to transfer smallish groups of files of much less girth than
> the ones previously mentioned, so have never had to push enough
> through it to get a feel for speed or resources required. 

If you're talking about small quantities of large files in a one time
transfer (i.e. the destination doesn't have any of the files you're
copying), I think the KISS principle applies - for instance, I know ftp
is significantly faster than scp in this situation (scp has some
overhead associated with the encryption) and rsync would most likely be
the same - the extra features of rsync (comparing source and destination
and only transferring differences) won't gain you anything here.

Sounds like wget/ftp is the way to go.

Don't have hard data on this, but a fair amount of experience...

Cheers, Tom

> 
> Anyone have any test data here?  Or even just opinions?
> _______________________________________________
> sf-lug mailing list
> sf-lug at linuxmafia.com
> http://linuxmafia.com/mailman/listinfo/sf-lug
-- 


----------------------------------
Tom Haddon
mailto:tom at greenleaftech.net
m +1.415.871.4180
www.greenleaftech.net