[sf-lug] File transfer technologies
tom at greenleaftech.net
Fri Sep 21 09:54:36 PDT 2007
On Fri, 2007-09-21 at 09:37 -0700, Jeff Bragg wrote:
> Does anyone have any hard data about the relative speeds file transfer
> technologies? I'm particularly interested in empirical data relating
> to wget vs rsync vs scp. This would be applied to an internal
> network, so latency isn't really an issue. The goal is to determine
> the best way to transfer largish tarballs (size varies, but is
> typically in the 10s of GBs) in the most efficient way in terms of
> both time and machine resources. At the moment we're using wget, and
> in my (anecdotal) experience wget performed at least as well as rsync,
> typically, at least in terms of speed, and probably better than rsync
> in terms of machine resources (that initial calculation of the job
> size, etc can be a little intense when you're moving 100-1000+ GB).
> My experience with scp in this sphere is more limited. I really only
> use it to transfer smallish groups of files of much less girth than
> the ones previously mentioned, so have never had to push enough
> through it to get a feel for speed or resources required.
If you're talking about small quantities of large files in a one time
transfer (i.e. the destination doesn't have any of the files you're
copying), I think the KISS principle applies - for instance, I know ftp
is significantly faster than scp in this situation (scp has some
overhead associated with the encryption) and rsync would most likely be
the same - the extra features of rsync (comparing source and destination
and only transferring differences) won't gain you anything here.
Sounds like wget/ftp is the way to go.
Don't have hard data on this, but a fair amount of experience...
> Anyone have any test data here? Or even just opinions?
> sf-lug mailing list
> sf-lug at linuxmafia.com
mailto:tom at greenleaftech.net
More information about the sf-lug