[sf-lug] File transfer technologies

[Jeff Bragg]

Does anyone have any hard data about the relative speeds file transfer
technologies?  I'm particularly interested in empirical data relating to
wget vs rsync vs scp.  This would be applied to an internal network, so
latency isn't really an issue.  The goal is to determine the best way to
transfer largish tarballs (size varies, but is typically in the 10s of GBs)
in the most efficient way in terms of both time and machine resources.  At
the moment we're using wget, and in my (anecdotal) experience wget performed
at least as well as rsync, typically, at least in terms of speed, and
probably better than rsync in terms of machine resources (that initial
calculation of the job size, etc can be a little intense when you're moving
100-1000+ GB).  My experience with scp in this sphere is more limited.  I
really only use it to transfer smallish groups of files of much less girth
than the ones previously mentioned, so have never had to push enough through
it to get a feel for speed or resources required.

Anyone have any test data here?  Or even just opinions?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://linuxmafia.com/pipermail/sf-lug/attachments/20070921/c57ef8f6/attachment.html>