[conspire] pitfalls of transfers on a usb. memory stick

[Rick Moen]

Quoting Bruce Coston (jane_ikari at yahoo.com):

> Pardus linux does not tend to see a stick unless  formatted fat32 .
> At 900k , it takes 5 hrs. to transfer 15G. from my 32G. stick . I can
> easily take 2 nights to transfer my 40G. of videos . I still had
> problems with one file stopping the process despite my using different
> permissions to do the transfer , I just bundled it into an archive .
> So I will examine the questions around local network configuration for
> file transfers . Do other people have this much trouble with usb.
> sticks ? 

My immediate reaction is that that's a stupendously large number of
bytes to try to move around via USB flash sticks.  I mean, I understand
that the fact that they now come routinely and cheaply in 32 GB sizes 
does seem to invite you to try, but, c'mon.  If nothing else, that's
going to be _so_ immensely slow.

The largest amount of data I ever moved that way was a 13 GB file
collection that I carefully divvied up between an 8GB stick, a 4 GB 
one, and a 1 GB one -- and I did it that way because that's what I had
in my pocket and had all day long to copy the files there while
accomplishing other things.  Which is A Very Good Thing, because the 
copying did take hours.  Not two nights, though, or even one.  Your 5
hours to copy 15 GB is in the ballpark, though.

So:  They're pretty slow.  Put a hard drive into a USB/Firewire/eSata
case if you want to get serious about moving that amount of data via
physical media -- or, if physical media aren't required, string an
ethernet cable between the hosts and use your choice of network copying
methods.  On a straight ethernet run, nc (netcat) without even bothering
to lavish CPU and RAM on compression might be fastest:

Destination host:  nc -l -p 2342 | tar -C /target/dir -x -
Source host:       tar -c /source/dir | nc Target_Host 2342

This general method can be improved using a nifty utility called
pipeviewer (pv):
http://www.anthonyldechiaro.com/blog/2009/05/14/efficient-file-copying-with-pipeviewer-and-netcat/