[sf-lug] Linux backup software .. that meets unique requirements

[David Rosenstrauch]

Well, (IIUC) the approach that duplicity takes solves that problem 
nicely.  It stores file fingerprint hashes on the remote site.  Then, at 
the next backup, it retrieves them and compares the actual files to see 
if they've changed.  This allows it to work in an rsync-like fashion, 
even if the files are encrypted remotely.  So something that works in a 
duplicity-like style would fit the bill.

(Of course duplicity itself *doesn't* fit it, unfortunately, since it 
has the irritating requirement of having to periodically go back and 
make - and upload - a fresh full backup.)

Secrecy at the remote end is definitely very important, by the way, as 
it's a shared server, which I don't own.

Thanks,

DR

On 03/11/2010 02:38 PM, David Hinkle wrote:
> The primary technical hurdle you face here is that good encryption by
> definition is going to change the entire output file for any minor
> change in the input file.   That means if you encrypt localy then use
> rsync or a similar algorithm to transmit you must transmit any
> changed file in it's entirety.   If you have many small files, each
> independently encrypted this might be fine.
>
> If you encrypt at the remote end, (even if you use an SSH transport
> for intermediary) you get to make best used of the rsync algorithm by
> only transmitting changes, but you loose perfect secrecy because the
> remote end could theoretically intercept your un-encrypted data after
> decrypting from the tunnel but before encrypting to the disk.
>
> You have a couple options, which one is best depends on how important
> your perfect secrecy is to you, and what your files look like.  Can
> you tell us more about the scenario?   I've got a great deal of
> experience with rsync based backup solutions so I can probably help
> you out.  You can implement either of the above with a good script.
>
> David
>
> -----Original Message----- From: sf-lug-bounces at linuxmafia.com
> [mailto:sf-lug-bounces at linuxmafia.com] On Behalf Of David
> Rosenstrauch Sent: Thursday, March 11, 2010 1:17 PM To: sf-lug
> Subject: [sf-lug] Linux backup software .. that meets unique
> requirements
>
> Trying to find linux backup software that meets what apparently is a
> unique set of requirements.  I've checked out all the major backup
> apps (backuppc, duplicity, bacula, drop.io, etc.) but none of them
> seem to meet all of my requirements.  Anyone here know of one that
> does?  Here's what I'm looking for:
>
> 1) Backs up efficiently.  e.g., uses rsync algorithm or similar so as
> to not repeatedly backup/store files that haven't changed.
>
> 2) Transmission of backups is encrypted.
>
> 3) Storage of backups is encrypted.
>
> 4) Can back up to offsite/remote file system.
>
> 5) Can back up to any old plain vanilla sftp/rsync-accessible remote
> file system.  (Reason behind this requirement is that I already have
> access to a large amount of (non-encrypted) space for backups, and
> don't want to pay extra for another service.)
>
> 6) Does not require that you periodically take the network i/o
> storage hit of making a new full backup.  (This is not an option, as
> what the data I want to back up is getting large, and I only want to
> do it once.)
>
> For some reason it's impossible to find something that fits all of
> the above.
>
> * backuppc and rsyn don't satisfy #3.
>
> * duplicity doesn't satisfy #6.
>
> * drop.io and bacula don't satisfy #5.
>
> etc.
>
>
> Anyone have any suggestions here?  I'm actually on the verge of
> writing my own backup software to do this, but I'd *really* like to
> avoid that unless there's no other choice.
>
>
> Thanks,
>
> DR