File Synchronisation Tools

Mohammad Jaudet, Jaroslaw Polock

This document describes information about file synchronisation tools, which can be used at CERN to synchronise user directories on different Linux machines, possibly a desktop and a laptop. It is also available in multiple formats (HTML, PDF, PS, DVI and TEX)

1. Tools

unison, FAQ, DL, Manual-Page, home-page, Tutorial.
rsync, FAQ, DL, Manual-Page, home-page, Tutorial ( 1, 2 )

Few file-synchronisation tools which appear only in comaparison and conclusion were also tested and their details are described in other file-synchronisation tools separately.

2. Comparison

rsync is a mirroring tool and needs to be told ``this replica contains the true versions of all the files in it; please make that replica look exactly the same. '' It is rather complicated to use and requires a user friendly front-end.

unison is a synchroniser and is capable of recognising updates. It can be used to keep a remote replica of a directory structure up to date with a central repository and is capable of propagating updates in both directions and recognising conflicting updates .It uses the same ``rsync algorithm'' which streamlines updates of small parts of large files by transferring only the parts that have changed.

cvs is a full-blown version control system, and it has lots of other features (version history, multiple branches, etc.). It can also be used to keep a remote replica of a directory structure up to date with a central repository and is capable of propagating updates in both directions and recognising conflicting updates like unison. cvs also uses the rsync protocol for file transfer and requires manual notification whenever files are added or deleted. Moving files in cvs is , however, a bit tricky business.

Xfiles is set of independents Java based graphical applications for file synchronisation.

fsync is Perl script for files synchronisation on remote machines.

drsync is a front-end to rsync.

3. Comparison Table

Property unison rsync cvs xfiles fsync drsync

gui yes no no yes no no

slow conn. supported yes yes yes yes yes yes

ssh/rsh required yes yes no yes ² yes yes

automatic update with diff yes yes yes yes yes yes

dry run support uk ¹ yes yes uk ¹ yes yes

Profiles yes yes yes yes yes yes

file selection pro ⁴ cl ³ uk ¹ uk ¹ pro ⁴ cl ³

file ignore no cl cl ³ cl ³ pro ⁴ cl ³

Scripting for RCS yes no yes yes no no

MS Windows support yes yes yes yes uk ¹ uk ¹

availability of CERN distribution yes yes yes no yes yes

¹ means, the data is not known/confirmed right now.
² just to start remote server and is optional.
³ means, option via command line.
⁴ optional via profile

4. Conclusion

unison is a recommended solution as it provides all most all the necessary options for synchronisation of home directories and is widely used in most of organisation and institutes such as Edinburgh University, etc.

The first time synchronisation takes time but subsequent one are relatively very fast. unison took approximately 13-15 minutes in scanning and then approximately 15 for copying while synchronising an AFS directory sized 570MB with the local directory. Both systems used for test were PIII 800 and with some what different memory configurations.

5. unison

Unison is a file-synchronisation tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other. Unison shares a number of features with tools such as configuration management packages (CVS, PRCS, etc.) distributed file-systems (Coda, etc.) uni-directional mirroring utilities (rsync, etc.) and other synchronisers (Intellisync, Reconcile, etc).

Unison provides both textual and graphical user interfaces.

5.1 Installation

unison must be installed on both systems and requires ssh or rsh on both machines.


  # cd /afs/cern.ch/project/linux/l2k/nomadic/file-sync

  # rpm -ivh unison-2.6.11-1.i386.rpm

5.2 Usage

5.2.1 While working on laptop machine connected to CERN without AFS

Synchronising the laptop with your desktop using rsh.


  $ unison ssh://pcdis25/~/sub-dir /home/jaudet/sub-dir -ui graphics \
     -group -owner -killServer -times

Updating the files on desktop after modifications on the laptop machine.


  $ unison /home/jaudet/sub-dir ssh://pcdis25/~/sub-dir -ui graphics \
    -group -owner -killServer -times

Note: You don't need to use a remote host name if a laptop in connected to CERN network with AFS running. Just specify the directory names.

5.3 unison syntax and options used in examples


Usage: unison [SRC] [[<rsh|ssh><:>//HOST</|//>DEST] [OPTIONS]
       // means absolute path.

Option Description

-group synchronise owner

-owner synchronise group

-ui xxx type of user interface ('text' or 'graphic')

-killServer kill server when done (even when using sockets)

-times synchronise modification times

For ignoring files through profiles, please visit this link.

6. rsync

rsync is a replacement for rcp that has many more features. rsync uses the "rsync algorithm" which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. At first glance this may seem impossible because the calculation of diffs between two files normally requires local access to both files.

6.1 Installation

rsync must be installed on both systems and requires ssh or rsh on both machines.

rsync is available on CERN Redhat 6.1 machines through ASIS. Check for GNU.DVP/rsync-2.4.3 if it is not there.

To install the latest version of rsync, use the following commands:


  # cd /afs/cern.ch/project/linux/l2k/nomadic/file-sync

  # rpm -ivh rsync-2.4.6-1.i386.rpm

6.2 Usage

6.2.1 While working on laptop machine connected to CERN without AFS

Synchronising the laptop with your desktop using ssh.


  $ rsync -v -c -r -u -R -l -H -P -g -t -Z --progress --rsh=ssh \
    pcdis25:~/sub-dir /home/jaudet/subdir

Updating the files on desktop after modifications on the laptop machine.


  $ rsync -v -c -r -u -R -l -H -P -g -t -Z --progress --rsh=ssh \
    /home/jaudet/sub-dir pcdis25:~/sub-dir

Use -n option to know what would have been transferred. rsync uses rsh by default, use -e ssh to use ssh rather then rsh. To use absolute paths on the remote machine use :: instead of : after the host-name.

Note: You don't need to use a remote host name if a laptop in connected to CERN network with AFS running. Just specify the directory names.

6.3 rsync syntax and options used in examples


Usage: rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... [USER@]HOST:SRC DEST
  or   rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
SRC on single-colon remote HOST will be expanded by remote shell
SRC on server remote HOST may contain shell wildcards or multiple
    sources separated by space as long as they have same top-level

Option Description

-v, -verbose increase verbosity

-c, -checksum always checksum

-r, -recursive recurse into directories

-u, -update update only (don't overwrite newer files)

-R, -relative use relative path names

-l, -links preserve soft links

-H, -hard-links preserve hard links

-p, -perms preserve permissions

-g, -group preserve group

-t, -times preserve times

-n, -dry-run show what would have been transferred

-z, -compress compress file data

-progressshow progress during transfer

-e, -rsh=COMMAND specify rsh replacement

-rsync-path=PATH specify path to rsync on the remote machine

-exclude=PATTERN exclude files matching PATTERN

-exclude-from=FILE exclude patterns listed in FILE

-include=PATTERN don't exclude files matching PATTERN

-include-from=FILE don't exclude patterns listed in FILE

About this document ...

File Synchronisation Tools

This document was generated using the LaTeX2HTML translator Version 99.1 release (March 30, 1999)

The command line arguments were:
latex2html -html_version 3.2,math -scalable_fonts -no_navigation -split 0 -show_section_numbers -dir . ftools.tex

The translation was initiated by Mohammad Jaudet on 2001-06-28

Mohammad Jaudet
2001-06-28

`Property`	`unison`	`rsync`	`cvs`	`xfiles`	`fsync`	`drsync`
`gui`	`yes`	`no`	`no`	`yes`	`no`	`no`
`slow conn. supported`	`yes`	`yes`	`yes`	`yes`	`yes`	`yes`
`ssh/rsh required`	`yes`	`yes`	`no`	`yes ²`	`yes`	`yes`
`automatic update with diff`	`yes`	`yes`	`yes`	`yes`	`yes`	`yes`
`dry run support`	`uk ¹`	`yes`	`yes`	`uk ¹`	`yes`	`yes`
`Profiles`	`yes`	`yes`	`yes`	`yes`	`yes`	`yes`
`file selection`	`pro ⁴`	`cl ³`	`uk ¹`	`uk ¹`	`pro ⁴`	`cl ³`
`file ignore`	`no`	`cl`	`cl ³`	`cl ³`	`pro ⁴`	`cl ³`
`Scripting for RCS`	`yes`	`no`	`yes`	`yes`	`no`	`no`
`MS Windows support`	`yes`	`yes`	`yes`	`yes`	`uk ¹`	`uk ¹`
`availability of CERN distribution`	`yes`	`yes`	`yes`	`no`	`yes`	`yes`

`Option`	`Description`
`-group`	`synchronise owner`
`-owner`	`synchronise group`
`-ui xxx`	`type of user interface ('text' or 'graphic')`
`-killServer`	`kill server when done (even when using sockets)`
`-times`	`synchronise modification times`

`Option`	`Description`
`-v, -verbose`	`increase verbosity`
`-c, -checksum`	`always checksum`
`-r, -recursive`	`recurse into directories`
`-u, -update`	`update only (don't overwrite newer files)`
`-R, -relative`	`use relative path names`
`-l, -links`	`preserve soft links`
`-H, -hard-links`	`preserve hard links`
`-p, -perms`	`preserve permissions`
`-g, -group`	`preserve group`
`-t, -times`	`preserve times`
`-n, -dry-run`	`show what would have been transferred`
`-z, -compress`	`compress file data`
`-progressshow`	`progress during transfer`
`-e, -rsh=COMMAND`	`specify rsh replacement`
`-rsync-path=PATH`	`specify path to rsync on the remote machine`
`-exclude=PATTERN`	`exclude files matching PATTERN`
`-exclude-from=FILE`	`exclude patterns listed in FILE`
`-include=PATTERN`	`don't exclude files matching PATTERN`
`-include-from=FILE`	`don't exclude patterns listed in FILE`