Synchronising data between computers/system using Unison for Linux/OSX
Introduction: What is Unison and why would I use it
Imagine you are working on a project. All files and directories related to this are in a directory in one of the departmental data spaces. You are going away and will be working on your project while on the beach without internet access. You copy your project directory onto your laptop.
While on holiday you do some project work as planned When you return to your office in DAMTP you continue with your project, but with the files you have in the data space. Then one day you remember you edited some files while you were away but you now cannot remember which files you edited on holiday on your laptop and which since returning on the departmental system. You will need to go through both systems picking out the newest copy of each file and watching out for any files you changed in both locations. Painful.
If you had Used unison to do the copy work you wouldn't have this issue. Unison would sync both the laptop directory and the departmental one so that both contained the most up to date copy of all the files. If you had edited a file in both locations, Unison would notice this and flag this for your attention, so that you can decide what to do.
Unison can be very useful if you work on different computers on a regular basis; maybe work and college or work and home.
This document describes how to use Unison to sync files between your department home directory and your Laptop for Linux and MacOSX. While people running windows on their laptops are welcome to use unison, we recommend that they use WinSCP instead.
Install Unison
This is a two step process. The same version of Unison needs to be installed both on your laptop and in your DAMTP home directory.
To install Unison on your laptop check to see if your operating system already has a Unison package. Systems like Ubuntu will, or you can download a binary version or build your own version from source. The examples on this page use the graphical versions.
-
We have pre-compiled versions of Unison for copying to your DAMTP home directory. Select a version to match the one you put on your laptop, download it and place it in your DAMTP home directory's bin directory calling it unison (not unison-number) If you cannot find a match contact help@damtp letting them know which version you need.
Profiles
Start the GUI interface to Unison on your laptop.
On some systems you will have a Unison Icon, on some linux
systems you may need to type unison-gtk or similiar. Your system should have its
own documentation to guide you if this step isn't obvious.
Step one: Create your profile. Some systems ask you to enter all the information on one screen, others over two or more screens. You will be asked for the following information.
- Profile name: use a meaningful name
- Full path of DAMTP Directory you are using
- Full path of your laptop directory
- Host name of DAMTP computer
- Username for DAMTP computer
- Depending on which Unison package you are using, you may be asked to select the connection method. SSH is the answer
At the time of writing, please do not use the recommended ssh.damtp.cam.ac.uk. While ssh.damtp is a round robin system; whenever it connects to a different host Unison throws up an error. For the moment it is easier to avoid this and connect to one of the DAMTP public machines instead or your desktop computer. Please make sure both directories exist.
Next open your profile, your system may do this part automatically for you, and enter your DAMTP password when asked. You must use your DAMTP unix password here because your DAMTP home directory and the DAMTP data spaces both reside on unix computers. If you only ever use Windows machines you may need to find your "New Account" letter as password changes on windows machines do not change your unix password.
You will now find yourself at a general transfer screen where Unison shows you which way it is transferring any files. At the top of that screen you will see various actions you can take if you do not want to take Unison's default action for any file, for example you may prefer to copy an older copy of a file back over a newer version. The picture under the introduction shows a typical screen.
Finish transferring the files, work on them and next time you want to sync, Unison will show you a list of your profiles, select the profile you want and transfer your files.
Technical part: In your DAMTP home directory you may notice a new directory called .unison. Don't delete this. This is where your Unison profiles are kept; these can be customised.
Linux laptop to departmental directory
This is the easist case, you are transferring from a linux computer to a linux computer.
The above instructions should work fine. The images may look slightly different but are in
essance the same. Ubuntu has a unison 2.27.72 package. Make sure you install both
unison and unison-gtk packages.
Open Unison on your laptop and set up a profile as described above.
Then let Unison transfer your files.
Mixed Case File names - WATCH OUT
Some systems are case insenstive, OSX, Windows and even your USB stick may be. This means you cannot create one file called "eggs.doc" and another called "Eggs.doc", as to these systems they are the same file. However the departmental system is case senstive and allows you to do this. This causes problems when it comes to using Unison.
The best way to avoid problems with Unison and Mixed Case Filenames is not to have any files with the same name but different case (no mouse.c and Mouse.c for example) Run the command below to catch all clashes and rename them before starting Unison e.g., to filename.old or filename.dup.
find . -print | tr 'A-Z' 'a-z' | sort | uniq -cd
Also make sure you don't have file case name clashes between the files in your remote and local directories you are syncing, as otherwise bad things may happen to your files. The OSX notes below demonstrate what happens if there is a clash and how to get Unison out of its confused state.
Case insenstive example (uses OSX for examples)
As described above OSX is a case insentive system, along with Windows and many formatted USB sticks.
In this image I have two files, one called "Pig" and one called "pig" in my DAMTP directory. Unison notices the conflict for the two files and reports it with a big red X and a message at the bottom of its screen.
I notice the problem, I quit from unison and I rename "pig" to "bacon". I keep the file "Pig". I run Unison again. But Unison is now confused and it cannot see the "Pig" file.
The best thing to do is avoid the conflict in the first place. By following the command given above.
If Unison does get into the confused state above you can get out of it by renaming the departmental directory you are transferring and creating a new unison profile. There is no need to rename the directory on your laptop. Creating a new unison profile on its own is not enough as unison leaves a state file behind when it runs and the new profile also finds this state.
Just renaming files when in this situation does not work and can lead to lost files as shown above. You must rename the target directory and create the new user profile.
Moral of story: Don't have names that clash for example "Pig" and "pig" when running Unison; rename them before using Unison and you'll save yourself a lot of time in the long run.
Note:
You may also with to remove old archive files. Only do this if you don't use Unison for any other purpose as otherwise you'll remove your working archive files as well.
The OSX the unison archive files are in
~/Library/Application\ Support/Unision/ar*
Linux, MacOSX and Profiles (and problems with the hash sign)
For each pair of directories being kept in sync, unison creates a profile from the choices you make when syncing files. You can also edit your Unison profile files by hand.
A typical profile may look like
root = /Volumes/DAMTP_home/abc123 root = ssh://abc123@hostname.damtp.cam.ac.uk/ ignore = Path mail ignore = Path .mozilla ignore = Path .openoffice.org ignore = Path .thunderbird ignore = Path .cxoffice ignore = Path .java ignore = Path \#
Here we are keeping two homedirectories in sync. We don't want to copy over changes to mail and web related folders. We are also asking unison to ignore all files with a hash (#) sign in their name as unison is known to crash when it encounters this character.
Where to find your Unison's profiles.
Linux: in your homedirectory's .unison directory
OSX: in your home directory
Library/Application Support/Unison/
If you are running a version of Unison greater than 2.9.20 on OSX then Unison will use a .unison folder in your homedirectory for your profiles if one exists, otherwise it will use your Library folder.
Conclusion
I have only described the basics of Unison here, with an emphasis on how it interacts with the deparmental systems. For further guidance on using Unison itself check the well documented official documenation. Some of the options differ on different versions.