For When You Can't Have The Real Thing
[ start | index | login ]
start > Monster rsync

Monster rsync

Created by dave. Last edited by dave, 11 years and 78 days ago. Viewed 2,678 times. #3
[diff] [history] [edit] [rdf]
labels
attachments
(8 January 2013)

Working with a vendor, and we have occasion to run an rsync. I'm in a shared desktop with the vendor watching me, and I use my usual rsync -avW invocation. And the vendor offers me this monster command as their recommended command:

# rsync -rvlHtogpc

Let's break that down, just because we can:

  • -r means recursive
  • -v means verbose
  • -l means keep symlinks as symlinks
  • -H means keep hard links as hard links
  • -t means preserve modification times
  • -o means preserve ownership
  • -g means preserve group ownership
  • -p means preserve file permissions
  • -c means run a checksum on the files at both ends and compare them
So really, what's different? -a is a macro that means -rlptgoD. So to review his command:
  • -r implied in -a
  • -v duplicated by me
  • -l implied in -a
  • -H missing
  • -t implied in -a
  • -o implied in -a
  • -g implied in -a
  • -p implied in -a
  • -c missing
  • -D added by implied in -a, means "specials" for device handling
  • -W added by me, means "copy the whole file, don't do deltas"
Oh yes, I forgot to mention I'm doing this on Windows under cygwin. So hard links are not relevant. And there won't be any "specials" requiring device handling. So the only extra value from that mouthful is the checksumming, something the vendor admitted would slow the sync down a fair bit. (Which it does.)

(Note: I didn't write this up to slam the vendor, I just wanted to work out what all the options did and see if it was functionally any different than what I usually use.)

Sidebar: deciding to use -W

Generally I use -W in cases where I'm doing a rsync on a single filesystem -- that is, a sync from one place to another on the same computer. The reasoning is that in order to do the full delta comparison you have to read the whole file on both ends anyways, so once you decide to make the update you might as well just read the source file and write it out on the destination file. It should save you some reads, which in a long exercise should save you some time. When you are running an rsync across a pipe like an ssh, odds are the speed of doing the block deltas by reading both sides will be faster than blindly copying. This assumes that the local disk on both sides are faster than the network in between the two systems. So a dir-to-dir sync would look like this:

# cd /mnt/system/src ; rsync -avW stuff /local/dest
And a rsync between two systems would look like this:
# cd /local/src ; rsync -av -e ssh stuff user@remote:/local/dest
Naturally the nature of your data, the amount of changes that need to be written, and the network in between the two systems will guide your decisions to use -W or not.
no comments | post comment
This is a collection of techical information, much of it learned the hard way. Consider it a lab book or a /info directory. I doubt much of it will be of use to anyone else.

Useful:


snipsnap.org | Copyright 2000-2002 Matthias L. Jugel and Stephan J. Schmidt