When syncing large files over a slow link, it is often useful to use delta compression in order to reduce the bandwidth used. It is also useful to compress files as they take up much less space.
However, many compression algorithms have the unwanted side affect of changing large portions of the compressed output when only a small change has been made in the source.
So, what are some compression algorithms/utilities which create similar compressed blobs from similar source files?
You may want to consider using the --rsyncable argument to gzip. This slightly reduces the compression because it resets the compression algorithm at the beginning of each file in your tarball.
Note, though, that you can also use the -z option with rsync to compress the data transmitted. If you are comfortable with uncompressed files on the source and destination, this may be sufficient for your needs.
We use this to sync up a compressed MySQL database backup that's around 20 GB. Using gzip --rsyncable, we can often transfer only a fraction of the compressed file via rsync if we have the prior day's snapshot already. I have not tried transferring the uncompressed tar file and using rsync's -z option so I cannot comment on its relative efficiency.
--rsyncable appears to be a Debian-only patch.
One solution is to manually compress a file block-by-block. From simple test with the following script files are only slightly (~0.1%) larger with 1MiB blocks. It works for algorithms which support concatenation (gzip and bzip2 to name two). It is also a little slower, but trivial to parallelize.
#!/bin/bash
alg="${2:-gzip}"
size=$(stat -c %s "$1")
cur=0
block=$((1024*1024)) # 1M blocks.
while [ $cur -lt $size ]; do
#echo $alg $size $block $cur >&2
tail -c +$cur "$1" | head -c $block | $alg
cur=$((cur+block))
done
This way changes can only affect the changed block.
You may want to look in to using rdiff. It is similar to rsync however it does not require a two way connection with the server.
The way you would use it is
rdiff signature file.ext file.ext.sig This creates a signature file containing the hashes for the blocks future diffs will use. keep the .sig file around on the source machine.rdiff delta file.ext.sig file.ext file.ext.delta this creates a delta file that only contains the changes where the hashes did not match from the signature file.
rdiff signature file.ext file.ext.sig to update the signature file.rdiff patch file.ext file.ext.delta file.ext this will update the destination copy of the file to match the source copy. Depending on the build of rdiff the 2nd argument and 4th argument may need to be different file names.
-z option to let rsync do the compression transparently.