I have a very old DS5K series storage. One day DS5K felt lonely, wanted to draw attention (a special kind of love!) and two disks failed at the same time.
Sadly, some iso files in my NFS share became corrupt. File sizes are similar to original ones, but checksum differs.
Uploading them from home to LAB would have taken aprox. 32 days:)
Again, main problem with big file transfers is the possibility of file corruption. Tools like wget,ftp,sftp will resume but not fix corrupted data.
rsync is the ultimate tool for such occasions, it is available for Linux, AIX, Solaris,MacOS.
Lets fix my corrupted iso's. 4 iso files consists around 16 GB data.
Rsync recipe are below;
–checksum (skip based on checksum, not mod-time & size),
–in-place (to transfer the file in place as rsync normally writes a temporary file, then moves)
–no-whole-file (to not copy the whole file, but use deltas instead).
[root@p8kvm images]# rsync -Pa --checksum --inplace --no-whole-file RHEL-* -e ssh root@xxxxx://mnt/stg-files/Linux/rhel
The authenticity of host 'xxxxx xxxx)' can't be established.
RSA key fingerprint is f7:db:fa:xx:xx:2d:61:75:7d:c1:e6:75:0d:dc:07:58.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'xxxxxxxx' (RSA) to the list of known hosts.
sending incremental file list
3396677632 100% 37.73MB/s 0:01:25 (xfer#1, to-check=3/4)
3267020800 100% 59.13MB/s 0:00:52 (xfer#2, to-check=2/4)
3395985408 100% 65.84MB/s 0:00:49 (xfer#3, to-check=1/4)
3368349696 100% 41.20MB/s 0:01:17 (xfer#4, to-check=0/4)
Four iso files, finished around 10 minutes:)
I love rsync.