5  Data Transfert

5.1 Introduction

Transferring data efficiently between local and remote systems is crucial in a networked environment. This section covers foundational tools and commands for secure and reliable data transfer, including downloading files, copying between systems, and synchronizing directories.

Overview:

  • wget: For direct file downloads.
  • scp: Securely copies files between local and remote systems.
  • rsync: Synchronizes files and directories efficiently, with options for selective copying and deletion.


5.2 Downloading Files with wget

To download files, use:

wget [URL]

Cancel with Ctrl + C, and resume with wget --background -c [URL].

5.3 File Transfer with scp

5.3.1 Local to Remote

scp [file] [user]@[host]:[path]

5.3.2 Remote to Local

scp [user]@[host]:[path/to/file] [dest]

5.4 Directory Synchronization with rsync

Synchronize directories while preserving file attributes:

rsync -arv [source] [destination]

Options: - -a: Archive mode. Allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps - -r: Recursive - -v: Verbose

5.4.1 Local Synchronization

rsync -arv --delete --progress [source]/ [destination]/

5.4.2 Remote Synchronization

  • Local to Remote:

    rsync -arv --delete --progress [source]/ [user]@[ip]:[destination]/

  • Remote to Local:

    rsync -arv --delete --progress [user]@[ip]:[source]/ [destination]/

  • Secure with SSH:

    rsync -arv --delete --progress -e ssh [user]@[ip]:[source]/ [destination]/

5.4.3 Custom Inclusions and Exclusions

Tailor synchronization with --include and --exclude flags:

rsync -arv --delete --progress --include 'R*' --exclude [Folder1] --exclude [folder2/subfolder1] [source]/ [user]@[ip]:[destination]/

5.4.4 Cleaning Up Source After Transfer

Remove source files post-transfer with --remove-source-files:

rsync -arv --remove-source-files [source]/ [destination]/

5.5 Conclusion

These tools and commands offer a robust framework for managing data transfers, enhancing efficiency and reliability in handling files and directories across diverse environments.