[Linux/SSH] Copying large data from server to server

[chakorules]

I had a situation where I needed to copy large amounts of data from server to server with a big directory tree. (7.5 GBs) Instead of downloading the whole directory local, you can transfer from server to server with the scp command. I optioned to use the scp command for two reasons:

a. It can recursivally copy the whole directory tree and keep the chmod of each file in tack as it copies them to the new server.
b. It's more secure than using FTP and the MGET command.

Let's begin and assume the following information:
Server A is our host where the original files are located.
Server B is where we need to copy files to.

1. Make sure on Server A you have enabled Shell access for the user account you are going to use. Let's say our files are in /home/user1/public_html/myfiles. In this case, make sure you enable user1 shell access.

2. Login SSH on Server B, login as root "su", or as user1 should work too so long as you stick the files in the same place.

3. Type this command:
where xx.xx.xx.xx is the IP address of Server A

scp -rv user1@xx.xx.xx.xx:/home/user1/public_html/myfiles/ /home/user1/public_html/myfiles/

notice there is a *SPACE* after /myfile/ and /home/

You will be prompted for the SSH password of user1 at Server A. Enter this, and the copy process will start.

the -r option copies everything in the directory tree.
/home/user1/public_html/myfiles/dir1/
/home/user1/public_html/myfiles/dir2/
/home/user1/public_html/myfiles/dir3/moredir1/
etc.....

the -v option is verbose, lets you see the transfer in action in your ssh terminal.

The scp command is just like the cp command only it uses SSH intertwined with it all. Very slick if you need to copy whole directory trees from server to server.

If you are transfering at server beach. The transfer rate between SA and VA datacenters are lightning FAST...it didn't take long to move 7.5GB of data....

[dynamix]

thank you, thank you, thank you, - you have saved me sooooo much time and effort :)

[bleachga]

I can copy a file between Linux machines by using scp and typing in my password, but I need to do this from a shell script via cron (daily backup).

I've seen vague hints about using ssh-keygen, but I really need a step-by-step explanation of how to do this . . .

Is there a howto anywhere?

[chakorules]

I did a write up on another forum, here it part of my write up.

I use RSYNC and SSH-Keygen.





Step Three: Creating an SSH Key for each account.
The reason for this is so you can use SSH without a username and password login from box to box.

As I said I wanted each CPanel account to have it’s own backup script per say. So this is how I do it. Make sure that you enable SHELL access for each Cpanel domain user account. In my case User1, User2 and User3 must have SHELL access turned on briefly to create a key. Turn SHELL access off when you are done if you’re paranoid like me. Shell into your VPS account with user1 ID and password.
Create an SSH Key for that User.

Type:
ssh-keygen –t rsa

That command will generate a key file id_rsa and is_rsa.pub I believe. (can’t remember)
Take that file(s) and copy it to your home linux box in the folder you are going to be backing up to. I created a .ssh folder and copied the rsa files I created on the VPS account there. How did I get the key files from VPS to home box? I used FTP. Logout user1, Login user2, type the same command to generate a key for user2. Logout user2, Login user3, type the same command to generate a key for user3.

Now at home you should have something like this:

Files copied from VPS key generation to Home Box User1
/home/mybackupaccountathome/user1backup/.ssh/id_rsa
/home/mybackupaccountathome/user1backup/.ssh/id_rsa.pub

Files copied from VPS key generation to Home Box User2
/home/mybackupaccountathome/user2backup/.ssh/id_rsa
/home/mybackupaccountathome/user2backup/.ssh/id_rsa.pub

Files copied from VPS key generation to Home Box User3
/home/mybackupaccountathome/user3backup/.ssh/id_rsa
/home/mybackupaccountathome/user3backup/.ssh/id_rsa.pub

Once your keys are in place on the home box, go ahead and shell back into the VPS box with user1 again. Now type:

rsync -a -e ssh /home/user1/ mybackupaccountathome@192.168.0.1: /home/mybackupaccountathome/user1backup

mybackupaccountathome better be the linux user account that you create on your home box that you will be using to backup all these accounts.

You should get a prompt to accept keys, and then Rsync will go ahead and run this time. Everything in the /home/user1 folder on the (VPS) is going to get copied to /home/mybackupaccountathome/user1backup

Also the 192.168.0.1 should be changed to your fixed IP of your home linux machine. If you have a router, this makes it nice to route traffic. I setup my router to allow SSH port traffic at 3AM to about 6AM and then redirect and route to the correct IP in my internal network. This keep my home box protected during hours that I am not expecting a SSH connection from the VPS box.

Now logout user1 when rsync is done. The reason I am telling you to manually do this once, is because the first time you use the KEY, you get a prompt that you have to accept. So running it manually this time will force you to accept the prompt, but after that it won’t prompt you again.

Login user2 and repeat process, Logout user2 and Login user3 and repeat process.

Ok now all the SHELL work is done. Disable all shell accounts for each of your Cpanel users.

Step 4: Setup a cron in Cpanel.
Yeah we could have setup a cron in crontab while we were in SHELL already but I like the cron in Cpanel for user account stuff because it sends you a nice e-mail when it ran and if it had any errors.

Login to Cpanel with user1, click on the Cron Icon, click on Standard.
Enter this command:

rsync -a -e ssh /home/user1/ mybackupaccountathome@192.168.0.1: /home/mybackupaccountathome/user1backup

Logout and login each other user in their own accounts too.

Now remember way up at the top when we copied the mySQL backup to /home/user1/mysql? That will automatically get backed up when user1 cron from Cpanel runs.

If you don’t already know the most AWESOMEest reason to use Rsync is because it only transfers files that have changed. So if all the files match at home and on the VPS no bandwidth is wasted from your high speed connection at home.

Hopefully I didn’t forget a step and this process works for you too.

[guillaume]

Another nice thing of using ssh with rsync is that you can use ssh to compress the transmission...
So, not only rsync makes sure that you only transfer the blocks that have changed, but ssh compresses them.

rsync --rsh="ssh -C"

or better, configure your ~/.ssh/config:

Host thehost
Hostname server.example.com
User yourremoteuser
Compression yes
CompressionLevel 9

[crocodile]

....
3. Type this command:
where xx.xx.xx.xx is the IP address of Server A

scp -rv user1@xx.xx.xx.xx:/home/user1/public_html/myfiles/ /home/user1/public_html/myfiles/

notice there is a *SPACE* after /myfile/ and /home/

You will be prompted for the SSH password of user1 at Server A. Enter this, and the copy process will start.

the -r option copies everything in the directory tree.
/home/user1/public_html/myfiles/dir1/
/home/user1/public_html/myfiles/dir2/
/home/user1/public_html/myfiles/dir3/moredir1/
etc.....

the -v option is verbose, lets you see the transfer in action in your ssh terminal.
....
scp with -p options is better
scp -prv user1@xx.xx.xx.xx:/home/user1/public_html/myfiles/ /home/user1/public_html/myfiles/

the -p option Preserves modification times, access times, and modes from the original file.

[beachbum]

For some reason I could not get this to work my password didn't work for my second site.. look like I will be downloading the uploading..:violin: