scp snippets

Resume scp download

Tagged rsync, scp, resume download  Languages bash

Unfortunately there is no way for scp to resume a download. Don't despair though if your download was interrupted. Rsync is capable of resuming it. Be aware though, that rsync makes a temporary file that will become as large as the target file is so your disk may become full during the resume process.

rsync --partial --progress --rsh=ssh host:/work/source.tar.bz2 .

Limiting scp bandwidth usage

Tagged scp, limit bandwidth, kaistan rajaaminen, begrã¤nsad bandbredd  Languages bash

Use the -l parameter of scp to limit the speed of a file transfer (both upload or download). An empirically tested, good value for a 54Mbit wireless connection is 18000:

scp -l 18000 remotehost:file.img .

Using expect for automation of bulk scp copying.

Tagged expect, bash, automation, scp, bulk  Languages bash

Expect can come in handy when you can't configure ssh public key authentication on the servers :) (and the system "architect" hasn't yet realized the wonderfulness of a log host).

#!/bin/bash
#
# Usage: script <username> <password> <build>
#
# Example ./copy_logs_from_production.sh marko hubbabubba current
#

username=$1
password=$2
build=$3
mkdir $build

instance_1_server=10.0.0.1
instance_2_server=10.0.0.1
instance_3_server=10.0.0.2
instance_4_server=10.0.0.2
instance_5_server=10.0.0.3
instance_6_server=10.0.0.3
instance_7_server=10.0.0.4
instance_8_server=10.0.0.4
instance_9_server=10.0.0.5

servers=("$instance_1_server" "$instance_2_server" "$instance_3_server" "$instance_4_server" "$instance_5_server" "$instance_6_server" "$instance_7_server" "$instance_8_server" "$instance_9_server" )
i=1
for server in ${servers[@]}; do
  expect -c "
            # exp_internal 1 # uncomment for debugging
            spawn /usr/bin/scp $username@$server:/var/logs/application/$build/server${i}/error.log $build/error-${i}.log
            expect { 
              "*password:*" { send $password\r\n; interact } 
              eof { exit }
            }
            exit
            "
  let "i=i+1"
done