scp snippets

Resume scp download

Tagged rsync, scp, resume download  Languages bash

Unfortunately there is no way for scp to resume a download. Don't despair though if your download was interrupted. Rsync is capable of resuming it. Be aware though, that rsync makes a temporary file that will become as large as the target file is so your disk may become full during the resume process.

rsync --partial --progress --rsh=ssh host:/work/source.tar.bz2 .

Limiting scp bandwidth usage

Tagged scp, limit bandwidth, kaistan rajaaminen, begrã¤nsad bandbredd  Languages bash

Use the -l parameter of scp to limit the speed of a file transfer (both upload or download). An empirically tested, good value for a 54Mbit wireless connection is 18000:

scp -l 18000 remotehost:file.img .

Using expect for automation of bulk scp copying.

Tagged expect, bash, automation, scp, bulk  Languages bash

Expect can come in handy when you can't configure ssh public key authentication on the servers :) (and the system "architect" hasn't yet realized the wonderfulness of a log host).

#!/bin/bash
#
# Usage: script <username> <password> <build>
#
# Example ./copy_logs_from_production.sh marko hubbabubba current
#

username=$1
password=$2
build=$3
mkdir $build

instance_1_server=10.0.0.1
instance_2_server=10.0.0.1
instance_3_server=10.0.0.2
instance_4_server=10.0.0.2
instance_5_server=10.0.0.3
instance_6_server=10.0.0.3
instance_7_server=10.0.0.4
instance_8_server=10.0.0.4
instance_9_server=10.0.0.5

servers=("$instance_1_server" "$instance_2_server" "$instance_3_server" "$instance_4_server" "$instance_5_server" "$instance_6_server" "$instance_7_server" "$instance_8_server" "$instance_9_server" )
i=1
for server in ${servers[@]}; do
  expect -c "
            # exp_internal 1 # uncomment for debugging
            spawn /usr/bin/scp [email protected]$server:/var/logs/application/$build/server${i}/error.log $build/error-${i}.log
            expect { 
              "*password:*" { send $password\r\n; interact } 
              eof { exit }
            }
            exit
            "
  let "i=i+1"
done

SCP via an SSH jump host

Tagged jump host, rsync, scp, ssh  Languages bash

I prefer using rsync to scp when copying files via jump hosts, for example, to target-host via the intermediate host jump-host:

rsync -v -e 'ssh -A -t [email protected] ssh -A -t target-host' nasty.file :/tmp

rsync syntax is stable across versions and platforms while the syntax of SCP is different across platforms and versions.

man ssh:

     -A      Enables forwarding of the authentication agent connection.  This can also be specified on a per-host basis in a configuration
             file.

             Agent forwarding should be enabled with caution.  Users with the ability to bypass file permissions on the remote host (for the
             agent's UNIX-domain socket) can access the local agent through the forwarded connection.  An attacker cannot obtain key material
             from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into
             the agent.
             
     -t      Force pseudo-terminal allocation.  This can be used to execute arbitrary screen-based programs on a remote machine, which can be
             very useful, e.g. when implementing menu services.  Multiple -t options force tty allocation, even if ssh has no local tty.