Expect can come in handy when you can’t configure ssh public key authentication on the servers :) (and the system “architect” hasn’t yet realized the wonderfulness of a log host).

   1  #!/bin/bash
   2  #
   3  # Usage: script <username> <password> <build>
   4  #
   5  # Example ./copy_logs_from_production.sh marko hubbabubba current
   6  #
   7  
   8  username=$1
   9  password=$2
  10  build=$3
  11  mkdir $build
  12  
  13  instance_1_server=10.0.0.1
  14  instance_2_server=10.0.0.1
  15  instance_3_server=10.0.0.2
  16  instance_4_server=10.0.0.2
  17  instance_5_server=10.0.0.3
  18  instance_6_server=10.0.0.3
  19  instance_7_server=10.0.0.4
  20  instance_8_server=10.0.0.4
  21  instance_9_server=10.0.0.5
  22  
  23  servers=("$instance_1_server" "$instance_2_server" "$instance_3_server" "$instance_4_server" "$instance_5_server" "$instance_6_server" "$instance_7_server" "$instance_8_server" "$instance_9_server" )
  24  i=1
  25  for server in ${servers[@]}; do
  26    expect -c "
  27              # exp_internal 1 # uncomment for debugging
  28              spawn /usr/bin/scp $username@$server:/var/logs/application/$build/server${i}/error.log $build/error-${i}.log
  29              expect { 
  30                "*password:*" { send $password\r\n; interact } 
  31                eof { exit }
  32              }
  33              exit
  34              "
  35    let "i=i+1"
  36  done
  37  

Achieve major speed up of e.g scp tab completions or other consequent connections to the same server by using ControlMaster. Append these rows into ~/.ssh/config

   1  Host *
   2  ControlMaster auto
   3  ControlPath ~/.ssh/.sock_%r@%h:%p 

Use the -l parameter of scp to limit the speed of a file transfer (both upload or download). An empirically tested, good value for a 54Mbit wireless connection is 18000:

   1  scp -l 18000 remotehost:file.img .

Unfortunately there is no way for scp to resume a download. Don’t despair though if your download was interrupted. Rsync is capable of resuming it. Be aware though, that rsync makes a temporary file that will become as large as the target file is so your disk may become full during the resume process.

   1  rsync --partial --progress --rsh=ssh host:/work/source.tar.bz2 .