shell snippets

Log file analysis with AWK - Calculating the sum and average

Tagged awk, shell, performance, average, sum  Languages bash

This AWK script is useful when you want to calculate the average and sum for a set of values found in a log file.

awk '{ s += $1 } END { print "sum: ", s, " average: ", s/NR, " samples: ", NR }' rails_production_log_or_whatever.log

Note that $1 means that column one contains the values you want to use. NR is the total number of rows in the file. As an example, let's say you have this log file:

1
2
3
4
5

The output would then be:

sum:  15  average:  3  samples:  5

Combine it with grep or sed to do more advanced log file analysis -- you can for example calculate the average time it took to render action xyz in Rails on the 21th of July at 21:00 PM.

Note that by default the column values should be space separated--use the following switch to parse CSV (comma separated) files: -F,

How to start a bash shell in Docker

Tagged bash, docker, shell  Languages bash

To start a bash shell in an already running container:

docker exec -it <container id/name> bash

To start a bash shell using an image of a container that is not running:

docker run -i -t --entrypoint /bin/bash <image id>

How to clean ENV variables when executing external commands with Open3

Tagged bundler, command, env, open3, popen3, security, shell  Languages ruby

Running commands with Open3#popen3 from, for example, a Rails application will give the command access to all ENV variables, including potential secrets.

Before running commands in you need to clean the ENV variables:

# Clean the ENV
def with_env(env)
  backup = ENV.to_hash
  ENV.replace(env)
  yield
ensure
  ENV.replace(backup)
end
# Set PWD, ENV, and options
pwd = '/Alset/YledoM/'
env = {
  'oLLEH' => 'DLROw',
  'RACK_ENV' => 'test',
  'PATH' => ENV['PATH']
}
options = {
  chdir: pwd
}
# The command to run
cmd = "bundle exec ruby script.rb"
# Prints a 'dirty' env
ENV.sort.to_h.each do |key, val|
  puts "#{key} => #{val}"
end
# Run the command with the given ENV
with_env(env) do
  Open3.popen3(env, cmd, options) do |stdin, stdout, stderr, wait_thr|
    # YXES
  end
  # Prints a clean env
  ENV.sort.to_h.each do |key, val|
    puts "#{key} => #{val}"
  end
end