logs snippets

How to parse Apache logs with Ruby

Tagged apache, statistics, logs, ruby, combined, format  Languages ruby

Only supports the combined format at the moment...

class ApacheLog
  FORMATS = {
    :combined => %r{^(\S+) - - \[(\S+ \+\d{4})\] "(\S+ \S+ [^"]+)" (\d{3}) (\d+|-) "(.*?)" "([^"]+)"$}
  }
  
  class << self
    def each_line(log_file, log_format = FORMATS[:combined])

      f = File.open(log_file, "r")

      f.each_line do|line|
        data = line.scan(log_format).flatten

        if data.empty?
          p "Line didn't match pattern: #{line}"

          next
        end

        yield data
      end
    end
  end
end


log_file   = ARGV[0]

ApacheLog.each_line(log_file) do |data|
  host, date, url_with_method, status, size, referrer, agent = data
end

How to read and grep compressed & rotated log files

Tagged zgrep, zless, logs, apache, grep  Languages bash

Your typical server log directory looks something like this:

access.log
access.log.1
access.log.1.gz
access.log.2.gz

Now what if you want to extract data from day x to day y? You could use gzip and grep to uncompress the files, but there's a better way: the z commands.

All you have to do is:

zgrep "2010" /var/log/apache2/access.log*

More info on the subject can be found here: http://www.thegeekstuff.com/2009/05/zcat-zless-zgrep-zdiff-zcmp-zmore-gzip-file-operations-on-the-compressed-files/