Perl script that can be used to calculate min, max, mean, mode, median and standard deviation for a set of log records

2 The best thing about this script is that it's easy to customize, right now it's optimized for comma delimited data. <code> use strict; use warnings; # Import stdev, average, mean and other statistical functions # A copy of http://search.cpan.org/~brianl/Statistics-Lite-3.2/Lite.pm do('stats.pl'); my %page_runtimes; my $delimitor = ';'; my @columns = ("page", "samples", "min", "max", "mean", "mode", "median", "stddev\n"); my $line; my $first_timestamp, my $last_timestamp; # ========================================== # Parse log file # ========================================== # # Don't use foreach as it reads the whole file into memory: foreach $line (<>) { # while ($line=<>) { # remove the newline from $line, otherwise the report will be corrupted. chomp($line); my @columns = split(';', $line); my $timestamp = $columns[0]; my $page_name = $columns[1]; my $page_runtime = $columns[2]; if(!defined($first_timestamp)) { $first_timestamp = $timestamp; } # print what we find if(!defined(@{$page_runtimes{$page_name}})) { print "Found page '$page_name'\n"; } # add page runtimes to one hash push(@{$page_runtimes{$page_name}}, $page_runtime); $last_timestamp = $timestamp; } # ========================================== # Calculate and print page statistics # ========================================== open(PAGE_REPORT, ">report.csv") or die("Could not open report.csv."); print PAGE_REPORT "First sample\n".$first_timestamp."\nLast sample\n".$last_timestamp."\n\n"; print PAGE_REPORT join($delimitor, @columns); for my $page_name (keys %page_runtimes ) { my @runtimes = @{$page_runtimes{$page_name}}; my $samples = @runtimes; my $min = min(@runtimes); my $max = max(@runtimes); my $mean = mean(@runtimes); my $mode = mode(@runtimes); my $median = median(@runtimes); my $stddev = stddev(@runtimes); my @data = ($page_name, $samples, $min, $max, $mean, $mode, $median, $stddev); my $line = join($delimitor, @data); # Use comma instead of decimal $line =~ s/\./\,/g; print PAGE_REPORT "$line\n"; } close(PAGE_REPORT); </code> To use it simply pipe some data into it like this: <code> grep "2008-31-12" silly-data.log | perl analyze.pl </code> 0

2008-09-02T16:52:09+03:00

239

101

The best thing about this script is that it’s easy to customize, right now it’s optimized for comma delimited data. <pre class="active4d"> 1 use strict; 2 use warnings; 3 4 # Import stdev, average, mean and other statistical functions 5 # A copy of http://search.cpan.org/~brianl/Statistics-Lite-3.2/Lite.pm 6 do('stats.pl'); 7 8 my %page_runtimes; 9 my $delimitor = ';'; 10 my @columns = ("page", "samples", "min", "max", "mean", "mode", "median", "stddev\n"); 11 my $line; 12 my $first_timestamp, my $last_timestamp; 13 14 # ========================================== 15 # Parse log file 16 # ========================================== 17 18 # 19 # Don't use foreach as it reads the whole file into memory: foreach $line (<>) { 20 # 21 while ($line=<>) { 22 # remove the newline from $line, otherwise the report will be corrupted. 23 chomp($line); 24 25 my @columns = split(';', $line); 26 my $timestamp = $columns[0]; 27 my $page_name = $columns[1]; 28 my $page_runtime = $columns[2]; 29 30 if(!defined($first_timestamp)) 31 { 32 $first_timestamp = $timestamp; 33 } 34 35 # print what we find 36 if(!defined(@{$page_runtimes{$page_name}})) 37 { 38 print "Found page '$page_name'\n"; 39 } 40 41 # add page runtimes to one hash 42 push(@{$page_runtimes{$page_name}}, $page_runtime); 43 44 $last_timestamp = $timestamp; 45 } 46 47 # ========================================== 48 # Calculate and print page statistics 49 # ========================================== 50 open(PAGE_REPORT, ">report.csv") or die("Could not open report.csv."); 51 52 print PAGE_REPORT "First sample\n".$first_timestamp."\nLast sample\n".$last_timestamp."\n\n"; 53 print PAGE_REPORT join($delimitor, @columns); 54 55 for my $page_name (keys %page_runtimes ) 56 { 57 my @runtimes = @{$page_runtimes{$page_name}}; 58 59 my $samples = @runtimes; 60 my $min = min(@runtimes); 61 my $max = max(@runtimes); 62 my $mean = mean(@runtimes); 63 my $mode = mode(@runtimes); 64 my $median = median(@runtimes); 65 my $stddev = stddev(@runtimes); 66 67 my @data = ($page_name, $samples, $min, $max, $mean, $mode, $median, $stddev); 68 69 my $line = join($delimitor, @data); 70 71 # Use comma instead of decimal 72 $line =~ s/\./\,/g; 73 74 print PAGE_REPORT "$line\n"; 75 } 76 close(PAGE_REPORT); </pre> To use it simply pipe some data into it like this: <pre class="active4d"> 1 grep "2008-31-12" silly-data.log | perl analyze.pl </pre>

Perl script that can be used to calculate min, max, mean, mode, median and standard deviation for a set of log records

2009-04-22T10:31:41+03:00