Scraping and crawling a website with pjscrape and PhantomJS

JavaScript posted about 8 hours ago by christian

   1  pjs.addSuite({
   2    url: 'http://google.com',
   3    maxDepth: 1,
   4    loadScript: ['underscore-min.js'],
   5    ignoreDuplicates: true,
   6    moreUrls: function() {
   7      var urls = _pjs.getAnchorUrls('a', false);
   8      var result = _.filter(urls, function (url) {
   9        return url.indexOf('javascript') == -1
  10      });
  11      console.log("Found " + urls.length + " urls. Using " + result.length)
  12      return result
  13    },
  14    scraper: function() {
  15      var result = []
  16      var links = $('a')
  17      links = links.map(function(index, elem) { 
  18        return $(elem).text()
  19      }).toArray()
  20      result = result.concat(links)
  21      return result
  22    }
  23  });

Tagged phantomjs, pjscrape

How to set up email alerts for ModSecurity

Lua posted 5 days ago by christian

Sending email alerts/notifications from ModSecurity is easy with Lua. Just put this script somewhere:

   1  from = "hell@hell.com"
   2  to = "heaven@heaven.com"
   3  
   4  -- We use the mail command to send emails. You could also use the lua socket library:
   5  -- http://w3.impa.br/~diego/software/luasocket/smtp.html
   6  function main()
   7    -- create email
   8    local subject = m.getvar("TX.email_subject")
   9    local body = m.getvar("TX.email_body")
  10  
  11    if subject == nil then
  12      return nil
  13    end
  14  
  15    if body == nil then
  16      body = subject .. ":\n" ..
  17       "IP:\t " .. m.getvar("REMOTE_ADDR") .. "\n " ..
  18       "HOST:\t " .. m.getvar("REMOTE_HOST") .. "\n " ..
  19       "URI:\t " .. m.getvar("REQUEST_URI")
  20    end
  21  
  22    m.log(4, "Sending email " .. subject)
  23    m.log(5, body)
  24  
  25    -- use the mail command to send emails
  26    local cmd = "echo -e \"" .. body .. "\" | mail -s \"" .. subject .. "\" ".. from .." -- -r \"" .. to .. "\""
  27  
  28    -- remove variables
  29    m.setvar('tx.email_subject', nil)
  30    m.setvar('tx.email_body', nil)
  31  
  32    -- execute command
  33    local f = io.popen(cmd)
  34  
  35    -- read result
  36    local l = f:read("*a")
  37    m.log(5, "mail output: " .. l)
  38    f:close()
  39    --print(l)
  40    return nil
  41  end

Next set up a rule that executes the script after a user has been blocked:

   1  SecRule IP:bf_ip_counter "@gt 3" \
   2                  "phase:5,pass,t:none, \
   3                  setvar:'tx.email_subject=Blocking IP. Too many authentication failures.',exec:/xxx/email.lua, \
   4          ...

Now verify that it works. Note that this script uses the mail command, so make sure sending mails with Postfix and mail works.

Tagged modsecurity, email, alerts

Brute-Force Authentication Protection with ModSecurity

Apache posted 5 days ago by christian

IP-Based Blocking

The following ModSecurity script protects from brute-force authentication attacks by blocking IPs. It does this by checking the response code sent by the login page (/sessions). HTTP status 200 means a failed authentication attempt. After 3 attempts the IP is blocked.

   1  <LocationMatch /sessions>
   2           # Uncomment to troubleshoot
   3          #SecDebugLogLevel 9
   4          #SecDebugLog /tmp/troubleshooting.log
   5  
   6          # Enforce an existing IP address block
   7          SecRule IP:bf_block "@eq 1" \
   8                  "phase:2,deny,\
   9                  msg:'IP address blocked because of suspected brute-force attack'"
  10  
  11          # Check that this is a POST
  12          SecRule REQUEST_METHOD "@streq POST" "phase:5,chain,t:none,nolog,pass"
  13                  # AND Check for authentication failure and increment counters
  14                  # NOTE this is for a Rails application, you probably need to customize this
  15                  SecRule RESPONSE_STATUS "^200" \
  16                          "setvar:IP.bf_counter=+1"
  17  
  18          # Check for too many failures from a single IP address. Block for 10 minutes.
  19          SecRule IP:bf_counter "@ge 3" \
  20                  "phase:5,pass,t:none, \
  21                  setvar:IP.bf_block,\
  22                  setvar:!IP.bf_counter,\
  23                  expirevar:IP.bf_block=600"
  24  </LocationMatch>

Username-based Blocking

A serious hacker will have billions of IPs, yes billions in the near future, so it’s better to block by username. To block usernames, use this script:

   1  <LocationMatch /sessions>
   2          # Retrieve the username
   3          SecAction phase:2,nolog,pass,initcol:USER=%{ARGS.username}
   4  
   5          # Enforce an existing username block
   6          SecRule USER:bf_block "@eq 1" \
   7                  "phase:2,deny,\
   8                  msg:'Username \"%{ARGS.username}\" blocked because of suspected brute-force attack'"
   9  
  10          # Check that this is a POST
  11          SecRule REQUEST_METHOD "@streq POST" "phase:5,chain,t:none,nolog,pass"
  12                  # AND Check for authentication failure and increment counters
  13                  # NOTE this is for a Rails application, you probably need to customize this
  14                  SecRule RESPONSE_STATUS "^200" \
  15                          "setvar:IP.bf_counter=+1"
  16  
  17          # Check for too many failures for a single username
  18          SecRule USER:bf_counter "@ge 3" \
  19                  "phase:5,t:none,pass,\
  20                  setvar:USER.bf_block,\
  21                  setvar:!USER.bf_counter,\
  22                  expirevar:USER.bf_block=600"
  23  </LocationMatch>

Password-based Blocking

Hackers might want to try a reverse brute-force attack on passwords, so you could also block multiple failed login attemps that use the same password. Just modify the script to read the password parameter:

   1  # Retrieve the password parameter
   2          SecAction phase:2,nolog,pass,initcol:USER=%{ARGS.password}

Note, you might want to use the RESOURCE collection instead of USER, if you’re blocking both usernames and passwords.

On Learning ModSecurity

  • Buy the ModSecurity Handbook
  • Use Lua when possible. ModSecurity rules are severely limited by the Apache configuration language.
  • Use chain for controlling flow. chain = AND operator. OR operator is |.
  • There are 5 collections: GLOBAL, IP, RESOURCE, SESSION, USER. Future versions might give you unlimited collections…
  • Use LocationMatch and SecDebugLog when troubleshooting:

   1  <LocationMatch /sessions>
   2          SecDebugLogLevel 9
   3          SecDebugLog /tmp/troubleshooting.log  

Tagged modsecurity, brute-force, authentication

How to send emails from the command line in Linux

Plain Text posted 5 days ago by christian

Use the mail command…

Specify the sender with the -r switch

   1  mail -s "Yer subject" devil@devil.com -- -r "devil@devil.com"

Use echo to send your emails

   1  echo -e "TEST\n.\n" | mail -s "subject" devil@devil.com -- -r "devil@devil.com"

Note that we use the -e switch to preserve new lines.

Use a file as the contents of your email

   1  mail -s "subject" devil@devil.com -- -r "devil@devil.com" < /tmp/email

Tagged mail, postfix, smtp

How to configure sendmail/postfix to relay or forward emails via an SMTP server

Plain Text posted 5 days ago by christian

1. Uninstall sendmail
2. Install postfix

   1  yum install postfix
   2  # or
   3  apt-get install postfix

3. Configure postfix

   1  sudo vim /etc/postfix/main.cf
   2  
   3  Add:
   4  relayhost = smtp.devil.com

Tagged sendmail, postfix, stmp