Scraping and crawling a website with pjscrape and PhantomJS
1 pjs.addSuite({ 2 url: 'http://google.com', 3 maxDepth: 1, 4 loadScript: ['underscore-min.js'], 5 ignoreDuplicates: true, 6 moreUrls: function() { 7 var urls = _pjs.getAnchorUrls('a', false); 8 var result = _.filter(urls, function (url) { 9 return url.indexOf('javascript') == -1 10 }); 11 console.log("Found " + urls.length + " urls. Using " + result.length) 12 return result 13 }, 14 scraper: function() { 15 var result = [] 16 var links = $('a') 17 links = links.map(function(index, elem) { 18 return $(elem).text() 19 }).toArray() 20 result = result.concat(links) 21 return result 22 } 23 });
How to set up email alerts for ModSecurity
Sending email alerts/notifications from ModSecurity is easy with Lua. Just put this script somewhere:
1 from = "hell@hell.com" 2 to = "heaven@heaven.com" 3 4 -- We use the mail command to send emails. You could also use the lua socket library: 5 -- http://w3.impa.br/~diego/software/luasocket/smtp.html 6 function main() 7 -- create email 8 local subject = m.getvar("TX.email_subject") 9 local body = m.getvar("TX.email_body") 10 11 if subject == nil then 12 return nil 13 end 14 15 if body == nil then 16 body = subject .. ":\n" .. 17 "IP:\t " .. m.getvar("REMOTE_ADDR") .. "\n " .. 18 "HOST:\t " .. m.getvar("REMOTE_HOST") .. "\n " .. 19 "URI:\t " .. m.getvar("REQUEST_URI") 20 end 21 22 m.log(4, "Sending email " .. subject) 23 m.log(5, body) 24 25 -- use the mail command to send emails 26 local cmd = "echo -e \"" .. body .. "\" | mail -s \"" .. subject .. "\" ".. from .." -- -r \"" .. to .. "\"" 27 28 -- remove variables 29 m.setvar('tx.email_subject', nil) 30 m.setvar('tx.email_body', nil) 31 32 -- execute command 33 local f = io.popen(cmd) 34 35 -- read result 36 local l = f:read("*a") 37 m.log(5, "mail output: " .. l) 38 f:close() 39 --print(l) 40 return nil 41 end
Next set up a rule that executes the script after a user has been blocked:
1 SecRule IP:bf_ip_counter "@gt 3" \ 2 "phase:5,pass,t:none, \ 3 setvar:'tx.email_subject=Blocking IP. Too many authentication failures.',exec:/xxx/email.lua, \ 4 ...
Now verify that it works. Note that this script uses the mail command, so make sure sending mails with Postfix and mail works.
Brute-Force Authentication Protection with ModSecurity
IP-Based Blocking
The following ModSecurity script protects from brute-force authentication attacks by blocking IPs. It does this by checking the response code sent by the login page (/sessions). HTTP status 200 means a failed authentication attempt. After 3 attempts the IP is blocked.
1 <LocationMatch /sessions> 2 # Uncomment to troubleshoot 3 #SecDebugLogLevel 9 4 #SecDebugLog /tmp/troubleshooting.log 5 6 # Enforce an existing IP address block 7 SecRule IP:bf_block "@eq 1" \ 8 "phase:2,deny,\ 9 msg:'IP address blocked because of suspected brute-force attack'" 10 11 # Check that this is a POST 12 SecRule REQUEST_METHOD "@streq POST" "phase:5,chain,t:none,nolog,pass" 13 # AND Check for authentication failure and increment counters 14 # NOTE this is for a Rails application, you probably need to customize this 15 SecRule RESPONSE_STATUS "^200" \ 16 "setvar:IP.bf_counter=+1" 17 18 # Check for too many failures from a single IP address. Block for 10 minutes. 19 SecRule IP:bf_counter "@ge 3" \ 20 "phase:5,pass,t:none, \ 21 setvar:IP.bf_block,\ 22 setvar:!IP.bf_counter,\ 23 expirevar:IP.bf_block=600" 24 </LocationMatch>
Username-based Blocking
A serious hacker will have billions of IPs, yes billions in the near future, so it’s better to block by username. To block usernames, use this script:
1 <LocationMatch /sessions> 2 # Retrieve the username 3 SecAction phase:2,nolog,pass,initcol:USER=%{ARGS.username} 4 5 # Enforce an existing username block 6 SecRule USER:bf_block "@eq 1" \ 7 "phase:2,deny,\ 8 msg:'Username \"%{ARGS.username}\" blocked because of suspected brute-force attack'" 9 10 # Check that this is a POST 11 SecRule REQUEST_METHOD "@streq POST" "phase:5,chain,t:none,nolog,pass" 12 # AND Check for authentication failure and increment counters 13 # NOTE this is for a Rails application, you probably need to customize this 14 SecRule RESPONSE_STATUS "^200" \ 15 "setvar:IP.bf_counter=+1" 16 17 # Check for too many failures for a single username 18 SecRule USER:bf_counter "@ge 3" \ 19 "phase:5,t:none,pass,\ 20 setvar:USER.bf_block,\ 21 setvar:!USER.bf_counter,\ 22 expirevar:USER.bf_block=600" 23 </LocationMatch>
Password-based Blocking
Hackers might want to try a reverse brute-force attack on passwords, so you could also block multiple failed login attemps that use the same password. Just modify the script to read the password parameter:
1 # Retrieve the password parameter 2 SecAction phase:2,nolog,pass,initcol:USER=%{ARGS.password}
Note, you might want to use the RESOURCE collection instead of USER, if you’re blocking both usernames and passwords.
On Learning ModSecurity
- Buy the ModSecurity Handbook
- Use Lua when possible. ModSecurity rules are severely limited by the Apache configuration language.
- Use chain for controlling flow. chain = AND operator. OR operator is |.
- There are 5 collections: GLOBAL, IP, RESOURCE, SESSION, USER. Future versions might give you unlimited collections…
- Use LocationMatch and SecDebugLog when troubleshooting:
1 <LocationMatch /sessions> 2 SecDebugLogLevel 9 3 SecDebugLog /tmp/troubleshooting.log
How to send emails from the command line in Linux
Use the mail command…
Specify the sender with the -r switch
1 mail -s "Yer subject" devil@devil.com -- -r "devil@devil.com"
Use echo to send your emails
1 echo -e "TEST\n.\n" | mail -s "subject" devil@devil.com -- -r "devil@devil.com"
Note that we use the -e switch to preserve new lines.
Use a file as the contents of your email
1 mail -s "subject" devil@devil.com -- -r "devil@devil.com" < /tmp/email
How to configure sendmail/postfix to relay or forward emails via an SMTP server
1. Uninstall sendmail
2. Install postfix
1 yum install postfix 2 # or 3 apt-get install postfix
3. Configure postfix
1 sudo vim /etc/postfix/main.cf 2 3 Add: 4 relayhost = smtp.devil.com
