proxy snippets

Example of how to use Ruby's NET::HTTP

Tagged debug, timeout, proxy, example, https, http, net::http, ruby  Languages ruby

I always forget how to use NET:HTTP and examples online are of little help, so I use this code as a starting point when I work with NET:HTTP.

The code supports: * HTTP redirects * POST/GET * Headers * SSL/HTTPs * Proxies * Debugging HTTP traffic * HTTP connection and read timeout

require 'net/http'
require 'net/https'
require 'uri'

class HTTP
  class << self
    def get(url, options = {})
      execute(url, options)
    end

    def post(url, options = { :method => :post })
      execute(url, options)
    end

    protected
      def proxy
        http_proxy = ENV["http_proxy"]
        URI.parse(http_proxy) rescue nil
      end

      def to_uri(url)
        begin
          if !url.kind_of?(URI) 

            url = URI.parse(url)
          end
        rescue
          raise URI::InvalidURIError, "Invalid url '#{url}'"
        end

        if (url.class != URI::HTTP && url.class != URI::HTTPS)
          raise URI::InvalidURIError, "Invalid url '#{url}'"
        end

        url
      end

      def execute(url, options = {})
        options = { :parameters => {}, :debug => false, 
                    :http_timeout => 60, :method => :get, 
                    :headers => {}, :redirect_count => 0, 
                    :max_redirects => 10 }.merge(options)

        url = to_uri(url)
        
        if proxy
          http = Net::HTTP::Proxy(proxy.host, proxy.port).new(url.host, url.port)
        else
          http = Net::HTTP.new(url.host, url.port)
        end
        
        if url.scheme == 'https'
          http.use_ssl = true
          http.verify_mode = OpenSSL::SSL::VERIFY_NONE
        end
        
        http.open_timeout = http.read_timeout = options[:http_timeout]
        
        http.set_debug_output $stderr if options[:debug]
        
        request = case options[:method]
          when :post
            request = Net::HTTP::Post.new(url.request_uri)
            request.set_form_data(options[:parameters])
            request
          else
            Net::HTTP::Get.new(url.request_uri)
        end

        options[:headers].each { |key, value| request[key] = value }
        response = http.request(request)

        if response.kind_of?(Net::HTTPRedirection)      
          options[:redirect_count] += 1

          if options[:redirect_count] > options[:max_redirects]
            raise "Too many redirects (#{options[:redirect_count]}): #{url}" 
          end

          redirect_url = redirect_url(response)

          if redirect_url.start_with?('/')
            url = to_uri("#{url.scheme}://#{url.host}#{redirect_url}")
          end

          response = execute(url, options)
        end

        response
      end

      # From http://railstips.org/blog/archives/2009/03/04/following-redirects-with-nethttp/
      def redirect_url(response)
        if response['location'].nil?
          response.body.match(/<a href=\"([^>]+)\">/i)[1]
        else
          response['location']
        end
      end
  end
end

You can also find my Net:HTTP example on Github

How to use file_get_contents with a proxy

Tagged file_get_contents, proxy  Languages php
$url = 'http://www';

$proxy = 'tcp://xxx:8080';

$context = array(
    'http' => array(
        'proxy' => $proxy,
        'request_fulluri' => True,
        ),
    );
    
$context = stream_context_create($context);

$body = file_get_contents($url, False, $context);

The code was found here. Note that it doesn't seem to work with HTTPS.

Simple Webrick proxy.

Tagged webrick, ruby, proxy  Languages ruby

A handy oneliner which I found here.

ruby -r webrick/httpproxy -e 's = WEBrick::HTTPProxyServer.new(:Port => 9999, :RequestCallback => Proc.new{|req,res| puts req.request_line, req.raw_header}); trap("INT"){s.shutdown}; s.start'

Configuring Apache to be a forward proxy

Tagged proxy, rewrite, forward proxy, apache, forward  Languages apacheconf

First enable the proxy and proxy_http modules:

sudo a2enmod proxy
sudo a2enmod proxy_http

Then use this configuration to make Apache act as an HTTP proxy:

<VirtualHost *:8080>
# Enable forward proxy
ProxyRequests On
# Add "Via" header
ProxyVia On
#ProxyRemote * http://...:8080 Uncomment to route requests through another proxy

<Proxy *>
    Order deny,allow
      Deny from all
  # Allow access only from local network
  Allow from 192.168.1
</Proxy> 
    
# Enable caching proxy
CacheRoot "/tmp"
CacheMaxExpire 24
CacheLastModifiedFactor 0.1
CacheDefaultExpire 1

ServerName my-proxy

ErrorLog "/var/log/apache2/proxy-error.log"
CustomLog "/var/log/apache2/proxy-access.log" common
</VirtualHost>

Also read this.

Tips

You can use mod_rewrite to rewrite requests. To rewrite root (/) to /temporary_outage you could use the following rewrite:

RewriteCond %{HTTP_HOST} ^(www\.)?xxx\.com 
RewriteRule /$ http://%{HTTP_HOST}/temporary_outage/ [P,L]