debian snippets

nginx startup script for Debian

Tagged nginx, start, debian, linux, script  Languages bash
sudo vim /etc/init.d/nginx

Paste in the following (remember to run 'set :paste' in VIM when pasting):

#! /bin/sh
##
# nginx start script
##

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/sbin/nginx
NAME=nginx
DESC=nginx

if [ ! -x $DAEMON ]
then
   echo "Couldn't find $DAEMON. Please set path to DAEMON."
   exit 0
fi


# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
    . /etc/default/nginx
fi

set -e

case "$1" in
  start)
    echo -n "Starting $DESC: "
    start-stop-daemon --start --pidfile /var/run/$NAME.pid \
        --exec $DAEMON -- $DAEMON_OPTS
    echo "$NAME."
    ;;
  stop)
    echo -n "Stopping $DESC: "
    start-stop-daemon --stop --pidfile /var/run/$NAME.pid \
        --exec $DAEMON
    echo "$NAME."
    ;;
  restart|force-reload)
    echo -n "Restarting $DESC: "
    start-stop-daemon --stop --pidfile \
        /var/run/$NAME.pid --exec $DAEMON
    sleep 1
    start-stop-daemon --start --pidfile \
        /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS
    echo "$NAME."
    ;;
  reload)
      echo -n "Reloading $DESC configuration: "
      start-stop-daemon --stop --signal HUP --pidfile /var/run/$NAME.pid \
          --exec $DAEMON
      echo "$NAME."
      ;;
  *)
    N=/etc/init.d/$NAME
    echo "Usage: $N {start|stop|restart|force-reload}" >&2
    exit 1
    ;;
esac

exit 0

Now make the script executable with this command:

sudo chmod 755 /etc/init.d/nginx

Lastly, run this command to make the script run when the server starts and stops:

sudo /usr/sbin/update-rc.d -f nginx defaults

How to install and use the Sphinx search engine and acts_as_sphinx plugin on Debian Etch

Tagged sphinx, search, acts_as_sphinx, debian, etch, rails, install, libstemmer  Languages bash

Inspiration for this snippet was taken from this post on the Sphinx forum, plus this blog post.

Compiling Sphinx

First install the prerequisites:

sudo aptitude install libmysql++-dev libmysqlclient15-dev checkinstall

Next download sphinx, libstemmer and install everything and the fish:

cd /usr/local/src

wget http://sphinxsearch.com/downloads/sphinx-0.9.9.tar.gz
tar zxvf sphinx-0.9.9.tar.gz 

cd sphinx-0.9.9/

# Add stemming support for Swedish, Finnish and other fun languages.
wget http://snowball.tartarus.org/dist/libstemmer_c.tgz
tar zxvf libstemmer_c.tgz

./configure --with-libstemmer
make

make install

Configure Sphinx

Create a sphinx.conf file in your Rails config directory, as described here, or use this template.

Install acts_as_sphinx plugin

./script/plugin install http://svn.datanoise.com/acts_as_sphinx

Add acts_as_sphinx to your model:

class Documents
   acts_as_sphinx
end

Indexing content

rake sphinx:index

(in /var/www/xxx.com/releases/20080429144230)
Sphinx 0.9.8-rc2 (r1234)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file './sphinx.conf'...
indexing index 'xxx.com'...
collected 5077 docs, 0.6 MB
sorted 0.1 Mhits, 100.0% done
total 5077 docs, 632096 bytes
total 0.160 sec, 3950427.25 bytes/sec, 31729.86 docs/sec

Reindexing content

sphinx:index shouldn't be run while the searchd process is running, so use rake sphinx:rotate instead, which restarts the searchd process after indexing.

Starting the daemon

mkdir -m 664 /var/log/sphinx
rake sphinx:start

(in /var/www/xxx.com/releases/20080429144230)
Sphinx 0.9.8-rc2 (r1234)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file './sphinx.conf'...
Sphinx searchd server started.

Searching

Documents.find_with_sphinx 'why did I write this'

How to install sar on Debian

Tagged io, cpu, sar, usage, report, debian, linux  Languages bash

First install the sysstat package which includes sar:

apt-get install sysstat

Then run crontab -e and add the following to collect stats:

# Collect measurements at 10-minute intervals
0,10,20,30,40,50   * * * *   /usr/lib/sa/sa1
# Create daily reports and purge old files
0                  0 * * *   /usr/lib/sa/sa2 -A

You now have access to IO and CPU usage history.

Use this command see the data for the 5th day of the current month:

sar -f /var/log/sysstat/sa05

And remember to enable collection of statistics:

sudo vim /etc/default/sysstat

How to Fix "bash: warning: setlocale: LC_ALL: cannot change locale (en_US)"

Tagged lc_all, setlocal, locale, debian, ubuntu  Languages 

Tested on Debian.

Edit the default locale:

sudo vim /etc/default/locale

Add the following:

LANGUAGE=en_US.UTF-8
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8

Run these commands:

sudo locale-gen en_US.UTF-8
sudo dpkg-reconfigure locales

Note if you're connecting to a server the issue might be on either your machine or the server.

Nginx start script for Debian

Tagged debian, startup, nginx  Languages bash
#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/opt/nginx/sbin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx
PID=/var/run/nginx.pid

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
        . /etc/default/nginx
fi

set -e

case "$1" in
  start)
        echo -n "Starting $DESC: "
        start-stop-daemon --start --quiet --pidfile $PID \
                --exec $DAEMON -- $DAEMON_OPTS
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        start-stop-daemon --stop --quiet --pidfile $PID \
                --exec $DAEMON
        echo "$NAME."
        ;;
  restart|force-reload)
        echo -n "Restarting $DESC: "
        start-stop-daemon --stop --quiet --pidfile \
                $PID --exec $DAEMON
        sleep 1
        start-stop-daemon --start --quiet --pidfile \
                $PID --exec $DAEMON -- $DAEMON_OPTS
        echo "$NAME."
        ;;
  reload)
          echo -n "Reloading $DESC configuration: "
          start-stop-daemon --stop --signal HUP --quiet --pidfile     $PID \
              --exec $DAEMON
          echo "$NAME."
          ;;
      *)
            N=/etc/init.d/$NAME
            echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
            exit 1
            ;;
esac

exit 0

Installing Scrapyd on Debian

Tagged scrapyd, scrapy, debian  Languages bash

Install Debian pre-requisites

sudo apt-get install python-pip python-dev
# Scrapyd image module requirements
sudo apt-get install libjpeg-dev libfreetype6-dev zlib1g-dev libpng12-dev

Install Python libraries

Customize for your project:

sudo pip install unidecode mysql-python pyes pil

Use pip freeze if you want:

pip freeze > requirements.txt

Install scrapy

git clone https://github.com/scrapy/scrapy.git
cd scrapy
sudo python setup.py install

Install scrapyd

git clone https://github.com/scrapy/scrapyd.git
cd scrapyd
sudo python setup.py install

Deploy project to Scrapyd (http://localhost:6800)

cd project
scrapy deploy

Run

- Start a spider
curl http://localhost:6800/schedule.json -d project=default -d spider=xxx

- List spiders
curl http://localhost:6800/listspiders.json?project=default

- List jobs
curl http://localhost:6800/listjobs.json?project=default

- Cancel job
curl http://localhost:6800/cancel.json -d project=default -d job=4abb6c78fd1a11e28ed9fefdb24fae0a