scrapyd snippets

Installing Scrapyd on Debian

Tagged scrapyd, scrapy, debian  Languages bash

Install Debian pre-requisites

sudo apt-get install python-pip python-dev
# Scrapyd image module requirements
sudo apt-get install libjpeg-dev libfreetype6-dev zlib1g-dev libpng12-dev

Install Python libraries

Customize for your project:

sudo pip install unidecode mysql-python pyes pil

Use pip freeze if you want:

pip freeze > requirements.txt

Install scrapy

git clone https://github.com/scrapy/scrapy.git
cd scrapy
sudo python setup.py install

Install scrapyd

git clone https://github.com/scrapy/scrapyd.git
cd scrapyd
sudo python setup.py install

Deploy project to Scrapyd (http://localhost:6800)

cd project
scrapy deploy

Run

- Start a spider
curl http://localhost:6800/schedule.json -d project=default -d spider=xxx

- List spiders
curl http://localhost:6800/listspiders.json?project=default

- List jobs
curl http://localhost:6800/listjobs.json?project=default

- Cancel job
curl http://localhost:6800/cancel.json -d project=default -d job=4abb6c78fd1a11e28ed9fefdb24fae0a