NER with Python and NLTK

Python posted 9 months ago by christian

Code from https://gist.github.com/322906/90dea659c04570757cccf0ce1e6d26c9d06f9283

   1  import nltk
   2  
   3  def named_entities(text):
   4  	sentences = nltk.sent_tokenize(text)
   5  	tokenized_sentences = [nltk.word_tokenize(sentence) for sentence in sentences]
   6  	tagged_sentences = [nltk.pos_tag(sentence) for sentence in tokenized_sentences]
   7  	chunked_sentences = nltk.batch_ne_chunk(tagged_sentences, binary=True)
   8  	def extract_entity_names(t):
   9  		entity_names = []
  10  		if hasattr(t, 'node') and t.node:
  11  			if t.node == 'NE':
  12  				entity_names.append(' '.join([child[0] for child in t]))
  13  			else:
  14  				for child in t:
  15  					entity_names.extend(extract_entity_names(child))
  16  		return entity_names
  17  	entity_names = []
  18  	for tree in chunked_sentences:
  19  		entity_names.extend(extract_entity_names(tree))
  20  	return set(entity_names)

Tagged ner, python

SQLAlchemy example

Python posted 9 months ago by christian

SQLAlchemy example:

   1  from sqlalchemy import *
   2  from sqlalchemy.ext.declarative import declarative_base
   3  from sqlalchemy.orm import sessionmaker
   4  
   5  engine = create_engine("mysql://username:password@localhost/database_name")
   6  engine.echo = False
   7  
   8  Base = declarative_base()
   9  
  10  class Entry(Base):
  11      __tablename__ = "entries"
  12      id = Column(Integer, primary_key=True)
  13      title = Column(String(255))
  14      url = Column(String)
  15  
  16      def __repr__(self):
  17          return "<Entry('%s', '%s')>" % (self.title, self.url)
  18  
  19  # Set up handles
  20  entry_table = Entry.__table__
  21  metadata = Base.metadata
  22  metadata.create_all(engine)
  23  
  24  # Start a session
  25  Session = sessionmaker(bind=engine)
  26  session = Session()
  27   
  28  # Query entries
  29  entries = session.query(Entry) \
  30      .filter(Entry.title != 'Zermatt')
  31  
  32  # Print all entries
  33  for entry in entries.all():
  34      print entry.update_named_entities()
  35  
  36  # Print first entry
  37  entry = entries.first()
  38  
  39  # Update entry
  40  entry.title = 'Zermatt, Verbier'
  41  
  42  # Commit changes
  43  session.commit()

Tagged sqlalchemy, python

How to extract the palette from an image with Python

Python posted 9 months ago by christian

Detect the color palette of an image:

   1  # See https://github.com/99designs/colorific/blob/master/colorific.py
   2  # min_saturation = The minimum saturation needed to keep a color
   3  # min_prominence = The minimum proportion of pixels needed to keep a color
   4  import colorific
   5  palette = >>> colorific.extract_colors('test.jpg', min_prominence=0.1)
   6  colorific.print_colors('test.jpg', palette)

Example

This example will scan a directory for images and create an HTML file showing the images and the detected color palette for each image:

   1  import colorific
   2  import glob
   3  
   4  html = open("index.html", "w")
   5  
   6  for filename in glob.glob('./images/*'):
   7      html.write("<div>")
   8      html.write("<img width=\"150px\" src=\"" + filename + "\">")
   9      print filename
  10      palette = colorific.extract_colors(filename)
  11      print palette
  12      for color in palette.colors:
  13          print color
  14          hex_value = colorific.rgb_to_hex(color.value)
  15          html.write("""
  16              <div style="background: {color}; width: 500px; height: 50px; color: white;">
  17              {prominence}
  18              </div>
  19          """.format(color=hex_value, prominence=color.prominence))
  20          html.write("</div>")
  21  
  22      if palette.bgcolor != None:
  23          hex_value = colorific.rgb_to_hex(palette.bgcolor.value)
  24          html.write("""
  25              <div style="background: {color}; width: 500px; height: 50px; color: white;">
  26              {prominence}
  27              </div>
  28          """.format(color=hex_value, prominence=palette.bgcolor.prominence))
  29          html.write("</div>")

Issues

Note, on OSX I had to edit colorific.py (/Library/Python/2.7/site-packages/colorific-0.2.0-py2.7.egg/colorific.py) slightly to get it to work:

   1  #from PIL import Image as Im
   2  #from PIL import ImageChops, ImageDraw
   3  import Image as Im
   4  import ImageChops, ImageDraw

Before this, I got this error:

   1  ImportError: No module named PIL

Tagged python, palette, colors, extract, colorific

Removing BOM with Ruby 1.9

Ruby posted 9 months ago by christian

Removing the BOM with Ruby 1.9:

   1  body = File.open(file, "r:bom|utf-8").read()
   2  File.open(file, 'w') do |xass| 
   3    xass << body
   4  end

In Ruby 1.8 you could do something like this:

   1  body = body[3..-1] if body[0..2] == "\357\273\277"

Tagged bom, ruby

How to use History.js or jQuery Address for HTML5 pushState navigation

JavaScript posted 9 months ago by christian

Please don’t use History.js. It’s not maintained and has over 100 issues. Use jQuery Address and remember to read the documentation.

Example usage:

   1  # muy importante
   2  $.address.state('/') 
   3  # mas importante
   4  $.address.value(window.location.pathname + window.location.search)
   5  # donde ocurre
   6  $.address.change (event) ->
   7    loadPage()

Another example:

   1  $.address
   2    .state('/')
   3    .path(window.location.pathname + window.location.search)
   4    .change (event) =>
   5      loadPage()

jQuery address examples

Tagged history.js, jquery-address, pushstate, replacestate, html5