sam keen's corner of the web

Getting MySQLdb For Python Working on Snow Leopard

I’m just getting into Python and Django programming so I’ve been tinkering with the tutorials online to get familiar with it.  I develop on a Macbook running 10.6.x and the pre-installed Python was working just fine for me until I needed to connect to MySQL for some of the Django work.  To do this you need MySQLdb for python installed.  I’m new to setting up Python environments and I’m sure there is a way to get Apple’s Python working with MySQL but it just was not working for me, so after many, many failed attempts, I gave up and started developing Python on an Ubuntu VM on the Macbook (gotta love apt-get).

Some co-workers at Mozilla then turned me onto a new(ish) package manager for OSX called HomeBrew.  Figuring that Apple’s ’special’ python was the root of my problems above (I’ve been down that road with apache and PHP), I thought I’d replace Apple’s python with one installed using Homebrew and attempt the MySQLdb install again.

Here is a summary of that process.

Install Homebrew:

$ curl http://gist.github.com/raw/323731/HEAD/install_homebrew.rb -o /tmp/install_homebrew.rb

Recursively change the ownership of /usr/local to your user (not the most secure thing to do but I am comfortable with it on my dev laptop.  Keeps you from running brew as root)

$ sudo chown -R $USER /usr/local

Run the installer:

$ ruby /tmp/install_homebrew.rb

Adjust your PATH to get /usr/local/bin at start of path so brew installed is picked up first

I did this by adding the line below to ~/.profile

export PATH="/usr/local/bin:$PATH"

Now you can start installing with brew:

$ brew install python

You will need setuptools to install MySQLdb

$ brew install setuptools

Install 64bit Mysql.  Folks accomplish this in many different ways, from src, binaries, I believe you can event brew install it.  I originally had the 32-bit binary installed (from http://dev.mysql.com/downloads/mysql/) and was unable to get MySQLdb working with that.  So I just downloaded and ran the install for the 64bit version binary.

Now on to MySQLdb for Python,  download it at http://sourceforge.net/projects/mysql-python/

Follow the install in the README file which should be something like this:

 $ tar xfz MySQL-python-1.2.1.tar.gz
 $ cd MySQL-python-1.2.1
 $ # edit site.cfg if necessary
 $ python setup.py build
 $ sudo python setup.py install # or su first

Then test

$ cd
$ python
Python 2.6.5 (r265:79063, Apr 27 2010, 12:36:14)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>

As is often the case in Linux/Unix the ‘no output’ you get after import MySQLdb is the good news you were looking for.  You are most likely all set to use Python and MySQL together now.

Using Unicode To Increase Information Density in Tweets

Quite a while back I grabbed the twitter handle @pdxweather with plans to broadcast the local weather over it.

I first thought I would send out a daily broadcast for today’s weather and tomorrow.  Being Portland, Oregon, I’m centering around the “Chance of Rain” for those days at least as a starting point. This has proven to be quite a challenge to accomplish with 140 characters.

The technology needed to produce such tweets is not all that difficult.  I built a little PHP script that would parse the forecasts from form the weather.gov API.  I then have a PHP script that builds a short summary of that information and tweets it (once a day at this point). http://twitter.com/pdxweather

Attempt 1:

Sun_18th:light rain precip%day:13 precip%eve:29 Mon_19th:light rain precip%day:53 precip%eve:64

(NOAA splits the % chance of rain to day and eve.  I like this feature so I want to have that in the tweets also even though it makes it that much more difficult to stay under 140.)

Lots of room for improvement here.

  • Hard to see the dividing line between the days
  • Not even showing hi/low temps yet and we are at ~95 char so longer weather descriptions (i.e. “Light rain and high wind warning”") along with temps could easily put us over 140
  • Overall, it’s just hard to parse out the individual bits of information

Attempt 2:

Goals:

  • Reduce the overall character count of what we have so far so we can add more info.
  • Make the tweet more ‘human brain parse-able’

I first trimmed the day descriptions to gain some more characters.

Sun18:light rain precip%day:13 precip%eve:29 Mon19:light rain precip%day:53 precip%eve:64

I then determined correct spelling was optional and substituted lite for light (lite seemed a better solution than lght or lt)

Sun18:lite rain precip%day:13 precip%eve:29 Mon19:lite rain precip%day:53 precip%eve:64

These helped on the character count, but you could easily argue that these make the tweet slightly less readable.  What we really need is something that shortens the character count and at the same time increases its readability.  For this I turned to Unicode.  If I could find a single character to replace a word like ‘Sunny’ or ‘Rain’, I could regain many characters to use elsewhere in the Tweet.  I did find some ‘weather’ glyphs in a miscellaneous section of Unicode, unfortunately most seem to render into illegible blobs because of the typical small size you see them at.

I did find

  • sun:☼
  • snow:✽

but nothing good for rain that was legible at small sizes (does this look like rain on an umbrella?  ☔).  So I started scanning the Unicode field for something that ‘looked’ like rain and settled on ⑊.

So using ⑊ for rain, we are at:

Sun18:lite ⑊ precip%day:13 precip%eve:29 Mon19:lite ⑊ precip%day:53 precip%eve:64

I then did a major refactor to the order of things and dropped some ‘not completely necessary’ terms:

Sun18:lite ⑊ 13%day 29%eve Mon19:lite ⑊ 53%day 64%eve

At this point, a great deal of ground has been made on character count, down to 53 characters, or about a 45% reduction.

As a final step, I rely on alpha/digit/symbol boundaries and remove space between them.  In an effort to emphasize boundaries, it is back to Unicode with  → and ▮.

So here it is at 49 chars:

Sun18 lite⑊13%day→29%eveMon19 lite⑊53%day→64%eve

Original at 95

Sun_18th:light rain precip%day:13 precip%eve:29 Mon_19th:light rain precip%day:53 precip%eve:64

Now What

The goal now it to allow this logic to all weather tweets with will involve building a NOAA weather speak to Tweet Weather speak translator.  So the overall architecture will be

  • nightly, hit the NOAA API and and parse/cache the forecast on the local server
  • once a day (or more), use that cached data to tweet the forcast

So I’m off to finish that,  the starting point is here on github.  I’ll document and cleanup that code as I go along

Also looking for feedback on the compaction techniques I’m using for the Tweets, so please comment if you have suggestions

Installing Git Manpages on Snow Leopard

When Using Git do you get this frustration:

$ git help pull
No manual entry for git-pull

If so, there is an easy fix

First, note the git version you are using

$ git --version
git version 1.6.4.2

Then download the appropriate tarball from kernel.org

curl -O http://www.kernel.org/pub/software/scm/git/git-manpages-{YOUR VERSION}.tar.bz2

Then determine your manpath for /usr/local (here it is /usr/local/share/man, yours may be different)

$ cat /etc/manpaths
/usr/share/man
/usr/local/share/man

Lastly, untar the files into the man directory

sudo tar xjv -C /usr/local/share/man -f git-manpages-1.6.4.2.tar.bz2
Now you should get manpages when you issue something like git help merge

10 Minutes to a Leaner LAMP stack

For this quick article I am only going to quickly cover the A and the M of the LAMP stack.

The Setup

Assumptions: You have root on the server in question or you at least have enough server access to effect the configuration of Apache and MySQL.

The test server setup:

  • A Virtual Private server running on Rackspace cloud
  • Ubuntu 9.04 (Jaunty Jackalope)

Few quick commands and viola, you have a LAMP server…

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo locale-gen en_US.UTF-8
$ sudo apt-get install apache2-mpm-prefork php5 php5-dev mysql-server-5.0 mysql-client-5.0 php5-mysql
$ sudo a2enmod rewrite
  • Installed latest Wordpress and added some test posts

Baseline Performance:

Before you start changing anything be sure to record the baseline performance ot the server.  Place the server under load.  I used http_load but anything similar can be used

The Test URL’s that http_load will draw from

$ cat pdxphp-test

http://67.23.47.195/

http://67.23.47.195/2010/02/test-post-1/

http://67.23.47.195/2010/02/test-post-2/

http://67.23.47.195/2010/02/test-post-3/

http://67.23.47.195/2010/02/test-post-4/

http://67.23.47.195/2010/02/test-post-5/

Run the test

$ http_load -parallel 5 -seconds 20 pdxphp-test
83 fetches, 5 max parallel, 1.74373e+06 bytes, in 20.0009 seconds
21008.8 mean bytes/connection
4.14982 fetches/sec, 87182.7 bytes/sec
msecs/connect: 80.6013 mean, 99.26 max, 71.974 min
msecs/first-response: 795.394 mean, 5495.6 max, 225.29 min
HTTP response codes:
  code 200 -- 82

web server under load

To see how much RAM MySQL actually wants, we restart it with Apache stopped

MySQL running on it's own, gobbles up 21M RAM

Apache

Reduce the number of modules loaded

You can start with this.  It will typically not give you incredible results but you will save some resource and it is a good security policy.

You can list your loaded modules with

apache2ctl -t -D DUMP_MODULES
Loaded Modules:
 core_module (static)
 log_config_module (static)
 logio_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 alias_module (shared)
 auth_basic_module (shared)
 authn_file_module (shared)
 authz_default_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cgi_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 php5_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 status_module (shared)

Choosing to unload any of these modules is completely dependent of how you are using the server.  But for many PHP developers can safely unload cgi.

$ sudo a2dismod cgi
Module cgi disabled.
Run '/etc/init.d/apache2 restart' to activate new configuration!
sam@pdx-test:/etc/apache2/mods-enabled$ sudo apache2ctl start

Turn off htaccess

Explanation from Apache.  So all you need to do is take the contents of the .htaccess file in your site’s webroot and place that in the corresponding vhost.  Then turn off htaccess files with AllowOverride None and rm the .htaccess file in the web root after restarting apache.

sudo vi /etc/apache/sites-enabled/wordpress

<Directory /var/www/wordpress>

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

AllowOverride None

</Directory>

rm /var/www/wordpress/.htaccess

Now, how to really affect apache

This is all well and good and Apache will be better off for it but if you had a server that was paging and dying under load, after making these changes, that will probably not change.

The bigger issue shown in the top output above is that, for the amount of physical RAM, there are just too many 20+MB processes

the apache conf file controls these settings (/etc/apache2/apache2.conf on Ubuntu)

The stock settings:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

Possible more appropriate settings for a little personal blog site.  Overall you need to lower MaxClients to the amount you can run without paging.  If this is too few, you probably need more RAM.

<IfModule mpm_prefork_module>
    StartServers          1
    MinSpareServers       1
    MaxSpareServers       5
    MaxClients            5
    MaxRequestsPerChild   1000
</IfModule>

The Results thus far

Top output with Apache Trimmed and under load

MySQL

Now onto MySQL.  We saw above that MySQL is staring up and grabbing ~21M of RAM.  A low to medium volume LAMP server probably doesn’t need all that but, again every situation is different.  Here though are some notes of how to tune the Low Hanging Fruit of MySQL

If you don’t use it, Turn it off

Same as with Apache modules, turn off what you do not need.  Storage engines are a great place to start.  Very few frameworks (including WP),  utilize InnoDB by default.  Most use good old MyISAM.

Stopping MySQl from loading the InnoDb engine (and its predecessor:bdb) is quite simple, the lines are in my.cnf, just uncomment them

skip-innodb
skip-bdb

Now there are many, many MySQL config options that affect the amount of resource it uses.  I’ll just cover a couple here, the key buffer and the query cache

The stock settings for these are

# used for holding built indexes
key_buffer              = 16M
# largest query that is cache-able
query_cache_limit       = 1M
# used for caching queries
query_cache_size        = 16M

For many types of sites,  these can be a bit large,  for small sites, some suggested settings might be:

# used for holding built indexes
key_buffer		= 4M
# largest query that is cache-able
query_cache_limit       = 500K
# used for caching queries
query_cache_size        = 8M

To see what the runtime values are concerning the query cache and key buffer use (once server has been under load)

mysql> SHOW STATUS WHERE Variable_name LIKE 'Key_blocks_%' OR Variable_name LIKE 'Qcache_%';
+-------------------------+----------+
| Variable_name           | Value    |
+-------------------------+----------+
| Key_blocks_not_flushed  | 0        |
| Key_blocks_unused       | 13369    |
| Key_blocks_used         | 27       |
| Qcache_free_blocks      | 4        |
| Qcache_free_memory      | 16326632 |
| Qcache_hits             | 5969     |
| Qcache_inserts          | 95       |
| Qcache_lowmem_prunes    | 0        |
| Qcache_not_cached       | 410      |
| Qcache_queries_in_cache | 46       |
| Qcache_total_blocks     | 106      |
+-------------------------+----------+
11 rows in set (0.00 sec)

As you reduce key_buffer and query_cache_size, check the output of the SHOW STATUS command shown above while the server is under load to verify that you are not running low on Key_blocks_unused or Qcache_free_memory

The Final result of our tinkering

So now that we have trimmed Apache and MySQL lets see what the top output looks like with the server under load

Our CPU load has reduced (idle is up to ~65%). More importantly we are no longer paging and we still have a bit of RAM left over.  MySQL is running with ~8M of RAM, down from ~21M.

Also as seen below, we’ve roughly doubled our performance.  Up to 6.85 req/sec and first response in down to 276ms.

$ http_load-parallel 5 -seconds 20 pdxphp-test
137 fetches, 5 max parallel, 4.69493e+06 bytes, in 20 seconds
34269.6 mean bytes/connection
6.85 fetches/sec, 234747 bytes/sec
msecs/connect: 80.6284 mean, 96.353 max, 73.043 min
msecs/first-response: 276.766 mean, 447.707 max, 225.995 min
HTTP response codes:
  code 200 -- 137

Links:

Some Experiments with Canvas Drag and Drop

I’ve been toying around with canvas tag and learning the ins and outs of Drag and Drop

I created a little proof of concept app that allows you to add markers to an image.  The meta about the markers (relative location, marker symbol, desc, color, etc) are stored in a DB on another server using JSONP requests.

See the demo here (FF3+ recommended)

The code repo is currently on google code

ipad: My impressions

Sexy little (well medium i guess) device. As a software developer though I cannot see it replacing my laptop (currently a msi u123). I don’t see writing code on a ipad.
Also, it would not be replacing my phone. I guess we will be able to use skype (or skype-like services) on the 3G models. But still, the form factor of the iPad does not make for a convenient phone.

I do see the iPad as the dream machine for bloggers (though I’m surprised there is no camera) and makes an excellent “couch computer”. I believe in many people’s minds, it competes in the iPod touch market.

Do we see Ubuntu or Android ever being ported to this device? That could make things interesting.

vnstat : excellent networking tool

I recently found myself considering moving one of our servers to the Rackspace Cloud. I needed to get an estimate of the cost for bandwidth usage.
I remembered I had installed vnstat on that server when it was deployed.  This made the bandwidth calculation completely simple:

Just have vnstat give me the bandwidth sliced by Month:

$ vnstat -m

 eth0  /  monthly

   month         rx      |      tx      |   total
-------------------------+--------------+--------------------------------------
  Aug '09       7.00 GB  |     8.48 GB  |    15.48 GB   %::
  Sep '09      14.98 GB  |    15.75 GB  |    30.72 GB   %%%::::
  Oct '09      31.70 GB  |    14.79 GB  |    46.50 GB   %%%%%%%%:::
  Nov '09      45.46 GB  |    14.64 GB  |    60.10 GB   %%%%%%%%%%%:::
  Dec '09      73.71 GB  |    15.40 GB  |    89.11 GB   %%%%%%%%%%%%%%%%%%::::
  Jan '10      66.25 GB  |    16.89 GB  |    83.14 GB   %%%%%%%%%%%%%%%%::::
-------------------------+--------------+--------------------------------------
 estimated     94.97 GB  |    24.21 GB  |   119.18 GB

The install of vnstat on Ubuntu can be done with apt-get:

sudo apt-get install vnstat

Remember, once installed to initiate you vnstat DB.  Just execute vnstat and it will tell you how.   vnstat -u -i eth0 in my case

Then check the man page for the command to see all the different ways you will be able to slice and dice your network data

Your First PHP Development Environment

I’m teaching a PHP/MySQL course at the local community college this fall.  First order of business is to get everyone set up with a PHP development environment.  I decided to go ahead and post the strategy for the course as a post here in case it may be helpful others just starting out in PHP development.

This post is meant to help the individual who is just setting out to learn PHP web development and has no previous programming experience.  I’m listing a set of tools to help in that quest and giving some pointers as to how to initially configure them (though part of the reason these tools were chosen was that they require very little tweaking before you can start using them).

The OS used in this example is Windows, but all theses tools are cross platform (deliberately chosen for that reason), and the instructions will only change slightly for other OS’s.

Below are examples of installing a client (the web browser), a server stack (PHP and MySQL) and a code editor.  They’ve been chosen based on:

  • ease of install and use for someone new to all this technology
  • cross platform
    versions available for Windows and Linux flavors including OS X
  • extensibility
    In particular, Firefox and Netbeans.  As the user grows comfortable with these tools, they will discover that they contain many features to aid in web development.  (firefox: firebug, web-dev toolbar   netbeans: integrated source control, xdebug, etc)

PHP is a cross platform technology at heart so that enables a plethora of alternatives to the choices I’ve made above. So feel free to explore the other options and find the tools that work best for you.

Firefox

Install

Get the files from: http://www.mozilla.com/en-US/firefox/firefox.html

Install the firebug addon : http://getfirebug.com/

XAMPP

Install on Windows (windows 7 64bit edition shown)

Get the files from: http://sourceforge.net/projects/xampp/files

xampp0

xampp-install

XAMPP should do this by default, but be sure to install at C:\

xampp-intall-root

In service section check

  • Install apache as service
  • Install mysql as service

This causes those services to automatically start when Windows starts

xampp-services

Test install by opening a browser and going to http://localhost

xampp-browser-test

Netbeans Editor

Netbeans PHP IDE

Download here

Configure after Install

Apply Updates

Picture 59

Go to Help > Check For Updates

Test

Now create a new PHP application project called {YOUR NAME}-week1

netbeans-test1

netbeans-test2

Place the project in the XAMPP htdocs folder

Picture 64

Confirm the URL your project will be found at. (In our case, Netbeans should make the right guess here, so just confirm and click ‘Finish’

Picture 62

Editor Will Open and Netbeans will automatically create a starter index.php file for you

Picture 65

Clean up some unneeded windows

close these windows (you can always get them back later if you like)

  • navigator
  • files
  • services
  • palette
Picture 66

Now to make a few edits and create a new file so we can start to get familiar with the IDE.

Add some PHP

netbeans-add-php

See the file in a browser by clicking the green Play (>) button on the toolbar.  The Browser should open at the URL for the file.

Picture 67

create another file called new-file.php

  • Menu: File > New File
  • Choose category PHP, File Type PHP Web Page
  • Next
  • Change the filename to new-file.php
  • Finnish


Picture 68

In new-file.php create a hyper link to index.php and in index.php create a link to new-file.php
Picture 69
The resulting filesystem
Picture 70
Note, the nbproject folder is used by Netbeans (it is full of projects metadata).  You can simply ignore it (but don’t edit or delete it)

Portland Restaurant Health Map

Inspired by Toby Segaran’s creation of a heat map of restaurant health inspection scores for San Francisco, I set out to do the same for Portland, Oregon.  I was able to scrape establishment records from the existing Multnomah County Food Establishment Inspections Search.  Then I got lat/longs for those addresses using Google’s geocode API.  Using the gathered establishment records I was able to scrape the MCHD search site once more for the Inspection records of those businesses.  Lastly, I displayed the data on a Google map graded green for good score down through to red for the not so good scores.

See the map here. (It’s plotting ~3000 markers so it can take a bit of time to load).

Health Scores Map

After looking at the map I realized, just as Toby did, other than displaying ‘concerning’ red areas, this map exposes restaurant ‘clusters’ that you may not have known about and can then choose to explore.

Below is some more detail on the techniques I used to build this particular map.

Get the data

I searched around the web and found the Multnomah County Food Establishment Inspections Search.  I made an attempt to get an export of the back-end data for this by calling county health.  They were helpful but when I received the export of the data it was just the business records, not the inspection records for the businesses.  I think they may have just misunderstood me and  I probably could have pressed more and gotten data I needed, but I thought it would be fun the polish the old web scraping skills instead.  Also, a scraping strategy would allow me to refresh the data whenever needed and I wouldn’t have to keep bugging MCHD.

Selenium is a suite of web app testing tools.  Selenium IDE is a firefox addon that records all the actions you take in the browser.  Meant for building functional testing scripts but doubles as a great web scrape script building tool.  What I find most useful about Selenium IDE is that after recording your script you can export it to the xUnit version of your choice, PHPUnit in my case.  This gives you the starting point for your script and you can then add things like database persistence for the information you are scraping and utilize the phpUnit assertion methods to let you know if your script has broken.  Then, you can use a tool called Selenium Remote Control to re-run the script (and drive the browser) anytime you need.  This intro video does a great job of explaining some of Selenium’s features.

mchd-running

Selenium Script Churning Through Data

So I needed to accomplish 4 things.

(I included links to the scripts I used, these are by no means ‘production ready’ but some might find them useful as a starting point to their own projects.  Also here is the DB sql used.)

Step I: Pull the establishment records from the MCHD site

I noticed that on the MCHD site, if you just clicked ’search’ without any criteria you were taken to a holistic, paginated set of the records (currently 130 pages for total of just over 3000 records).  So my first script was started with selenium IDE and worked through these 130 pages, gathering establishment meta-data and storing it in a mysql database. The Script

Step II: Pull the inspection scores for the gathered establishments from the MCHD site.

This was done by again, using Selenium IDE to start a PHPUnit script that scraped the inspection record for each gathered establishment.  It simply queries all the establishment ids from out database and builds URLs to scrape the inspection data for each business and store that information in an inspections table in the database. The Script

Step III: Geocode the Establishment Records using Google

In preparation for displaying the data on a map, I needed to get the lat/long for each establishment in the database.  I did this using Google’s Geocode service but there are many options for services that can accomplish this.  Just be sure you stay within their Terms Of Use. The Script

Step IV: Display the Establishments on something akin to a heat-map

This is the easy part once you’ve done the heavy lifting in steps I,II and III.   It involves one query from the Db to pull the score, latitude, and longitude for each establishment we know about.  Then using a fairly simple php web page we build the HTML and javascript to display this data on a Google map.  The Script

SOLR : very impressive lucene based search tool

Very impressed with Solr and the development community behind it. http://lucene.apache.org/solr/

1.5 days of research, submitted bug and resulting immediate patch and I have a fully functional search engine for a project at work.

If you’ve ever thought it would be nice to have your very own google type appliance, I highly recommend you check out solr and the underlying IR engine: lucene.