Simple asynchronous form submission using jQuery

It’s not hard to submit a form asynchronously in jQuery. Here’s a little example. I put a block like this right below my form element.

<script type="text/javascript">
$("#my_form").submit(function() {
    $.ajax({
        type: "POST",
        url: "/path/to/post/handler/",
        data: $("#my_form").serializeArray(),
        success: function() {
            // Whatever you want here, like close dialog box, etc.
        }
        });
    return false;
});</script>

I did have to be careful to make the form like this:

<form action="javascript:true;">...
Posted in General | Tagged , , , , , , | 2 Comments

Breaking out of the middle of a try block in Python

I had some code where I wanted to break out of a try block in Python. The code looked something like this:

    try:
        print "First"
        if some_condition:
            # What do I put here to break out of the try block?
        print "Second"
    except Exception as e:
        if e:
            print "Exception"
    print "Last"

(If some_condition is true, I want the output to print “First” and “Last”)

Neither break or continue works as expected, since “try” isn’t a loop. I could put the body of the try block in a function, and then at the “if some_condition” section, I could just return, but that seemed messy. I never found a clean enough solution, but one possibility is this wonky syntax:

for _ in [True]:
    try:
        print "First"
        if some_condition:
            # Now, since we're in a loop, we can break out of it.
            break
        print "Second"
    except Exception as e:
        if e:
            print "Exception"
            print type(e)

print "Last"

But, it’s kind of ugly, and not clear what’s going on, so I decided not to do it that way.

I’m really surprised that there isn’t a standard syntax for this. Something like “raise” but that isn’t an exception. I guess I could raise a special value and handle it specially in th except block, but that seems messy as well.

Posted in General | Tagged , , , , | 7 Comments

Twitterbots unite!

So, when you post a link to twitter, what happens?

Within a matter of seconds, you’ll see traffic from at least all of the following:

50.18.61.103 - - [10/Mar/2011:07:46:15 +0000] "HEAD / HTTP/1.1" 404 221 "-" "UnwindFetchor/1.0 (+http://www.gnip.com/)"
184.72.240.15 - - [10/Mar/2011:07:46:17 +0000] "HEAD / HTTP/1.1" 404 183 "-" "MetaURI API/2.0 +metauri.com"
199.59.149.24 - - [10/Mar/2011:07:46:17 +0000] "HEAD / HTTP/1.1" 404 221 "-" "Twitterbot/0.1"
67.195.115.120 - - [10/Mar/2011:07:46:18 +0000] "GET / HTTP/1.0" 200 4210 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
38.113.234.181 - - [10/Mar/2011:07:46:18 +0000] "HEAD / HTTP/1.1" 404 164 "-" "Voyager/1.0"
89.151.99.94 - - [10/Mar/2011:07:46:19 +0000] "GET / HTTP/1.1" 200 4191 "-" "Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot"
89.151.116.53 - - [10/Mar/2011:07:46:24 +0000] "GET / HTTP/1.1" 200 4191 "-" "Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot"
50.16.20.4 - - [10/Mar/2011:07:46:57 +0000] "HEAD / HTTP/1.1" 404 164 "-" "PycURL/7.18.2"
204.236.254.109 - - [10/Mar/2011:07:47:10 +0000] "HEAD / HTTP/1.1" 404 164 "-" "PostRank/2.0 (postrank.com)"
88.73.99.54 - - [10/Mar/2011:07:52:56 +0000] "HEAD / HTTP/1.1" 404 183 "-" "Twitmunin Crawler http://www.twitmunin.com"

And that’s just the tip of the iceberg. I’ve filtered out many repeated HEAD requests (which, BTW, I now know I need to properly implement for these bots to be happy). And, over the next 10-30 minutes, I’ll continue to see more hits from other places as well (Baidu, etc.) Wow, amazing how one simple action like posting a link in a tweet can generate so much traffic.

Posted in General | 1 Comment

pymacs, ropemacs, and virtualenv, all at the same time.

So, you’re developing for Python, and you want to use rope and ropemacs and pymacs, but it’s totally busted when you use your virtualenv.

In addition, the packaged versions of pymacs+ropemacs for ubuntu 10.10 (and 11.04) are also kind of busted, because they complains with really odd errors.

Here’s the solution:

Install the latest, greatest versions of everything into your virtualenv:

. ./env/bin/activate
pip install -e hg+https://bitbucket.org/agr/rope#egg=rope
pip install -e hg+https://bitbucket.org/agr/ropemacs#egg=ropemacs
pip install -e hg+https://bitbucket.org/agr/ropemode#egg=ropemode
pip install -e git+https://github.com/pinard/Pymacs.git#egg=pymacs

Unfortunately, pymacs doesn’t cleanly install, so then need to:

cd env/src/pymacs
make

Once that’s done, you should have env/src/pymacs/pymacs.el. Great. Now you just need to import that from your .emacs file. So, add something that looks like this to your .emacs:

(setq virtual-env (getenv "VIRTUAL_ENV"))

(if (not (equal virtual-env 'nil))
      (setq load-path (append
                 (list (concat virtual-env "/src/pymacs" ))
                 load-path))
      (let ((foo 'bar))
      (require 'pymacs)
      (pymacs-load "ropemacs" "rope-")
      (setq ropemacs-enable-autoimport 't)
      ))

Activate your virtualenv, and run emacs from inside there, and you should be good to go.

Posted in General | Tagged , , , , , , | Leave a comment

An extremely simple config pattern for Python

So, you’re developing a Python application.
And, you have needs for “production” versus “development” configs.

If you’re using a web framework, you might have some support for doing this, but what about your other modules that you’d like to configure in a similar way?

What you sort of want to be able to say, anywhere, is something like this:

import config

if config.MODULE['parameter'] == 'value':
    # do something
else:
    # do something else

And, you want “import config” to automatically decide (via some means) if it’s in production or development.

Great, here’s a really simple pattern for you.

Make a directory named “config” put it somewhere on your python import path. (It’s okay if this is app.config or some_module.config, whatever you want.) Make the __init__ look like this:

class Config(object):
    pass

if some_condition:
    from config.production import ProductionConfig
    config = ProductionConfig
else:
    config = DevelopmentConfig

You need to write some_condition yourself. You’ll probably do something like check for existance of a file, check the hostname, look at an environment variable, whatever.

Then, create config/production.py and config/development.py and make them look pretty much like this:

from config import Config
class DevelopmentConfig(Config):
    MODULE = { 'param': 'value' }
    OTHER_MODULE = { 'other_param': 'other_value' }
    # etc.

Make sure that production and development configs both specify the same arguments. If you have shared things you want to configure, then you can make “base.py” or “shared.py” and derive from that.

Happy hacking!

Posted in General | Tagged , , , , , | Leave a comment

Making pep8, pyflakes, etc work with virtualenv.

Unfortunately, the pre-built binaries for the Python pep8 checker and pyflakes both point directly at #!/usr/bin/python. (FYI I have at least one git pull request out to fix this).

This means that if you’re using virtualenv, they’ll complain about a ton of missing packages when you actually just have them in your virtualenv environment.

Since these “binaries” are just simple wrappers, it’s fairly easy to replace them with local copies. I’ve always put ~/bin early in my PATH so that I can override things like this. So, I just created ~/bin/pep8 and ~/bin/pyflakes that each look like this:

pep8:

#!/usr/bin/env python
# EASY-INSTALL-ENTRY-SCRIPT: 'pep8==0.5.0','console_scripts','pep8'
__requires__ = 'pep8==0.5.0'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('pep8==0.5.0', 'console_scripts', 'pep8')()
    )

pyflakes:

#!/usr/bin/env python

import sys
from pyflakes.scripts.pyflakes import main
sys.exit(main(sys.argv[1:]))

And this is why you wan to always say #!/usr/bin/env python instead of #!/usr/bin/python.

Posted in General | Leave a comment

What am I doing wrong?

You know, it’s very obvious that some python modules are high quality, and others are utter crap. Please tell me what I’m doing wrong:

import logging
logging.basicConfig(level=logging.DEBUG)
logging.debug("Hello")

Produces no output on console. Oh? Maybe I need to specify a stream handler? Tried that, with sys.stderr, and still nothing. Maybe I need to set a file handler? Tried that.

Also, behavior is different in an interactive shell versus saved to a file and invoked by commandline python. Jeebus I hate stuff like this.

Posted in General | Leave a comment

Extremely simple XML to Python converter

There are a ton of different ways you can encode XML, but for the style that looks like this:

<xml>
<name>Bob</name>
...

You can use this nice little function to convert it to a native Python object.  I’m using this to turn the results of the LinkedIn API from XML to Python, which then gets stored as JavaScript in MongoDB.

Set “force_list” to a list of field names that you know are list-values and it will convert those to lists in Python.  Yeah, that could be cleaner and more automatic.  Not really that proud of this, but it does work…

import xml.dom.minidom

def typeconvert(str_data):
    try:
        int_value = int(str_data)
        return int_value
    except:
        pass
    try:
        float_value = float(str_data)
        return float_value
    except:
        pass

    if str_data.lower() == 'false':
        return False
    if str_data.lower() == 'true':
        return True
    if str_data.lower() == 'null':
        return None

    return str_data

def xml_to_py(xml_text, force_list=[], convert_types=True):

    def setval(data, key, value):
        if key in force_list:
            if key not in data:
                data[key] = []
            data[key].append(value)
        else:
            if key in data:
                logging.warn("WARN: key %s, value %s in XML seems like a list" % (key, value))
                pass
            data[key] = value

    def recurse(node):
        nodename = node.nodeName
        result = {}
        if node.nodeType == xml.dom.Node.TEXT_NODE:
            nodetext = node.data.strip(' \n')
        else:
            nodetext = ''
        for child in node.childNodes:
            if child.nodeType == xml.dom.Node.TEXT_NODE:
                nodetext += child.data.strip(' \n')
            else:
                (newtext, newdata) = recurse(child)
                if newtext and newdata:
                    raise Exception("Can't have both text and data")
                if newtext != '':  # newtext could be an integer with value 0
                    if convert_types:
                        newtext = typeconvert(newtext)

                    setval(result, child.nodeName, newtext)
                elif newdata:
                    setval(result, child.nodeName, newdata)

        return (nodetext, result)

    dom = xml.dom.minidom.parseString(xml_text)
    (_text, pydict) = recurse(dom)
    return pydict
Posted in General | Tagged , , , , | Leave a comment

Why I’m unhappy with the Pyramid web framework.

Here I am, writing this post instead of coding.  Why?  Because I’m having to read through the Pyramid source code trying to figure out what the heck is going on with configuration.  Or, to be more specific, how can I load module X when I’m in production, but module Y when I’m in development.

I find myself in this state pretty often, and I don’t mind reading the source code.  Often times when using other frameworks, I’d learn something new about Python when reading the source.

At it’s heart, the problem with Pyramid is also what makes it great:  They don’t prescribe any one way to do anything.  It’s totally agnostic to which database, template engine, session framework, and auth/auth framework you want to use.  This is what I like.  What I don’t like is that it also seems to not be able to make up it’s mind about some things.  Is it repoze? Is it zope?  Is it paster?  Is it pyramid?

When I read the source to Pyramid, it makes me throw up in my mouth a little bit.  Here are the high level points about Pyramid that really get on my nerves:

  • Configuration.  You can configure this thing in 3 different ways.  zope’s ZCML (XML-derived language), via the paster config language (looks like Microsoft .INI files), and via declarative configs (the Python code).   Why in the world they need 3 different ways to configure 1 framework is beyond me.

    There’s a class for “Settings” and 2 different classes both named “Configuration”, one in pyramid.config, one in pyramid.configuration with odd deprecation messages in both places.  How can a web framework that just hit 1.0 have deprecated APIs?

  • Reliance on zope and repoze, paster, WebOb, Pygments, and a million other little libraries.  This means that figuring out something simple by reading code or docs leads down to bouncing between 2 or even 3 different libraries, trying to understand how they all fit together.  This is unnecessary and annoying.
  • zope.interface  Yes, this is super-annoying.  I’m not sure what value it adds, and seems very non-Pythonic to me.  I’ve never seen any other code that does this.
  • Anything related to zope.  It sucks.  ZODB?  Seriously?  Who would start a project today using this thing?  It seems to have stuck it’s legacy fingers in all over the place.  For example, routes vs. traversal.  Every other framework I’ve ever seen/used is doing something like routes.  Traversal (as best as I can tell) seems to be a ZODB-ism that’s worked its way in there, and confuses people all the time.
  • Lack of organized documentation.  The Pyramid web site looks great, but the actual contents of the documentation leaves a lot to be desired.  Figuring out simple things like configuration parameters, routes or traversal, which database to use (and how), user & session objects, auth/auth, etc.  It’s all a mess.  Each section starts of with some great version of “Pyramid supports whatever you want” and then devolves into minutia of several possible half-baked solutions.

My recommendations for the Pyramid team:

  1. Ditch zope.  Yes, the whole thing.  Ditch ZCML, ZODB and traversal.
  2. Ditch paster.  Like Pyramid itself, this feels good on the surface, but gets itchy under your skin.
  3. Ditch repoze.  Take the parts you depend on, and pull them right into Pyramid itself.  Don’t depend on all of repoze just because you can.
  4. Keep the flexibility, but start with a sane default.  Default to routes.  Default to a reasonable session implementation.  Default to a reasonable auth/auth implementation.

I’m stuck with Pyramid for now, but since I’m still fairly early in this process, I’m going to start looking around to see what it takes to switch to another framework (likely flask or bottle, but I’m not sure)

Posted in General | 13 Comments

A simple pattern for short URLs for objects in MongoDB.

So, you need short URLs for your pages, and pages are objects in your MongoDB database. How do you generate these short ids?

The pattern I’m using is quite simple, but requires that you create a unique index on a field called “short_id.”   Since I’m using minimongo, it’s easy to declare these Indexes programatically, and I can override the save() method on an object to do “extra stuff”, like set the short_id and retry if there’s a conflict.  Here’s how the code came out:

import random
import string
from pymongo.errors import DuplicateKeyError
from minimongo import Model, Index, MongoCollection

class TestModel(Model):
    mongo = MongoCollection(database='test',
                            collection='test_model')
    indices = [Index('short_id', unique=True)]

    def save(self, *args, **kwargs):
        short_id_length = 3
        retry_count = 0
        short_chars = string.letters + string.digits
        if hasattr(self, 'short_id'):
            return super(Model, self).save(*args, **kwargs)
        while True:
            try:
                self.short_id = ''.join(
                    [random.choice(short_chars) for x in
                     xrange(short_id_length)])
                kwargs['safe'] = True
                super(Model, self).save(*args, **kwargs)
                return
            except DuplicateKeyError, e:
                retry_count += 1
                if retry_count > 3:
                    short_id_length += 1
                    retry_count = 0

Then, when I create a TestModel and save it, the short_id field is filled in automatically, and I can use that in my URL generation.

It’s this exact kind of use case that made me want to write minimongo.  Simple method overrides for behavior like this can be really powerful.  Also note how the unique index on the short_id field is automatically created.  Sweet!

Posted in General | Tagged , , | Leave a comment