~blog~

Adding search to a Django site in a snap

Posted on 16 August 2008, 11:20 a.m. | 16 comments
Category: Web Development | Tags: django, search

Search is a feature that is -- or at least, should be -- present on most sites containing dynamic or large content.

There are a few projects around to tackle that. Here's a non-exhaustive list: djangosearch, django-search (with a dash), django-sphinx.

Those search engines are great, but they seem like overkill if you just need a simple search feature for your CMS or blog.

To deal with that, I've come up with a generic and simple trick. All you need is copy/paste the following snippet anywhere in your project:

import re

from django.db.models import Q

def normalize_query(query_string,
                    findterms=re.compile(r'"([^"]+)"|(\S+)').findall,
                    normspace=re.compile(r'\s{2,}').sub):
    ''' Splits the query string in invidual keywords, getting rid of unecessary spaces
        and grouping quoted words together.
        Example:
        
        >>> normalize_query('  some random  words "with   quotes  " and   spaces')
        ['some', 'random', 'words', 'with quotes', 'and', 'spaces']
    
    '''
    return [normspace(' ', (t[0] or t[1]).strip()) for t in findterms(query_string)] 

def get_query(query_string, search_fields):
    ''' Returns a query, that is a combination of Q objects. That combination
        aims to search keywords within a model by testing the given search fields.
    
    '''
    query = None # Query to search for every search term        
    terms = normalize_query(query_string)
    for term in terms:
        or_query = None # Query to search for a given term in each field
        for field_name in search_fields:
            q = Q(**{"%s__icontains" % field_name: term})
            if or_query is None:
                or_query = q
            else:
                or_query = or_query | q
        if query is None:
            query = or_query
        else:
            query = query & or_query
    return query

What the above does is generate a django.db.models.Q object (see doc) to search through your model, based on the query string and on the model's fields that you want to search. Importantly, it also analyses the query string by splitting out the key words and allowing words to be grouped by quotes. For example, out of the following query string...

'  some random  words "with   quotes  " and   spaces'

...the words 'some', 'random', 'words', 'with quotes', 'and', 'spaces' would actually be searched. It performs an AND search with all the given words, but you could easily customise it to do different kinds of search.

Then, your search view would become as simple as:

def search(request):
    query_string = ''
    found_entries = None
    if ('q' in request.GET) and request.GET['q'].strip():
        query_string = request.GET['q']
        
        entry_query = get_query(query_string, ['title', 'body',])
        
        found_entries = Entry.objects.filter(entry_query).order_by('-pub_date')

    return render_to_response('search/search_results.html',
                          { 'query_string': query_string, 'found_entries': found_entries },
                          context_instance=RequestContext(request))

And that's it! I use this on a site that has about 10,000 news items and it works pretty fast... And I've just added the same thing on this blog, although I don't have so many entries to search through yet :)

Now you have no excuse not to add a search box to your site! ;)

Comments

# Michael Warkentin 17 Aug 2008, 2:13 a.m.

Thanks for this! Was looking for a simple search implementation for a project I'm working on!

Cheers.

# anonymous 18 Aug 2008, 4:03 a.m.

Please note that with MySQL backend Django can use boolean full-text: http://www.djangoproject.com/document...

# Ryan Flores 20 Aug 2008, 10:11 a.m.

Works great. Thank you.

# Julien Phalip 20 Aug 2008, 10:13 a.m.

@Michael and Ryan

Thanks for dropping a line. I'm glad it helped! ;)

# Andrew Brehaut 22 Aug 2008, 2:33 p.m.

Here is a comparable (if a bit golfed) version:

def get_query(q, fields):
reduce(Q.__or__, [Q(**{'%s__icontains'%f:re.sub('(^[\'"])|([\'"]$)','',p[0])}) for p in re.findall('(("[^"]+")|(\'[^\']+\')|(\S+))', q) for f in fields])

# krylatij 18 Oct 2008, 4:05 a.m.

There is better way to do this:
or_query = None
for field_name in search_fields:
...
if or_query is None:
or_query = q
else:
or_query = or_query | q
Do:
or_query = Q()
for field_name in search_fields:
...
or_query = or_query | q

# Mohammad Rafiee 17 Jan 2009, 8:18 p.m.

great thanks dude!

i used this code it was very easy and useful.

there is not any resource about searching in the django models in internet even in django website and this piece of code was very useful for me

i suggest you to make an application named search or django search with more flexibility and put it into the internet

for example fixings could be like these:
1. relationships like many to many fields and foreign keys
2.searching in multiple models
3.searching in multiple applications

# Julien Phalip 18 Jan 2009, 7:23 p.m.

Mohammad, thank you for your comment. I'm glad this code was useful to you. I've thought of making a small app for this but at this stage it is so simple it's best kept as a code snippet. As for your point #1, you can already search external relationships by using a double underscore, for example:

entry_query = get_query(query_string, ['title', 'body', 'author__name'])

I hope this help ;)

# Nick Sergeant 20 Jan 2009, 12:33 p.m.

You are a lifesaver. Thanks for this.

# Typolleanyday 11 Feb 2009, 6:23 p.m.

Interesting and communicative, but would make something more on this topic?

# Joshua Jonah 14 Feb 2009, 8:18 a.m.

Thanx for this great little function. Just wanted to add a simple but necessary function to this:

def normalize_delimited_query(query_string, delimiter):
''' Splits the query string in invidual keywords, getting rid of unecessary spaces.
Example:

>>> normalize_delimited_query(' some,random , words', ',')
['some', 'random', 'words']

'''
return [t.strip() for t in query_string.split(delimiter)]

# Mohammad Rafiee 04 Mar 2009, 12:56 a.m.

hi dude !

today i faced a problem this function returns only the records with all key words occured in it not all records with keywords am i right?

i need a function now to return all records with such that keywords

can you help?
thank again

# ghiotion 21 Apr 2009, 11:22 a.m.

man. thank you so much for this. really brilliant.

# Jerry 08 May 2009, 9:20 a.m.

Nice, very elegant.

Plus, you just saved me hours. I totally appreciate it.

# greg sadetsky 18 May 2009, 5:39 p.m.

thank you for the simple and elegant code!

# simon Litchfield 23 Jun 2009, 12:13 p.m.

This looks great, I'm about to give it a whirl --
http://haystacksearch.org/

~blog~

Adding search to a Django site in a snap

Comments

Post a comment