Django: introducing django-crawl

Django: introducing django-crawl

Articles

Title: Django: introducing django-crawl - Adam Johnson

URL Source: https://adamj.eu/tech/2026/07/22/introducing-django-crawl/

Published Time: Thu, 23 Jul 2026 09:16:42 GMT

Markdown Content: 2026-07-22 I recently migrated one of my client projects from the legacy django-csp package to Django 6.0’s built-in Content Security Policy (CSP) support (release note). This security header is a powerful tool for preventing unwanted content from being loaded on your site, so configuration correctness is paramount. The migration was fairly straightforward, but a few pages had complicated overrides, so I wanted to be sure that no CSP headers had been changed by my swapping of CSP implementations.

I had the idea to verify no page had changed its content-security-policy header by crawling the site with Django’s test client, outputting URL and header contents during the process. By diffing the output from crawls before and after the migration, I could check for changes and track down which pages had been affected.

The core loop of that script looked something like this:

from collections import deque from django.test import Client

client = Client() client.force_login(superuser)

queue: deque[str] = deque(["/", "/admin/"]) ...

while queue: url = queue.popleft() ... response = client.get(url, follow=False) ... print(f"{url}\t{response.headers.get('content-security-policy')}") ... for anchor in BeautifulSoup(response.content, "html.parser").find_all( "a", href=True ): # Enqueue these found links ...

This simple crawl of the site ended up flushing out seven non-CSP bugs, despite the project having 100% test coverage and a full suite of integration tests. Those bugs were due to incorrect link generation, admin features not being disabled, and regular old broken code.

I was pretty impressed with the power of this technique for finding broken stuff! Given this experience, I wanted to expand the script into a reusable tool, which I have now done with django-crawl.

To use django-crawl, install it, add it to INSTALLED_APPS, and you can run the crawl management command:

$ ./manage.py crawl -v 2 🐛 Crawling up to 1000 URLs, logged in as Ad Min / /about/ /blog/ /contact/ /dev/ /blog/2026/ /blog/2025/ /blog/2026/07/22/introducing-django-crawl/ ... 🦋 Crawled 1000 URLs, encountered 0 errors, stopped due to reaching max URL limit of 1000.

The command reports any errors it encounters, from broken links to exceptions. The output uses Rich for pretty formatting and a live spinner while it runs. Adding -v 2 prints the URLs as they are crawled.

The crawler discovers links in various forms in HTML responses (<a href>, <link href>, <script src>, <img src>, etc.), sitemaps, and feeds. If you have a sitemap, you can start your crawl with a simple:

$ ./manage.py crawl /sitemap.xml

The HTML parsing is done with a custom Rust extension built with html5ever, the HTML parser from the Servo project, so it’s very fast. And with no overhead from real HTTP requests or inter-process communication, the crawl is limited only by how fast your application code can run.

django-crawl also provides a Python API that you can use to make a mega-test that crawls your whole site with example data, raising an ExceptionGroup if any errors are encountered:

from django.contrib.auth.models import User from django.test import TestCase

import django_crawl

class CrawlTests(TestCase): @classmethod def setUpTestData(cls): cls.admin = User.objects.create_superuser(username="admin")

def test_crawl(self):
    client = django_crawl.CrawlClient()
    client.force_login(self.admin)
    django_crawl.crawl("/", "/admin/", client=client)

I am not sure if this is a good fit for most projects, since it will leave you with one long test that exercises many views. But it might be good for building a “safety net” on untested projects, something I know Jeff Triplett likes to do (ref Django Chat #24).

Fin

Please try out django-crawl today and let me know how it goes.

May your site not slow to a crawl,

—Adam

😸😸😸 Check out my new book on using GitHub effectively, Boost Your GitHub DX! 😸😸😸

One summary email a week, no spam, I pinky promise.

Related posts:

Tags:django