Think Build Implement Repeat
London, UK +44 7367 067226
WhatsApp FOLLOW f in X
  1. Home
  2. Blog
  3. Queries, Connections and Performance
Python & Django

Queries, Connections and Performance

Working with databases in Python services: connection pooling, parameterised queries, avoiding query multiplication, indexing, transactions and migrations.

Updated 2 min readBy SpiderHunts Technologies

Free estimateNo obligation

Get a free estimate

Tell us what you need. A senior engineer reads every enquiry.

Takes under a minute. We never share your details.

  • Free consultation
  • No commitment
  • NDA on request

Prefer to talk? Book a free 30-minute call →

Quick answer — TL;DR

Use connection pooling, parameterise every query, load relations explicitly rather than lazily, and index what you filter on. Those four prevent most database problems.

Four things that prevent most problems

  1. Connection pooling — opening a connection per request does not scale
  2. Parameterised queries, always, without exception
  3. Explicit relation loading rather than lazy loading in loops
  4. Indexes on the columns you filter, join and sort by
Lazy loading inside a loop is the most common performance problem in any framework with an ORM. One query becomes four hundred and nobody notices until the data grows.

Query multiplication is the usual cause

Fetching a list and then accessing a related object per item issues one query per row. On ten rows nobody notices; on a thousand the page takes twenty seconds.

  • Load relations explicitly with the parent query
  • Log query count per request during development
  • Set a threshold that fails a test if exceeded
  • Check the count on any page that handles lists

Transactions where they matter

Anything that writes several related records should do so in a transaction, so a partial failure does not leave inconsistent data.

Keep transactions short. A transaction held open while calling an external API blocks other work and can exhaust the connection pool.

Migrations need discipline

PracticeWhy
Every schema change as a migrationEnvironments stay in step
Backwards compatible where possibleCode can roll back independently
Tested on a production-sized copyLong-running migrations are discovered early
Reviewed before applyingSchema mistakes are expensive
Never edited after being appliedEnvironments diverge otherwise

Watch the growth

Queries that were fast at ten thousand rows can be slow at a million. Applications frequently slow down with no change in usage, purely because data accumulated.

Monitor query duration over time, and archive data that is no longer needed in the working set.

FAQ

Frequently asked questions

The questions readers ask us after this guide.

Still have a question?

Ask us directly — a senior engineer will get back to you.

Ask about your project

ORM or raw SQL?

An ORM for ordinary work, raw SQL for complex reporting queries. Both, used where each is stronger.

How do we find slow queries?

The database's slow query log plus per-request query logging in the application. Both are free and both point directly at the problem.

Should we use a read replica?

When reporting queries affect the application. That is usually the first genuine architectural step.

What about connection limits?

Pool size multiplied by process count must stay within the database's limit. Exceeding it under load is a common and confusing failure.

Keep reading

More on Python & Django

Python & Django

An API Other Systems Can Depend On

Designing a Python API service others can depend on: validation at the boundary, consistent errors and status codes, early versioning and documentation.

Python & Django

Moving and Transforming Data Reliably

Building data pipelines in Python that cope with malformed input: restartable stages, quarantining failures, reconciling counts and alerting on absence.

Start here

Application slowing as data grows?

Count the queries on your slowest page first. It is usually the whole answer.

  1. You tell us what you needTwo minutes on the form, or a message on WhatsApp.
  2. A senior engineer reviews itAnd comes back with questions, a realistic range and an honest view on fit.
  3. Free 30-minute scoping callWe talk through scope, options and a realistic estimate — with no obligation.
Free estimateNo obligation

Talk to someone who builds this

Send a short brief and we will come back with an honest view and a realistic range.

Takes under a minute. We never share your details.

  • Free consultation
  • No commitment
  • NDA on request

Prefer to talk? Book a free 30-minute call →