<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[elijah.dev]]></title><description><![CDATA[elijah.dev]]></description><link>https://elijahilnero.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 08 Nov 2026 09:00:26 GMT</lastBuildDate><atom:link href="https://elijahilnero.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Demystifying Django ORM & PostgreSQL Indexing: Benchmarking B-Tree, Composite, and GIN Indexes on 50,000 Records]]></title><description><![CDATA[When building data-intensive applications with Django, the Object-Relational Mapper (ORM) abstracts raw SQL queries seamlessly. However, as tables grow…]]></description><link>https://elijahilnero.hashnode.dev/demystifying-django-orm-postgresql-indexing-benchmarking-b-tree-composite-and-gin-indexes-on-50-000-records</link><guid isPermaLink="true">https://elijahilnero.hashnode.dev/demystifying-django-orm-postgresql-indexing-benchmarking-b-tree-composite-and-gin-indexes-on-50-000-records</guid><dc:creator><![CDATA[Emmanuel Amujo]]></dc:creator><pubDate>Sun, 27 Sep 2026 19:50:30 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ab7277b757c25a2359abeb0/744c306b-5948-4598-8bab-29f1d3698301.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>When building data-intensive applications with Django, the Object-Relational Mapper (ORM) abstracts raw SQL queries seamlessly. However, as tables grow into tens or hundreds of thousands of rows, naive ORM queries can quietly turn into severe application bottlenecks.</p>
<p>To understand how PostgreSQL handles different index types under realistic loads, I built a benchmark suite comparing execution plans on a dataset of <strong>50,000 records per table (100,000 total rows)</strong>.</p>
<p>Using <strong>Django 6.1.1</strong>, <strong>PostgreSQL 18</strong>, and <code>EXPLAIN ANALYZE</code>, this deep dive explores:</p>
<ul>
<li><p>Single-column B-Tree indexes</p>
</li>
<li><p>Composite indexes</p>
</li>
<li><p><code>JSONB</code> GIN indexes</p>
</li>
<li><p>PostgreSQL execution plans</p>
</li>
<li><p>Query selectivity</p>
</li>
<li><p>A few counter-intuitive cases where adding an index can actually make a query slower</p>
</li>
</ul>
<hr />
<h2>1. Environment &amp; Database Schema</h2>
<p>The benchmark compares two identical tables:</p>
<ul>
<li><p><code>UnindexedOrder</code> — no database indexes beyond the defaults</p>
</li>
<li><p><code>IndexedOrder</code> — equipped with B-Tree, composite, and GIN indexes</p>
</li>
</ul>
<pre><code class="language-python"># orders/models.py

from django.db import models
from django.contrib.postgres.indexes import GinIndex


class UnindexedOrder(models.Model):
    customer_email = models.EmailField()
    status = models.CharField(max_length=20)
    created_at = models.DateTimeField(auto_now_add=True)
    metadata = models.JSONField(default=dict)


class IndexedOrder(models.Model):
    customer_email = models.EmailField()
    status = models.CharField(max_length=20)
    created_at = models.DateTimeField(auto_now_add=True)
    metadata = models.JSONField(default=dict)

    class Meta:
        indexes = [
            # Standard B-Tree index for single-column equality
            models.Index(
                fields=["customer_email"],
                name="idx_customer_email",
            ),

            # Composite index matching filter + ordering direction
            models.Index(
                fields=["status", "-created_at"],
                name="idx_status_created",
            ),

            # GIN index for JSONB metadata lookups
            GinIndex(
                fields=["metadata"],
                name="idx_order_metadata_gin",
            ),
        ]
</code></pre>
<p>Both models were populated with identical datasets containing <strong>50,000 records each</strong> using <code>bulk_create()</code> to mimic real transactional order history.</p>
<hr />
<h2>2. Benchmark Performance Breakdown</h2>
<table>
<thead>
<tr>
<th>Test Scenario</th>
<th>Query Filter</th>
<th>Unindexed Plan</th>
<th>Indexed Plan</th>
<th>Unindexed Time</th>
<th>Indexed Time</th>
<th>Performance Shift</th>
</tr>
</thead>
<tbody><tr>
<td><strong>B-Tree Index</strong></td>
<td>Single-column lookup</td>
<td><code>Seq Scan</code></td>
<td><code>Bitmap Index Scan</code></td>
<td><strong>81.73 ms</strong></td>
<td><strong>2.69 ms</strong></td>
<td><strong>~30× faster</strong></td>
</tr>
<tr>
<td><strong>Composite Index</strong></td>
<td>Filter + Sort</td>
<td><code>Seq Scan</code> + <code>Sort</code></td>
<td><code>Index Scan</code></td>
<td><strong>53.00 ms</strong></td>
<td><strong>0.76 ms</strong></td>
<td><strong>~70× faster</strong></td>
</tr>
<tr>
<td><strong>GIN (Low Selectivity)</strong></td>
<td>JSONB match (33.5% of dataset)</td>
<td><code>Seq Scan</code></td>
<td><code>Bitmap Index Scan</code></td>
<td><strong>79.36 ms</strong></td>
<td><strong>109.37 ms</strong></td>
<td><strong>~37% slower</strong></td>
</tr>
<tr>
<td><strong>GIN (High Selectivity)</strong></td>
<td>JSONB match (11.1% of dataset)</td>
<td><code>Seq Scan</code></td>
<td><code>Bitmap Index Scan</code></td>
<td><strong>132.93 ms</strong></td>
<td><strong>96.41 ms</strong></td>
<td><strong>~27% faster</strong></td>
</tr>
</tbody></table>
<blockquote>
<p><strong>Important:</strong> These results are specific to this benchmark environment and dataset. PostgreSQL's planner may choose a different execution strategy depending on table size, hardware, cache state, data distribution, statistics, and query shape.</p>
</blockquote>
<hr />
<h2>3. Execution Plan Analysis &amp; Key Findings</h2>
<h3>Benchmark 1: Single-Column B-Tree Index (~30× Speedup)</h3>
<p>Querying by email address on an unindexed table forces PostgreSQL to perform a <strong>Sequential Scan (</strong><code>Seq Scan</code><strong>)</strong>.</p>
<h4>Unindexed</h4>
<pre><code class="language-text">Seq Scan on orders_unindexedorder
  (cost=0.00..1470.00 rows=8 width=98)
  Filter: ((customer_email)::text = 'user_42@gmail.com'::text)
  Rows Removed by Filter: 49993
  Buffers: shared hit=845

Planning Time: 7.347 ms
Execution Time: 81.731 ms
</code></pre>
<p>The database had to examine all <strong>50,000 rows</strong> across <strong>845 shared buffers</strong> just to return 7 matching records.</p>
<p>With a standard B-Tree index on <code>customer_email</code>, PostgreSQL can use the index to locate candidate rows much more efficiently.</p>
<h4>Indexed</h4>
<pre><code class="language-text">Bitmap Heap Scan on orders_indexedorder
  (cost=4.35..34.12 rows=8 width=98)
  Recheck Cond: ((customer_email)::text = 'user_42@gmail.com'::text)
  Buffers: shared hit=9

  -&gt; Bitmap Index Scan on idx_customer_email
     (cost=0.00..4.35 rows=8 width=0)

Execution Time: 2.693 ms
</code></pre>
<p>Shared buffer hits dropped from <strong>845 to 9 blocks</strong>.</p>
<p>That's a dramatic reduction in the amount of table data PostgreSQL needed to touch. Conceptually, instead of walking through the entire table, PostgreSQL can use the B-Tree structure to locate matching values efficiently.</p>
<hr />
<h2>4. Benchmark 2: Composite Index &amp; Eliminating In-Memory Sorts (~70× Speedup)</h2>
<p>A common query pattern in web application dashboards involves filtering by status while sorting by creation timestamp:</p>
<pre><code class="language-python">IndexedOrder.objects.filter(
    status="completed"
).order_by("-created_at")[:10]
</code></pre>
<p>Without an appropriate index, PostgreSQL must:</p>
<ol>
<li><p>Scan the table</p>
</li>
<li><p>Find rows matching <code>status="completed"</code></p>
</li>
<li><p>Sort the matching rows by <code>created_at</code></p>
</li>
<li><p>Return the first 10 rows</p>
</li>
</ol>
<h4>Unindexed</h4>
<pre><code class="language-text">Limit
  (cost=1738.18..1738.20 rows=10 width=98)
  Buffers: shared hit=848

  -&gt; Sort
     (cost=1738.18..1769.20 rows=12410 width=98)
     Sort Key: created_at DESC
     Sort Method: top-N heapsort
     Memory: 28kB

     -&gt; Seq Scan on orders_unindexedorder

Execution Time: 53.004 ms
</code></pre>
<p>The important part here is <code>Sort Method: top-N heapsort</code>. PostgreSQL has to perform a sort because the table itself isn't organized in the order required by the query.</p>
<h4>Indexed</h4>
<p>The composite index is defined as:</p>
<pre><code class="language-python">models.Index(
    fields=["status", "-created_at"],
    name="idx_status_created",
)
</code></pre>
<p>The resulting plan is:</p>
<pre><code class="language-text">Limit
  (cost=0.41..3.53 rows=10 width=98)
  Buffers: shared hit=5

  -&gt; Index Scan using idx_status_created on orders_indexedorder

Execution Time: 0.759 ms
</code></pre>
<p>The composite index is ordered by <code>status</code> then <code>created_at DESC</code>. This matches the query's filtering and ordering requirements.</p>
<p>As a result, PostgreSQL can walk the relevant section of the index, retrieve the first 10 matching rows, and stop. There is no separate sort step. Buffer access also dropped from <strong>848 to 5 shared hits</strong>.</p>
<hr />
<h2>5. Benchmark 3: GIN Indexes, Django Syntax Traps &amp; the Selectivity Paradox</h2>
<p><code>JSONB</code> indexes behave differently from ordinary B-Tree indexes, and this is where things get particularly interesting.</p>
<h3>5.1 The Django ORM Syntax Trap</h3>
<p>Suppose we want to query the <code>metadata</code> JSON field:</p>
<pre><code class="language-python">IndexedOrder.objects.filter(
    metadata__plan="enterprise"
)
</code></pre>
<p>This looks perfectly reasonable from a Django perspective. However, the generated SQL uses JSON key extraction rather than the JSONB containment operator that a standard GIN index is designed to accelerate.</p>
<p>For a default PostgreSQL <code>jsonb_ops</code> GIN index, containment queries are a much better match:</p>
<pre><code class="language-python">IndexedOrder.objects.filter(
    metadata__contains={"plan": "enterprise"}
)
</code></pre>
<p>This generates a PostgreSQL JSONB containment condition using <code>@&gt;</code>, which can make use of the GIN index.</p>
<blockquote>
<p><strong>Key point:</strong> Creating a GIN index doesn't automatically mean every JSON lookup will use it. The operator used by the query matters.</p>
</blockquote>
<hr />
<h2>6. The Selectivity Paradox</h2>
<p>Here's where the benchmark produced a counter-intuitive result.</p>
<p>For this query:</p>
<pre><code class="language-python">IndexedOrder.objects.filter(
    metadata__contains={"plan": "enterprise"}
)
</code></pre>
<p>The condition matched <strong>16,752 out of 50,000 rows</strong> (~<strong>33.5% of the table</strong>).</p>
<p>The benchmark produced:</p>
<table>
<thead>
<tr>
<th>Plan</th>
<th>Execution Time</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Sequential Scan</strong></td>
<td><strong>79.36 ms</strong></td>
</tr>
<tr>
<td><strong>GIN Index</strong></td>
<td><strong>109.37 ms</strong></td>
</tr>
</tbody></table>
<p>The indexed query was approximately <strong>37% slower</strong>. So why would adding an index make the query slower?</p>
<h3>6.1 Indexes Aren't Always Faster</h3>
<p>An index has overhead. For a GIN-backed query, PostgreSQL may need to:</p>
<ol>
<li><p>Search the GIN index</p>
</li>
<li><p>Build a bitmap of matching row locations</p>
</li>
<li><p>Visit the corresponding heap pages</p>
</li>
<li><p>Recheck the condition against the actual rows</p>
</li>
</ol>
<p>When a query matches a large percentage of the table, this additional work can become more expensive than simply reading the table sequentially.</p>
<p>In this case, PostgreSQL determined that a sequential scan was cheaper for the unindexed query.</p>
<p>This is an important database principle:</p>
<blockquote>
<p><strong>An index is a tool for reducing unnecessary work—not a guarantee that less work will always happen.</strong></p>
</blockquote>
<hr />
<h2>7. Reversing the Curve with Higher Selectivity</h2>
<p>Now consider a more selective query:</p>
<pre><code class="language-python">IndexedOrder.objects.filter(
    metadata__contains={
        "plan": "enterprise",
        "region": "EU",
    }
)
</code></pre>
<p>The additional condition reduced the matching dataset from <strong>16,752 rows</strong> down to <strong>5,557 rows</strong> (~<strong>11.1% of the table</strong>).</p>
<p>The execution times changed significantly.</p>
<h4>Unindexed</h4>
<pre><code class="language-text">Seq Scan on orders_unindexedorder
  Filter: (
    metadata @&gt; '{"plan": "enterprise", "region": "EU"}'::jsonb
  )

Execution Time: 132.933 ms
</code></pre>
<h4>Indexed</h4>
<pre><code class="language-text">Bitmap Heap Scan on orders_indexedorder
  Recheck Cond: (
    metadata @&gt; '{"plan": "enterprise", "region": "EU"}'::jsonb
  )

Execution Time: 96.408 ms
</code></pre>
<p>Now the GIN index wins:</p>
<table>
<thead>
<tr>
<th>Plan</th>
<th>Execution Time</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Sequential Scan</strong></td>
<td><strong>132.93 ms</strong></td>
</tr>
<tr>
<td><strong>GIN Index</strong></td>
<td><strong>96.41 ms</strong></td>
</tr>
</tbody></table>
<p>That's approximately a <strong>27% improvement</strong>.</p>
<hr />
<h2>8. Why Did Selectivity Change the Result?</h2>
<p>The important variable is the percentage of rows that match the query.</p>
<h3>Low Selectivity</h3>
<p><code>50,000 total rows</code> -&gt; <code>16,752 matches</code> -&gt; <code>33.5%</code></p>
<p>The query is asking for a large portion of the table. In this situation, PostgreSQL may decide that scanning the table sequentially is cheaper than <code>GIN Index</code> -&gt; <code>Bitmap</code> -&gt; <code>Heap pages</code> -&gt; <code>Recheck rows</code>.</p>
<h3>Higher Selectivity</h3>
<p><code>50,000 total rows</code> -&gt; <code>5,557 matches</code> -&gt; <code>11.1%</code></p>
<p>Now the index can eliminate much more unnecessary table access. The cost of using the index is outweighed by the amount of data PostgreSQL avoids scanning.</p>
<p>This is why <strong>selectivity matters when designing and evaluating indexes</strong>.</p>
<hr />
<h2>9. Practical Takeaways for Django Developers</h2>
<h3>9.1 Always Verify with <code>.explain()</code></h3>
<p>Never assume an index is being used simply because you've created one. Django allows you to inspect the generated execution plan:</p>
<pre><code class="language-python">queryset = IndexedOrder.objects.filter(
    customer_email="user_42@gmail.com"
)

print(queryset.explain())
</code></pre>
<p>For deeper analysis, you can inspect PostgreSQL's execution plan using <code>EXPLAIN ANALYZE</code> where appropriate. Look for operations such as <code>Seq Scan</code>, <code>Index Scan</code>, <code>Bitmap Index Scan</code>, <code>Bitmap Heap Scan</code>, and <code>Sort</code>.</p>
<hr />
<h3>9.2 Column Order Matters in Composite Indexes</h3>
<p>For queries like:</p>
<pre><code class="language-python">Order.objects.filter(
    status="completed"
).order_by("-created_at")
</code></pre>
<p>an index such as:</p>
<pre><code class="language-python">models.Index(
    fields=["status", "-created_at"],
    name="idx_status_created",
)
</code></pre>
<p>can allow PostgreSQL to combine filtering and ordering into a single index traversal.</p>
<p>The order of fields in a composite index matters. Don't blindly create <code>["created_at", "status"]</code> when your dominant query pattern is <code>WHERE status = ... ORDER BY created_at DESC</code>.</p>
<hr />
<h3>9.3 Use Appropriate Operators for JSONB GIN Indexes</h3>
<p>If you're using a standard PostgreSQL GIN index on a <code>JSONB</code> column, containment queries are an important use case:</p>
<pre><code class="language-python">Order.objects.filter(
    metadata__contains={"plan": "enterprise"}
)
</code></pre>
<p>rather than assuming every JSON lookup will automatically benefit from the GIN index.</p>
<hr />
<h3>9.4 Beware of Low Selectivity</h3>
<p>An index isn't automatically beneficial just because a column is indexed. If a query matches a large percentage of the table, PostgreSQL may determine that a sequential scan is cheaper.</p>
<p>The <strong>10–20% figure should be treated as a rough intuition, not a hard PostgreSQL cutoff</strong>. The planner considers many factors, including table statistics, page layout, correlation, caching, and estimated costs.</p>
<hr />
<h2>10. The Bigger Lesson</h2>
<p>Django's ORM makes database interaction convenient, but the ORM doesn't eliminate the underlying database mechanics.</p>
<p>When your application grows, you need to understand what PostgreSQL is actually doing underneath your Django queries.</p>
<p>A query like <code>Order.objects.filter(status="completed").order_by("-created_at")[:10]</code> looks simple, but PostgreSQL has to decide between scanning/sorting vs traversing a pre-sorted composite index.</p>
<p>The important lesson isn't <em>"Always add an index."</em></p>
<p>It's: <strong>"Understand the query, understand the index, and verify the execution plan."</strong></p>
<hr />
<h2>11. Final Benchmark Summary</h2>
<ul>
<li><p><strong>B-Tree (Equality):</strong> <code>81.73 ms</code> -&gt; <code>2.69 ms</code> (<strong>~30× faster</strong>). Excellent fit for highly selective equality lookups such as email addresses.</p>
</li>
<li><p><strong>Composite B-Tree (Filter + Order):</strong> <code>53.00 ms</code> -&gt; <code>0.76 ms</code> (<strong>~70× faster</strong>). Supports filtering and ordering simultaneously, avoiding explicit sorts.</p>
</li>
<li><p><strong>GIN (Low Selectivity):</strong> <code>79.36 ms</code> -&gt; <code>109.37 ms</code> (<strong>~37% slower</strong>). The query matched too much of the table for index overhead to pay off.</p>
</li>
<li><p><strong>GIN (Higher Selectivity):</strong> <code>132.93 ms</code> -&gt; <code>96.41 ms</code> (<strong>~27% faster</strong>). The smaller result set made the index traversal worthwhile.</p>
</li>
</ul>
<hr />
<h2>Conclusion</h2>
<p>Optimizing database performance in Django requires looking past the abstraction layer and understanding how PostgreSQL interprets query execution plans. Indexes aren't magic performance switches. Their effectiveness depends on the <strong>query pattern, index structure, operator, data distribution, selectivity, and execution plan</strong>.</p>
<p>The most useful habit you can develop is simple:</p>
<pre><code class="language-python">queryset.explain()
</code></pre>
<p>Don't guess what PostgreSQL is doing. <strong>Measure it. Inspect the plan. Then optimize based on evidence.</strong></p>
<hr />
<h2>Benchmark Environment</h2>
<ul>
<li><p><strong>Django:</strong> 6.1.1</p>
</li>
<li><p><strong>PostgreSQL:</strong> 18</p>
</li>
<li><p><strong>Dataset:</strong> 50,000 records per table (100,000 total)</p>
</li>
<li><p><strong>Analysis:</strong> <code>EXPLAIN ANALYZE</code></p>
</li>
<li><p><strong>Index types:</strong> B-Tree, Composite B-Tree, GIN (<code>jsonb_ops</code>)</p>
</li>
<li><p><strong>Data generation:</strong> Django <code>bulk_create()</code></p>
</li>
</ul>
]]></content:encoded></item></channel></rss>