Composite Index Column Order — The “equality first, range later” and leftmost-prefix principles
Composite indexes are effective for queries that filter on multiple columns, but performance depends on the column order. Because a B-tree is sorted from the leftmost column onward, putting equality-condition columns (=) first and range-condition columns (>= / <) later keeps the search in one contiguous range. In the reverse order, the database reads a broad range first and then filters on the equality condition, resulting in an inefficient scan.
-- Query: status is an equality condition; created_at is a range WHERE status = 'completed' AND created_at >= '2024-04-01' AND created_at < '2024-07-01' -- ✗ INDEX (created_at, status): range first → cannot narrow by status -- ✓ INDEX (status, created_at): equality first → scans only one contiguous range CREATE INDEX idx_orders_status_created ON orders (status, created_at);
(a, b, c) can be used when conditions appear continuously from the beginning (a / a,b / a,b,c). It is generally of little use for conditions on only b or c. Also, columns after the one reached by a range condition cannot be used for “narrowing”; they are treated as filters. That is why “equality → range” is the rule of thumb.From the orders table, retrieve orders with status = completed and a creation date 2024 (April–June). Also write the optimal composite-index CREATE INDEX statement for this query. Return order_id, created_at, amount, ordered by created_at ascending.
| order_id | status | created_at | amount |
|---|---|---|---|
| 1 | completed | 2024-01-10 | 1200 |
| 2 | cancelled | 2024-04-02 | 800 |
| 3 | completed | 2024-04-15 | 3000 |
| 4 | completed | 2024-05-20 | 2500 |
| 5 | pending | 2024-05-25 | 900 |
| 6 | completed | 2024-06-30 | 4000 |
| 7 | completed | 2024-08-01 | 1100 |
| 8 | cancelled | 2024-06-10 | 600 |
| order_id | created_at | amount |
|---|---|---|
| 3 | 2024-04-15 | 3000 |
| 4 | 2024-05-20 | 2500 |
| 6 | 2024-06-30 | 4000 |
CREATE INDEX idx_orders_status_created -- Optimal composite index: equality column (status) first, range column (created_at) later ON orders (status, created_at); SELECT order_id, created_at, amount FROM orders WHERE status = 'completed' -- Equality condition: pinpoint the leading index column AND created_at >= '2024-04-01' -- Range condition: scan a contiguous range in the second column (half-open interval) AND created_at < '2024-07-01' ORDER BY created_at; -- Matches index order, so no sort is needed /* Logical execution order: 1. Binary search the B-tree by (status, created_at) → reach the composite key 2. Follow the leaves to the right in a contiguous scan → scan only around the range 3. Fetch columns from the table → retrieve amount and other columns 4. ORDER BY created_at → no Sort node Plan comparison (conceptual): ✗ INDEX (created_at, status) → read all rows in April–June (id=2,3,4,5,6,8), then filter status ✓ INDEX (status, created_at) → read only April–June rows among completed orders in one range */
LEGEND
1. FROM orders — 8 rows (target of composite-index design)
FROM ordersFilter with the two conditions status (equality) and created_at (range). Even for a query returning the same 3 rows, the index column order changes the amount of scanning completely; that is the theme of this question.| order_id | status | created_at | amount |
|---|---|---|---|
| 1 | completed | 2024-01-10 | 1200 |
| 2 | cancelled | 2024-04-02 | 800 |
| 3 | completed | 2024-04-15 | 3000 |
| 4 | completed | 2024-05-20 | 2500 |
| 5 | pending | 2024-05-25 | 900 |
| 6 | completed | 2024-06-30 | 4000 |
| 7 | completed | 2024-08-01 | 1100 |
| 8 | cancelled | 2024-06-10 | 600 |
INDEX (status, created_at) is ordered by created_at. Since WHERE status=... ORDER BY created_at can return that order directly, the Sort node disappears from EXPLAIN. Designing the filtering and ordering columns as a set is the advanced perspective.(status, created_at) exists, it also serves queries with only WHERE status = ... (using just the leading column). Conversely, it is generally of little use for WHERE created_at = ... alone. Check whether an existing index can cover the query through its leftmost prefix before adding another index.INDEX(status) and INDEX(created_at) are created separately, a composite query may use only one, or combine them with BitmapAnd and become slower than one composite index. Also, indexes are all updated on every write (INSERT/UPDATE), so having too many directly harms write performance.INDEX(status), where there are only a few possible values, often has too high a hit rate per value and is ignored by the planner. Use a low-cardinality column as the equality “anchor” at the start of a composite index instead of relying on it alone.pg_stat_statements, (2) classify conditions as “equality / range / ordering”, (3) order columns as equality → range → (columns to cover with INCLUDE), and (4) compare before and after with EXPLAIN ANALYZE. If an existing index’s leftmost prefix is enough, do not add another; find unused indexes with pg_stat_user_indexes and remove them — designing what not to add is as important as designing what to add.Keyset Pagination — Cursor-based pagination that stays fast on deep pages
As discussed in the basic set, LIMIT 20 OFFSET 1000000 reads and discards one million rows before returning 20, so it gets linearly slower on deeper pages. The advanced solution is Keyset Pagination (cursor-based pagination): remember the key of the last row on the previous page and use a WHERE clause to specify the rows after that position directly. Every page can be retrieved with a single Index Scan, keeping the cost constant.
-- ✗ OFFSET: discarded reads increase with page depth O(offset) SELECT ... ORDER BY created_at DESC, order_id DESC LIMIT 3 OFFSET 3; -- ✓ Keyset: continue directly from the previous page’s last key O(limit) WHERE (created_at, order_id) < ('2024-06-05', 8) ORDER BY created_at DESC, order_id DESC LIMIT 3;
(created_at, order_id) < (value1, value2) expresses “created_at is smaller, or created_at is equal and order_id is smaller” in one row-value (tuple) comparison, and composite index (created_at DESC, order_id DESC) can be used directly.Retrieve the second page of a feed ordered by newest first (created_at DESC, order_id DESC, 3 rows per page). The last row on page 1 is (created_at, order_id) = ('2024-06-05', 8). Do not use OFFSET; write it with Keyset Pagination using tuple comparison. Return order_id, created_at, amount.
| order_id | created_at | amount |
|---|---|---|
| 10 | 2024-06-30 | 2200 |
| 9 | 2024-06-21 | 1800 |
| 8 | 2024-06-05 | 3500 |
| 7 | 2024-06-05 | 1500 |
| 6 | 2024-05-18 | 900 |
| 5 | 2024-05-02 | 4100 |
| 4 | 2024-04-11 | 700 |
| 3 | 2024-03-22 | 2600 |
| 2 | 2024-02-14 | 1300 |
| 1 | 2024-01-05 | 500 |
| order_id | created_at | amount |
|---|---|---|
| 7 | 2024-06-05 | 1500 |
| 6 | 2024-05-18 | 900 |
| 5 | 2024-05-02 | 4100 |
SELECT order_id, created_at, amount FROM orders WHERE (created_at, order_id) < ('2024-06-05', 8) -- [Cursor] after the previous page’s last key in sort order (tuple comparison) ORDER BY created_at DESC, order_id DESC -- Include the unique key to fully fix the order LIMIT 3; -- Fetch only one page and stop early /* Logical execution order: 1. Binary-search the index to the position after the cursor → no discarded OFFSET rows 2. Read only 3 rows in index order and stop → LIMIT 3 3. Return the SELECT columns → return the requested columns Tuple-comparison expansion (equivalent condition): (created_at, order_id) < ('2024-06-05', 8) ⇔ created_at < '2024-06-05' OR (created_at = '2024-06-05' AND order_id < 8) Plan comparison (conceptual): ✗ LIMIT 3 OFFSET 3 … read the first 6 rows and discard 3 (worse on deeper pages) ✓ Keyset … read only 3 rows from the cursor position (always O(limit)) */
LEGEND
1. FROM orders — entire feed (sorted index)
FROM ordersOn INDEX (created_at DESC, order_id DESC), rows are physically ordered as in this table. Page 1 is the first 3 rows (id=10, 9, 8). Notice that id=8 and 7 share the same created_at.| order_id | created_at | amount |
|---|---|---|
| 10 | 2024-06-30 | 2200 |
| 9 | 2024-06-21 | 1800 |
| 8 | 2024-06-05 | 3500 |
| 7 | 2024-06-05 | 1500 |
| 6 | 2024-05-18 | 900 |
| 5 | 2024-05-02 | 4100 |
| 4 | 2024-04-11 | 700 |
| 3 | 2024-03-22 | 2600 |
| 2 | 2024-02-14 | 1300 |
| 1 | 2024-01-05 | 500 |
ORDER BY created_at DESC, order_id DESC and (created_at, order_id) < (...) as a pair to make the order completely unique.(a, b) < (x, y) is equivalent to a < x OR (a = x AND b < y), but the tuple form is easier for the planner to treat as one composite-index range (PostgreSQL directly supports row-value comparison). Some databases produce a worse plan for the OR expansion, so use tuple form when supported.SELECT COUNT(*) FROM huge_table on every request for a pagination UI can make the count more expensive than the main query. Infinite scroll does not need a total; when a count is necessary, estimates such as pg_class.reltuples or a cache are practical alternatives.WHERE created_at < 'the previous page’s last timestamp' is a source of bugs because it skips or duplicates rows sharing that timestamp. This happens especially often when batch ingestion records created_at only to the second. Build the cursor as a tuple including the primary key.Top-N per Group with Window Functions — Get the top 2 in each category in one pass with ROW_NUMBER
The basic Top-N question selected the top K rows overall, but in real work the much more common case is “top K rows in each group” (Top-N per Group). A correlated subquery tends to repeat a scan once per group, whereas the window function ROW_NUMBER() needs only one table pass plus sorting within each partition. Window functions cannot be used directly in WHERE, so the standard pattern is to assign ranks in a CTE (or subquery) and filter in the outer query.
-- Standard pattern: rank in a CTE → filter rn <= K outside WITH ranked AS ( SELECT *, ROW_NUMBER() OVER ( PARTITION BY category ORDER BY amount DESC) AS rn FROM orders) SELECT ... WHERE rn <= 2;
ROW_NUMBER is always sequential (1,2,3... even for ties), RANK gives ties the same rank and skips the next rank (1,1,3), while DENSE_RANK gives ties the same rank without a gap (1,1,2). Choose ROW_NUMBER + a unique tie-breaker when you need exactly K rows, and RANK when you want all tied rows.From the orders table, retrieve the top 2 orders by sales amount in each category. When amounts tie, prefer the smaller order_id. Return category, order_id, amount, rn, ordered by category ascending and then rn ascending.
| order_id | category | amount |
|---|---|---|
| 1 | Books | 1200 |
| 2 | Books | 800 |
| 3 | Books | 2000 |
| 4 | Toys | 5000 |
| 5 | Toys | 3000 |
| 6 | Toys | 4500 |
| 7 | Games | 2500 |
| 8 | Games | 2500 |
| 9 | Games | 900 |
| category | order_id | amount | rn |
|---|---|---|---|
| Books | 3 | 2000 | 1 |
| Books | 1 | 1200 | 2 |
| Games | 7 | 2500 | 1 |
| Games | 8 | 2500 | 2 |
| Toys | 4 | 5000 | 1 |
| Toys | 6 | 4500 | 2 |
WITH ranked AS ( SELECT category, order_id, amount, ROW_NUMBER() OVER ( PARTITION BY category -- Create an independent numbering space for each category ORDER BY amount DESC, order_id -- Amount descending + unique key makes the ranking deterministic ) AS rn FROM orders ) SELECT category, order_id, amount, rn FROM ranked WHERE rn <= 2 -- Window functions cannot be used directly in WHERE; filter outside ORDER BY category, rn; /* Logical execution order: 1. FROM orders → read 9 rows in one pass 2. PARTITION BY category → divide into category partitions 3. ORDER BY amount DESC, order_id within each partition → sort within partitions 4. ROW_NUMBER() assigns sequential numbers → assign rn 5. Outer WHERE rn filters the top rows → only the top 2 rows in each partition pass 6. ORDER BY category, rn → final ordering Tie-breaker effect (Games): id=7 (2500) and id=8 (2500) have the same amount → ORDER BY amount DESC, order_id fixes id=7 at rn=1 and id=8 at rn=2 */
LEGEND
1. FROM orders — read 9 rows in one pass
FROM ordersThere are 3 categories × 3 rows each. A correlated subquery would repeat an inner scan for the 9 outer rows, but the window function reads every row only once.| order_id | category | amount |
|---|---|---|
| 1 | Books | 1200 |
| 2 | Books | 800 |
| 3 | Books | 2000 |
| 4 | Toys | 5000 |
| 5 | Toys | 3000 |
| 6 | Toys | 4500 |
| 7 | Games | 2500 |
| 8 | Games | 2500 |
| 9 | Games | 900 |
WHERE → SELECT (window evaluation), so writing WHERE rn <= 2 at the same level causes an error. The two-layer structure of fixing rn in a CTE or subquery and then filtering outside is the established Top-N per Group pattern.ORDER BY amount DESC, order_id.WHERE amount >= (SELECT ... ORDER BY amount DESC LIMIT 1 OFFSET 1) evaluates the subquery repeatedly for each outer row. It collapses as group and row counts grow, so replace it with a window function (or, in PostgreSQL, LATERAL + LIMIT).RANK() <= 2 can return 3 or more rows when there are many ties, despite the expectation of 2. Passing that directly into a UI with room for only 2 rows causes layout problems and pagination mismatches. Use ROW_NUMBER when the count must be guaranteed and RANK when all ties are required.INDEX (category, amount DESC, order_id), data is already ordered by “partition → rank within partition”, allowing an index scan to take over the sort. If there are few partitions and each is huge, PostgreSQL can sometimes win with LATERAL (SELECT ... LIMIT 2), which stops Top-N processing per partition. The advanced practice is to compare both approaches with EXPLAIN rather than treating window functions as the only option.Pre-Join Aggregation — Prevent row fan-out and double counting by aggregating before JOIN
When a one-to-many table is joined, the parent row fans out once per child row. Aggregating after that expansion not only increases the amount of work; joining multiple child tables at once also creates a double-counting bug. The advanced standard pattern is to aggregate the child table in a CTE first, then JOIN the smaller result to the parent. To retain parents with zero children, combine LEFT JOIN with COALESCE.
-- ✗ Aggregate after joining: fold only after rows have fanned out (heavy and error-prone) SELECT c.name, COUNT(o.order_id), SUM(o.amount) FROM customers c LEFT JOIN orders o ON ... GROUP BY c.name; -- ✓ Aggregate before joining: fold the child first, then JOIN the small result WITH agg AS (SELECT customer_id, COUNT(*) cnt, SUM(amount) total FROM orders GROUP BY customer_id) SELECT ... FROM customers c LEFT JOIN agg ON ...;
From customers and orders, retrieve the order count and total amount for every customer. Aggregate orders in a CTE before joining, and output customers with no orders (Bob) as 0 / 0. Return customer_id, name, order_count, total_amount, ordered by customer_id ascending.
| customer_id | name |
|---|---|
| 101 | Alice |
| 102 | Bob |
| 103 | Carol |
| order_id | customer_id | amount |
|---|---|---|
| 1 | 101 | 1200 |
| 2 | 101 | 800 |
| 3 | 103 | 5000 |
| 4 | 103 | 3000 |
| 5 | 103 | 500 |
| 6 | 101 | 2500 |
| customer_id | name | order_count | total_amount |
|---|---|---|---|
| 101 | Alice | 3 | 4500 |
| 102 | Bob | 0 | 0 |
| 103 | Carol | 3 | 8500 |
WITH order_agg AS ( SELECT customer_id, COUNT(*) AS order_count, -- Fix the child-table count before the JOIN SUM(amount) AS total_amount -- Compute the total in the same pass FROM orders GROUP BY customer_id -- Fold 6 rows into 2 before joining ) SELECT c.customer_id, c.name, COALESCE(a.order_count, 0) AS order_count, -- Customer with no orders: NULL → 0 COALESCE(a.total_amount, 0) AS total_amount FROM customers c LEFT JOIN order_agg a -- LEFT JOIN keeps every customer (does not drop Bob) ON a.customer_id = c.customer_id ORDER BY c.customer_id; /* Logical execution order: 1. CTE order_agg → aggregate by customer_id 2. FROM customers LEFT JOIN order_agg → join the aggregate (unmatched side is NULL) 3. SELECT applies COALESCE → convert NULL to 0 4. ORDER BY customer_id → order the rows Plan comparison (conceptual): ✗ JOIN then GROUP BY → the join intermediate result fans out (all orders participate) ✓ Aggregate then JOIN → only the “customer count” rows participate in the join */
LEGEND
1. FROM orders — child table to aggregate (6 rows)
WITH order_agg AS (... FROM orders ...)Fold the “many” side, orders, first. Grouping by customer_id shows 3 rows for 101, 3 rows for 103, and 0 rows for 102 (Bob).| order_id | customer_id | amount |
|---|---|---|
| 1 | 101 | 1200 |
| 2 | 101 | 800 |
| 6 | 101 | 2500 |
| 3 | 103 | 5000 |
| 4 | 103 | 3000 |
| 5 | 103 | 500 |
COUNT(*) after a JOIN turns a zero-order customer such as Bob into “1 order” because a NULL row is still one row. COUNT(o.order_id) also avoids it, but pre-aggregation + COALESCE makes the counting mistake structurally impossible.GROUP BY customer_id and the outer ON a.customer_id = c.customer_id use the same key, one orders(customer_id) index can help avoid aggregation sorting and support JOIN lookup. Aligning key design tightens the entire plan.customers ⨝ orders ⨝ payments and then run SUM(o.amount), each order row is duplicated once per payment and the classic over-counting bug occurs. Many “why did the number suddenly double?” incidents come from this. Aggregate each child table independently before joining.SELECT c.name, (SELECT COUNT(*) FROM orders o WHERE o.customer_id = c.customer_id) ... is readable, the subquery executes repeatedly for every customer row. It may be acceptable with few customers, but list queries should use pre-aggregation + LEFT JOIN as the standard pattern.CREATE MATERIALIZED VIEW customer_sales AS WITH order_agg AS (...) SELECT ... and run REFRESH MATERIALIZED VIEW CONCURRENTLY in a nightly batch or every few minutes; the read side then only reads a small ordinary table. The trade-off is freshness. If real-time data is essential, use an aggregate table with triggers or incremental updates; if a delay of a few minutes is acceptable, use a materialized view. Choose from the freshness requirement rather than from SQL alone.Recursive CTE Performance Management — Safely and quickly traverse hierarchical data with a depth limit
Use WITH RECURSIVE to follow parent-child relationships to arbitrary depths in category hierarchies, organization charts, bills of materials (BOMs), and similar data. The process is: the non-recursive term (seed) creates the initial rows, the recursive term repeatedly runs against the immediately preceding result (the working table), and stops when no new rows appear. Performance management has two key points: use a depth limit (depth guard) to contain tree explosion, and prevent infinite loops with cycle protection.
WITH RECURSIVE tree AS ( -- Non-recursive term (seed): the one starting row SELECT id, name, parent_id, 0 AS depth FROM categories WHERE id = 1 UNION ALL -- Recursive term: find only the children of rows t from the immediately preceding step SELECT c.id, c.name, c.parent_id, t.depth + 1 FROM categories c JOIN tree t ON c.parent_id = t.id WHERE t.depth < 2 -- Depth-limit guard ) SELECT * FROM tree;
parent_id is the key support for the JOIN in every generation.From the categories table, retrieve all categories beneath id=1 (Electronics) with their depth. For safety, add a depth guard of depth < 2 (maximum depth 2). Return id, name, depth, ordered by depth ascending and then id ascending.
| id | name | parent_id |
|---|---|---|
| 1 | Electronics | NULL |
| 2 | Computers | 1 |
| 3 | Laptops | 2 |
| 4 | Audio | 1 |
| 5 | Headphones | 4 |
| 6 | Books | NULL |
| id | name | depth |
|---|---|---|
| 1 | Electronics | 0 |
| 2 | Computers | 1 |
| 4 | Audio | 1 |
| 3 | Laptops | 2 |
| 5 | Headphones | 2 |
WITH RECURSIVE tree AS ( SELECT id, name, parent_id, 0 AS depth -- [Seed] Starting Electronics (depth=0) FROM categories WHERE id = 1 UNION ALL -- No duplicate removal is needed, so UNION ALL is lighter than UNION SELECT c.id, c.name, c.parent_id, t.depth + 1 FROM categories c JOIN tree t ON c.parent_id = t.id -- Explore only children of the immediately preceding generation (needs parent_id index) WHERE t.depth < 2 -- [Depth limit] Safety valve for runaway queries and bad data ) SELECT id, name, depth FROM tree ORDER BY depth, id; /* Logical execution order (generation by generation): 1. Run the seed → place the starting row in the working table 2. First recursion → find and add children through parent_id 3. Second recursion → find and add the next children 4. Third recursion → stop at the guard (0 new rows) 5. Outer SELECT → order the accumulated result by depth and id */
LEGEND
1. Seed — place the starting row with the non-recursive term (depth=0)
SELECT ... WHERE id = 1 (non-recursive term)The starting point of the recursion. The one Electronics row is both the “accumulated result” and the parent (working table) for the next search. The rows with parent_id = 1 are previewed as the next generated rows with dotted styling.| id | name | depth | Status |
|---|---|---|---|
| 1 | Electronics | 0 | Added now (seed) |
| 2 | Computers | 1 (planned) | Generated next |
| 4 | Audio | 1 (planned) | Generated next |
A→B→A). One line, WHERE t.depth < N, protects the database from runaway queries when data is abnormal. PostgreSQL 14+ also supports declarative cycle detection with CYCLE id SET is_cycle USING path.UNION ALL rather than UNION (which removes duplicates): tree structures do not produce duplicates, so UNION ALL avoids the hash de-duplication cost in every generation. The recursive term’s c.parent_id = t.id runs once per generation, so without a categories(parent_id) index the worst case is number of generations × Seq Scan.statement_timeout — preferably several — before deploying to production.