NOT IN (subquery) has two traps when you need rows that do not exist elsewhere. The correctness trap comes first: if the subquery returns even one NULL, SQL's three-valued logic makes x NOT IN (...) UNKNOWN for every candidate row, producing an empty result. The performance trap follows: because the planner must preserve that NULL-sensitive behavior, it has less freedom to choose an efficient Anti Join plan.
-- ✗ A NULL in the subquery empties the result and can produce an inefficient plan WHERE member_id NOT IN (SELECT member_id FROM event_entries) -- ✓ NULL-safe and optimizable as a Hash Anti Join WHERE NOT EXISTS (SELECT 1 FROM event_entries e WHERE e.member_id = m.member_id)
x NOT IN (2, 4, NULL) expands to x <> 2 AND x <> 4 AND x <> NULL. The last comparison is always UNKNOWN, so the conjunction can never be true and no row passes WHERE. This is a classic cause of a query that “suddenly returns nothing” after new data arrives.The members table contains 100,000 rows in production, while event_entries contains 3 million. Its nullable member_id represents guest attendance. Return the members who have never attended an event in a form that is safe from NULL and can be optimized as an Anti Join. Return member_id, name in ascending member_id order.
| member_id | name |
|---|---|
| 1 | Aoki |
| 2 | Baba |
| 3 | Chiba |
| 4 | Doi |
| 5 | Endo |
| 6 | Fujii |
| entry_id | member_id |
|---|---|
| 1 | 2 |
| 2 | 4 |
| 3 | NULL |
| 4 | 2 |
entry_id=3 is a guest entry whose member_id is NULL. Also consider how this single row breaks NOT IN.
Expected output (member_id ascending):
| member_id | name |
|---|---|
| 1 | Aoki |
| 3 | Chiba |
| 5 | Endo |
| 6 | Fujii |
NOT IN returns zero rows because of the NULL entry. NOT EXISTS returns the correct four rows with a Hash Anti Join.
SELECT m.member_id, m.name FROM members m WHERE NOT EXISTS ( -- directly express that no matching row exists SELECT 1 FROM event_entries e WHERE e.member_id = m.member_id -- NULL never matches with =, so it is harmless ) ORDER BY m.member_id; /* Execution order: 1. Build a hash table from event_entries member_id values 2. Scan members and probe the hash table → Hash Anti Join 3. Pass only rows with no match → WHERE 4. Sort by member_id ascending → return the result */
LEGEND
1. Target table — members (outer side)
FROM members mThis is the population in which we look for nonparticipants: 100,000 rows in production. The ideal plan reads the other side once to build a lookup structure instead of rerunning the subquery for every member.| member_id | name |
|---|---|
| 1 | Aoki |
| 2 | Baba |
| 3 | Chiba |
| 4 | Doi |
| 5 | Endo |
| 6 | Fujii |
LEFT JOIN event_entries e ON … WHERE e.entry_id IS NULL can produce the same answer and a similar Anti Join plan. However, duplicate join keys can multiply rows before a later DISTINCT, so NOT EXISTS is the clearest and safest first choice. Use the LEFT JOIN form when you also need columns from the other side.WHERE member_id IS NOT NULL inside the subquery.NOT IN (1, 2, 3, …) makes parsing and planning expensive. If the exclusions are stored in a table, match them inside the database with a subquery or join. For application-provided values, consider a temporary table or NOT EXISTS over a VALUES relation rather than negating = ANY(array).NULL = NULL and NULL <> 1 evaluate to UNKNOWN. WHERE passes only true rows, so UNKNOWN is filtered just like false. The same three-valued logic explains why col <> 'x' omits NULL rows and why COUNT(col) ignores NULL. Whenever you write a negation, aggregate, or comparison, spend a second asking what NULL would do. That habit prevents a broad class of production failures. Q7 next reduces repeated aggregation scans to one pass.Suppose one report needs the completed count and sales total, the pending count, and the canceled count from the same table. Separate subqueries or queries scan the table once per aggregate. Conditional aggregation instead reads the table once and routes each row to the appropriate counters as it goes.
-- ✗ One full scan per aggregate (5 million rows × 4) SELECT (SELECT COUNT(*) FROM orders WHERE status = 'completed'), (SELECT SUM(amount) FROM orders WHERE status = 'completed'), ... -- ✓ Every aggregate in one scan with PostgreSQL's FILTER clause SELECT COUNT(*) FILTER (WHERE status = 'completed'), ... FROM orders;
SUM(CASE WHEN status = 'completed' THEN amount END) and COUNT(CASE WHEN … THEN 1 END). Omitting ELSE makes unmatched rows NULL, which SUM and COUNT ignore.From the 5-million-row production orders table, return these four values in one row with only one scan: completed_cnt, completed_amt, pending_cnt, and cancelled_cnt.
| order_id | status | amount |
|---|---|---|
| 1 | completed | 1200 |
| 2 | pending | 800 |
| 3 | completed | 2000 |
| 4 | cancelled | 500 |
| 5 | completed | 700 |
| 6 | pending | 1500 |
Imagine replacing four KPI queries on a dashboard with a single query.
Expected output (one row):
| completed_cnt | completed_amt | pending_cnt | cancelled_cnt |
|---|---|---|---|
| 3 | 3900 | 2 | 1 |
Four subqueries scan the equivalent of 5 million × 4 = 20 million rows. Conditional aggregation scans 5 million rows once.
SELECT COUNT(*) FILTER (WHERE status = 'completed') AS completed_cnt, SUM(amount) FILTER (WHERE status = 'completed') AS completed_amt, COUNT(*) FILTER (WHERE status = 'pending') AS pending_cnt, COUNT(*) FILTER (WHERE status = 'cancelled') AS cancelled_cnt FROM orders; -- this is the only table scan /* Execution order: 1. Scan orders from beginning to end once (Seq Scan) 2. Evaluate every aggregate's FILTER condition for each row 3. Update only the counters or sum whose conditions match 4. Return all four aggregate values as one row after the scan finishes */
LEGEND
1. Target table — orders
FROM ordersThe status column divides the data into three groups, shown by the row colors. The performance question is how many times these six sample rows—or 5 million production rows—must be read.| order_id | status | amount |
|---|---|---|
| 1 | completed | 1200 |
| 2 | pending | 800 |
| 3 | completed | 2000 |
| 4 | cancelled | 500 |
| 5 | completed | 700 |
| 6 | pending | 1500 |
COUNT(*) FILTER (WHERE c) equals COUNT(CASE WHEN c THEN 1 END), and SUM(x) FILTER (WHERE c) equals SUM(CASE WHEN c THEN x END). FILTER states the intent more clearly and PostgreSQL permits it on any aggregate, including AVG, MIN, MAX, and ARRAY_AGG. Use CASE when targeting a database without FILTER support.GROUP BY region to build a region-by-status cross-tab in one scan. This is common when producing BI summaries or daily KPI batches, and Q10 uses the combination directly.GROUP BY status query returns every group for the application to arrange.SELECT 'completed', COUNT(*) … UNION SELECT 'pending', COUNT(*) … still scans once per aggregate. Without ALL it also adds needless duplicate elimination, as Q9 explains. Use GROUP BY for vertical rows and FILTER for horizontal columns.ORDER BY may look like a small final step, but it is a blocking operation that cannot begin producing ordered output until it has received the necessary input. Large sorts can exceed work_mem and fall back to an expensive disk-based external merge. A B-tree index, however, is always ordered. When its columns and order match the query, the database can remove the sorting stage entirely.
-- ✗ No index: read every row → sort → take the first three Limit → Sort (Sort Method: top-N heapsort) → Seq Scan -- ✓ published_at index: the Sort stage disappears Limit → Index Scan Backward using idx_articles_pub -- The index guarantees order; stop after three rows
Disk: nnnn kB is a signal to remove the sort with an index or review work_mem.Return the three newest articles from the 2-million-row production articles table in descending published_at order. Also define an index that eliminates the sorting stage. Return article_id, title, published_at.
| article_id | title | published_at |
|---|---|---|
| 1 | PostgreSQL Basics | 2026-05-01 |
| 2 | Index Design | 2026-06-09 |
| 3 | Reading Execution Plans | 2026-04-12 |
| 4 | JOIN Basics | 2026-06-11 |
| 5 | Understanding NULL | 2026-03-30 |
| 6 | Sorting and Memory | 2026-06-02 |
| 7 | Using CTEs | 2026-05-20 |
Think of a news site's “latest articles” panel: sorting 2 million rows on every request versus reading only three entries.
Expected output (published_at descending, 3 rows):
| article_id | title | published_at |
|---|---|---|
| 4 | JOIN Basics | 2026-06-11 |
| 2 | Index Design | 2026-06-09 |
| 6 | Sorting and Memory | 2026-06-02 |
With the index, the Sort node disappears and the plan reads only the first three index entries plus three heap rows.
CREATE INDEX idx_articles_pub ON articles (published_at DESC); -- create an index in the ORDER BY sequence SELECT article_id, title, published_at FROM articles ORDER BY published_at DESC -- matches index order, so no Sort node is needed LIMIT 3; -- stop the scan after three rows /* Execution order: 1. Position at the leading edge of idx_articles_pub (largest published_at) 2. Read index entries in order and fetch the three columns from the heap 3. LIMIT stops the scan after three rows; no Sort stage exists */
LEGEND
1. Target table — heap rows are not date-ordered
FROM articles (physical order follows inserts and updates)The table heap is not arranged by published_at. Someone must create the required order: either a Sort at query time or an index that maintains it in advance.| article_id | title | published_at |
|---|---|---|
| 1 | PostgreSQL Basics | 2026-05-01 |
| 2 | Index Design | 2026-06-09 |
| 3 | Reading Execution Plans | 2026-04-12 |
| 4 | JOIN Basics | 2026-06-11 |
| 5 | Understanding NULL | 2026-03-30 |
| 6 | Sorting and Memory | 2026-06-02 |
| 7 | Using CTEs | 2026-05-20 |
work_mem, commonly 4 MB by default, writes temporary files for an external merge and can multiply I/O. Check Sort Method and Disk: in EXPLAIN ANALYZE.WHERE category = 'tech' ORDER BY published_at DESC LIMIT 3 is well served by (category, published_at DESC): find the equality slice, then read its first three ordered entries. This extends the “equality before range” rule to “equality before sort column.”SET LOCAL work_mem.ORDER BY a, b DESC, c creates index sprawl. Sorting becomes a concern when both row count and execution frequency are high. Leave a few hundred display rows to sort normally and reserve indexes for genuinely expensive latest-item and ranking queries.UNION removes duplicates while UNION ALL preserves them, but their performance is fundamentally different. UNION must sort or hash every selected column of every combined row to eliminate duplicates. UNION ALL merely appends one result to another, adding almost no work. When the data model makes overlap impossible, duplicate elimination is a tax with no benefit.
-- UNION: eliminate duplicates after Append; every row enters Sort or HashAggregate HashAggregate -- a hash table or sort for 5 million rows └─ Append → Seq Scan ×2 -- UNION ALL: only append the streams Append → Seq Scan ×2 -- no duplicate-elimination stage exists
The 3-million-row domestic_orders table and 2-million-row overseas_orders table use separate order-ID ranges, so the same order can never occur in both. Return one combined list of order_id, customer, amount without incurring an unnecessary duplicate-elimination stage.
| order_id | customer | amount |
|---|---|---|
| 1001 | Sato | 5000 |
| 1002 | Suzuki | 3200 |
| 1003 | Tanaka | 7800 |
| order_id | customer | amount |
|---|---|---|
| 9001 | Smith | 12000 |
| 9002 | Lee | 4500 |
Is “use UNION to be safe” really safe for performance? Consider what happens across 5 million rows.
Expected output (5 rows, no duplicate-elimination stage):
| order_id | customer | amount |
|---|---|---|
| 1001 | Sato | 5000 |
| 1002 | Suzuki | 3200 |
| 1003 | Tanaka | 7800 |
| 9001 | Smith | 12000 |
| 9002 | Lee | 4500 |
UNION ALL needs only Append. UNION performs duplicate checks across 5 million rows to produce the same answer.
SELECT order_id, customer, amount FROM domestic_orders UNION ALL -- mutually exclusive sources need no duplicate elimination SELECT order_id, customer, amount FROM overseas_orders; /* Execution order: 1. Scan domestic_orders and stream each row directly to the output 2. Scan overseas_orders next and append its rows 3. Perform no sorting, hashing, or duplicate comparisons */
LEGEND
1. Input — two mutually exclusive source tables
domestic_orders / overseas_ordersThe separate 1000 and 9000 ID ranges make it structurally impossible for one row to appear in both tables. Row colors show the source. This design guarantee is the basis for choosing UNION ALL.| source | order_id | customer | amount |
|---|---|---|---|
| domestic | 1001 | Sato | 5000 |
| domestic | 1002 | Suzuki | 3200 |
| domestic | 1003 | Tanaka | 7800 |
| overseas | 9001 | Smith | 12000 |
| overseas | 9002 | Lee | 4500 |
This final problem combines the second half's building blocks: 1. UNION ALL for mutually exclusive sources (Q9), 2. NOT EXISTS for exclusions (Q6: NULL-safe and Anti Join-friendly), and 3. FILTER for multiple conditional aggregates in one scan (Q7). The opposite pattern—UNION + NOT IN + one subquery per aggregate—costs several times more and may return a wrong result when NULL appears.
-- Composition pattern, read from the inside outward SELECT aggregate FILTER (...) -- 3. conditional metrics in one scan FROM ( ... UNION ALL ... ) s -- 1. combine exclusive sources WHERE NOT EXISTS ( ... ) -- 2. NULL-safe exclusion GROUP BY ...
Build a shipment report. Combine domestic shipments_jp and international shipments_intl, whose ID systems are mutually exclusive and whose production tables each contain millions of rows. Exclude shipments from test accounts, then aggregate pending and shipped counts by region. test_accounts.customer_id is nullable because pending applications may appear. Return region, pending_cnt, shipped_cnt ordered by region ascending.
| ship_id | status | customer_id |
|---|---|---|
| 1 | pending | 101 |
| 2 | shipped | 102 |
| 3 | pending | 103 |
| 4 | shipped | 101 |
| ship_id | status | customer_id |
|---|---|---|
| 51 | pending | 201 |
| 52 | shipped | 202 |
| 53 | shipped | 999 |
| account_id | customer_id |
|---|---|
| 1 | 999 |
| 2 | NULL |
Also identify where a UNION + NOT IN + status-specific subqueries version would become slow or incorrect.
Expected output (region ascending):
| region | pending_cnt | shipped_cnt |
|---|---|---|
| INTL | 1 | 1 |
| JP | 2 | 2 |
Only ship_id=53, the shipment for test account 999, is excluded. NOT IN would return zero rows because test_accounts contains NULL.
SELECT s.region, COUNT(*) FILTER (WHERE s.status = 'pending') AS pending_cnt, COUNT(*) FILTER (WHERE s.status = 'shipped') AS shipped_cnt FROM ( SELECT 'JP' AS region, status, customer_id FROM shipments_jp UNION ALL -- exclusive sources: ALL avoids duplicate-elimination cost SELECT 'INTL', status, customer_id FROM shipments_intl ) s WHERE NOT EXISTS ( -- NULL-safe exclusion through a Hash Anti Join SELECT 1 FROM test_accounts t WHERE t.customer_id = s.customer_id ) GROUP BY s.region ORDER BY s.region; /* Execution order: 1. Append shipments_jp and shipments_intl while adding region 2. Build a hash table from test_accounts for matching 3. Exclude matching rows with an Anti Join 4. Group by region and update the FILTER counters 5. Sort by region ascending and return the result */
LEGEND
1. Input — two exclusive sources plus an exclusion list
shipments_jp / shipments_intl / test_accountsThe shipments are split across two tables with exclusive ID systems. test_accounts contains customer_id 999 and NULL. That NULL would make the entire report empty as soon as NOT IN was used.| source | ship_id | status | customer_id |
|---|---|---|---|
| JP | 1 | pending | 101 |
| JP | 2 | shipped | 102 |
| JP | 3 | pending | 103 |
| JP | 4 | shipped | 101 |
| INTL | 51 | pending | 201 |
| INTL | 52 | shipped | 202 |
| INTL | 53 | shipped | 999 |
WHERE s.region = 'JP' lets the planner push it into UNION ALL branches and skip the INTL scan entirely through Append pruning. Writing “combine, then filter” can execute as “filter, then combine,” which is why UNION ALL is foundational to partitioning.