CTEs and the WITH Clause — Learn Query Organization and Refactoring from the Basics
BasicCTEs (WITH clause)SQL refactoringReadabilityPostgreSQL-ready5 questions
QUESTION 6
Prevent Fan-Out with Pre-Aggregation — Aggregate Before Joining
Pre-aggregationAvoid fan-outLEFT JOINAvoid anti-patterns
Background

If you directly LEFT JOIN two child tables (for example, orders and reviews) to one parent table (for example, users), the rows multiply in a fan-out, producing incorrect aggregate results.

The cardinal rule for preventing this is to use CTEs to aggregate each child table into a one-to-one shape—one row per user_id—before joining them in the main query.

WITH child_a_totals AS (
  SELECT parent_id, SUM(value) AS total
  FROM child_a
  GROUP BY parent_id
),
child_b_counts AS (
  SELECT parent_id, COUNT(*) AS item_count
  FROM child_b
  GROUP BY parent_id
)
SELECT p.id, a.total, b.item_count
FROM parents p
LEFT JOIN child_a_totals a ON p.id = a.parent_id
LEFT JOIN child_b_counts b ON p.id = b.parent_id;
The danger of fan-out: If user_id=1 has placed three orders and written two reviews, a simple JOIN produces 3 × 2 = 6 rows and doubles the reported order total.
Problem

For each user in the users table, list the total order amount and review count.

  1. CTE user_orders: Aggregate orders and calculate the total amount per user (total_amount).
  2. CTE user_reviews: Aggregate reviews and calculate the review count per user (review_count).
  3. Main query: Use users as the base table and LEFT JOIN the two CTEs above. When a user has no orders or reviews, use COALESCE to display 0.
Tables used
▸ users
user_idname
1Alice
2Bob
▸ orders
order_iduser_idamount
115000
213000
3210000
▸ reviews
iduser_idrating
115
214
Expected Output
nametotal_amountreview_count
Alice80002
Bob100000
Model Answer
-- 1. Pre-aggregate orders by user
WITH user_orders AS (
  SELECT user_id, SUM(amount) AS total_amount
  FROM orders
  GROUP BY user_id
),
user_reviews AS (  -- 2. Pre-aggregate reviews by user
  SELECT user_id, COUNT(*) AS review_count
  FROM reviews
  GROUP BY user_id
)

-- 3. Join both pre-aggregated CTEs to users
SELECT
  u.name,
  COALESCE(o.total_amount, 0) AS total_amount, -- NULL → 0
  COALESCE(r.review_count, 0) AS review_count  -- NULL → 0
FROM users u
LEFT JOIN user_orders o ON u.user_id = o.user_id
LEFT JOIN user_reviews r ON u.user_id = r.user_id
ORDER BY u.user_id;

/*
  Execution order:
  1. user_orders (CTE)   → aggregate orders
  2. user_reviews (CTE)  → aggregate reviews
  3. LEFT JOIN to users   → join both CTEs (unmatched values are NULL)
  4. COALESCE            → convert NULL review_count to 0
  */
Explanation (table transitions & key points)
WITH user_orders AS ( SELECT user_id, SUM(amount) AS total_amount FROM orders GROUP BY user_id ), user_reviews AS ( SELECT user_id, COUNT(*) AS review_count FROM reviews GROUP BY user_id ) SELECT u.name, COALESCE(o.total_amount, 0) AS total_amount, COALESCE(r.review_count, 0) AS review_count FROM users u LEFT JOIN user_orders o ON u.user_id = o.user_id LEFT JOIN user_reviews r ON u.user_id = r.user_id ORDER BY u.user_id;
LEGEND
Grouping keys / aggregation targets
Group classification
1. Pre-Aggregate with Multiple CTEs
CTE: user_orders / user_reviewsGroup both orders and reviews by user in advance so that each contains one row per user. This is the key to preventing fan-out.
1 / 3
▸ user_orders (CTE)
user_idtotal_amount
18000
210000
▸ user_reviews (CTE)
user_idreview_count
12
CTEs complete: 2 order rows / 1 review row
LEARNING POINTS
Preventing fan-out with CTEs: Joining multiple one-to-many tables at once multiplies the data. CTEs are ideal for explicitly creating the processing units needed to prevent this: aggregate each child table into one row per parent—a one-to-one shape—before the JOIN.
Why COALESCE is essential: A LEFT JOIN returns NULL when no row exists in the child table. If an API response contains total_amount: null, the frontend can produce a calculation error (NaN), so it is standard API practice to use COALESCE(value, 0) to reliably return the numeric value 0.
QUESTION 7
CTEs × Cohort Analysis — Compare Groups Defined by Different Conditions
Condition comparisonINNER JOINDifferences and intersectionsCohort analysis
Background

Within a single table, comparing different periods or states—rows that satisfy condition A and also condition B—is not always straightforward.

In such cases, an intuitive and effective approach is to build one CTE for the list satisfying condition A, another for the list satisfying condition B, and JOIN them to find the intersection.

WITH set_a AS (
  SELECT DISTINCT id FROM events WHERE condition_a
),
set_b AS (
  SELECT DISTINCT id FROM events WHERE condition_b
)
SELECT a.id
FROM set_a a
JOIN set_b b ON a.id = b.id;
Problem

From the orders table, extract the user_id values of repeat customers who purchased in both January and February 2024.

  1. CTE jan_users: Select the user_id values that placed orders in January (remove duplicates).
  2. CTE feb_users: Select the user_id values that placed orders in February (remove duplicates).
  3. Main query: INNER JOIN both CTEs and return the user_id values present in both months.
Tables used
▸ orders
order_iduser_idordered_at
112024-01-15
222024-01-20
322024-02-10
432024-02-15
512024-03-05
Expected Output
user_id
2
Model Answer
-- 1. January customers
WITH jan_users AS (
  SELECT DISTINCT user_id  -- one row per customer
  FROM orders
  WHERE ordered_at >= '2024-01-01'
    AND ordered_at <  '2024-02-01'
),
feb_users AS (  -- 2. February customers
  SELECT DISTINCT user_id
  FROM orders
  WHERE ordered_at >= '2024-02-01'
    AND ordered_at <  '2024-03-01'
)

-- 3. Keep customers present in both months
SELECT
  j.user_id
FROM jan_users j
JOIN feb_users f  -- intersection only
  ON j.user_id = f.user_id;

/*
  Execution order:
  1. jan_users   → extract user_id from January orders
  2. feb_users   → extract user_id from February orders
  3. INNER JOIN  → keep only user_id values present in both lists
  */
Explanation (table transitions & key points)
WITH jan_users AS ( SELECT DISTINCT user_id FROM orders WHERE ordered_at >= '2024-01-01' AND ordered_at < '2024-02-01' ), feb_users AS ( SELECT DISTINCT user_id FROM orders WHERE ordered_at >= '2024-02-01' AND ordered_at < '2024-03-01' ) SELECT j.user_id FROM jan_users j JOIN feb_users f ON j.user_id = f.user_id;
LEGEND
Columns / keys under evaluation
1. Build a List for Each Condition
CTE: jan_users / feb_usersCreate the lists of customers who purchased in January and February as separate CTEs. Splitting complex conditions into parts makes them simple.
1 / 3
▸ jan_users (CTE)
user_id
1
2
▸ feb_users (CTE)
user_id
2
3
jan: 2 customers / feb: 2 customers
LEARNING POINTS
Finding differences and intersections without set operators: Here, an INNER JOIN finds the intersection. To find churned customers who bought in January but not February, LEFT JOIN feb_users f and then add WHERE f.user_id IS NULL to extract the difference easily (review of Q4).
Date ranges: BETWEEN '2024-01-01' AND '2024-01-31' can introduce a bug when timestamps are present because 2024-01-31 15:00:00 falls outside the range. The professional convention is to use >= start_of_month together with < start_of_next_month—a half-open interval.
QUESTION 8
CTEs × Window Functions — A Modern Latest-Row Query
ROW_NUMBERWindow functionsLatest-row queryModern SQL
Background

The technique from Q5—aggregate the maximum date and JOIN it back to the source table—is reliable, but it can return multiple rows when several orders share the same date.

In modern SQL, including PostgreSQL 8.4+ and MySQL 8.0+, combining the window function ROW_NUMBER() with a CTE has become the best practice—and de facto standard—for latest-row queries.

-- ROW_NUMBER() OVER (PARTITION BY grouping_key ORDER BY sort_order)
ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY ordered_at DESC)
-- → Within each user_id, number rows 1, 2, 3... from newest to oldest ordered_at
Why a CTE is necessary: SQL does not allow you to write ROW_NUMBER() = 1 directly in the WHERE clause. You must first assign row numbers in a CTE and then filter in the outer main query with WHERE rn = 1.
Problem

Use ROW_NUMBER() and a CTE to retrieve each user's latest order from the orders table.

  1. CTE ranked_orders: Use ROW_NUMBER() to assign a row number (rn) within each user_id, ordered by ordered_at in descending order (newest first).
  2. Main query: Keep only rn = 1 (the latest row) from the CTE and return user_id, order_id, amount, ordered_at.
Tables used
▸ orders
order_iduser_idamountordered_at
1130002024-01-10
2150002024-02-15
32100002024-01-20
4280002024-03-01
Expected Output
user_idorder_idamountordered_at
1250002024-02-15
2480002024-03-01
Model Answer
-- 1. Number each user's orders in a CTE
WITH ranked_orders AS (
  SELECT
    user_id,
    order_id,
    amount,
    ordered_at,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY ordered_at DESC) AS rn  -- newest first
  FROM orders
)

-- 2. Keep each user's latest row
SELECT
  user_id,
  order_id,
  amount,
  ordered_at
FROM ranked_orders
WHERE rn = 1  -- latest row
ORDER BY user_id;

/*
  Execution order:
  1. ranked_orders (CTE)  → assign rank (rn) within each user_id
  2. main query WHERE rn=1    → keep only each user's latest row
  */
Explanation (table transitions & key points)
WITH ranked_orders AS ( SELECT user_id, order_id, amount, ordered_at, ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY ordered_at DESC) AS rn FROM orders ) SELECT user_id, order_id, amount, ordered_at FROM ranked_orders WHERE rn = 1 ORDER BY user_id;
LEGEND
Columns / keys under evaluation
1. Assign Row Numbers with a Window Function
PARTITION BY user_id ORDER BY ordered_at DESCPARTITION BY separates rows by user, ORDER BY sorts each partition from newest to oldest, and ROW_NUMBER() assigns 1, 2, 3... to build a ranked temporary table.
1 / 2
order_iduser_idordered_atrn (row number)
212024-02-15 (latest)1
112024-01-10 (older)2
422024-03-01 (latest)1
322024-01-20 (older)2
Row numbers assigned to all 4 rows
LEARNING POINTS
The decisive difference between MAX + JOIN and ROW_NUMBER(): The MAX + JOIN approach from Q5 risks doubling rows if two orders happen to share the same timestamp, because both match the JOIN. By contrast, ROW_NUMBER() always forces a unique sequence such as 1 and 2 even for ties, so as long as you filter with rn = 1, rows can never multiply. This is the main reason ROW_NUMBER() is preferred in production.
Extending the pattern to Top-N queries: Simply change WHERE rn = 1 to WHERE rn <= 3 to retrieve each user's three most recent orders. This is another strength of window functions.
QUESTION 9
CTEs × Group Averages — A Modern Alternative to Correlated Subqueries
Group averagesAnalyticsSubquery alternativeComparative analysis
Background

Analytics workloads frequently compare an individual row with an aggregate value, such as an average, for the group to which that row belongs—for example, selecting products priced above their category average.

Historically, this was written with a correlated subquery—a SELECT executed from within the SELECT or WHERE clause—but that style is difficult to read and can perform poorly. The modern mainstream approach is to calculate category averages in a CTE, then JOIN and compare them in the main query.

WITH group_stats AS (
  SELECT group_id, AVG(value) AS avg_value
  FROM items
  GROUP BY group_id
)
SELECT i.id, i.value, g.avg_value
FROM items i
JOIN group_stats g ON i.group_id = g.group_id
WHERE i.value >= g.avg_value;
Problem

From the products table, select products whose price is greater than or equal to the average price of their own category.

  1. CTE category_avgs: Calculate the average price (avg_price) for each category_id.
  2. Main query: Join products with category_avgs and filter on price >= avg_price.
  3. Return the product id, name, price, and avg_price.
Tables used
▸ products
idnamecategory_idprice
1Laptop1120000
2Mouse15000
3Monitor125000
4Novel A21000
5Technical Book B25000
Expected Output
idnamepriceavg_price
1Laptop12000050000
5Technical Book B50003000
Model Answer
-- 1. Average price per category
WITH category_avgs AS (
  SELECT
    category_id,
    AVG(price) AS avg_price
  FROM products
  GROUP BY category_id
)

-- 2. Join products to their category average
SELECT
  p.id,
  p.name,
  p.price,
  c.avg_price
FROM products p
JOIN category_avgs c
  ON p.category_id = c.category_id
WHERE p.price >= c.avg_price  -- at or above average
ORDER BY p.id;

/*
  Execution order:
  1. category_avgs  → aggregate the average price for each category
  2. JOIN           → join each product to its category average
  3. WHERE          → keep only products at or above the average
  */
Explanation (table transitions & key points)
WITH category_avgs AS ( SELECT category_id, AVG(price) AS avg_price FROM products GROUP BY category_id ) SELECT p.id, p.name, p.price, c.avg_price FROM products p JOIN category_avgs c ON p.category_id = c.category_id WHERE p.price >= c.avg_price ORDER BY p.id;
LEGEND
Grouping keys / aggregation targets
Group classification
1. Calculate Each Category Average
GROUP BY category_id → AVG(price)First calculate the average price for each category and build a temporary table named category_avgs. These averages are the benchmark values used to compare individual products later.
1 / 3
category_idavg_price
150000
23000
5 rows → 2 groups (CTE complete)
LEARNING POINTS
Moving beyond correlated subqueries: You can also write this as WHERE price >= (SELECT AVG(price) FROM products p2 WHERE p.category_id = p2.category_id), using a correlated subquery. However, because the subquery may be evaluated for every row, performance can degrade. Using a CTE to aggregate everything first and then JOIN is faster and more modern. A window function such as AVG() OVER(PARTITION BY ...) can express the same logic cleanly as well.
QUESTION 10
Recursive CTEs (WITH RECURSIVE) — Traverse Hierarchical Trees
WITH RECURSIVERecursionTree structuresHierarchical data
Background

One of the most powerful CTE features is WITH RECURSIVE, a recursive CTE. It repeatedly uses one result to perform the next search until no more data is found.

It retrieves potentially unbounded hierarchies in a single SQL statement, including e-commerce category trees (parent → child → grandchild), threaded comment replies, and organization charts.

WITH RECURSIVE tree AS (
  SELECT id, parent_id              -- anchor term (starting point)
  FROM nodes
  WHERE id = 1

  UNION ALL

  SELECT n.id, n.parent_id         -- recursive term (next level)
  FROM nodes n
  JOIN tree t ON n.parent_id = t.id
)
SELECT * FROM tree;
Recursive CTE syntax:
1. Non-recursive term (initial step): SELECT the data that forms the starting point.
2. UNION ALL: The required keyword that connects the two terms.
3. Recursive term (repeating step): JOIN the source table with the CTE itself to find the next level.
Problem

The categories table represents parent-child relationships through parent_id. A NULL parent_id marks a top-level category.

Use a recursive CTE starting at id = 1 (Electronics) to retrieve all descendant categories below it, including children and grandchildren.

Tables used
▸ categories
idnameparent_id
1ElectronicsNULL
2PC1
3Laptop2
4Desktop2
5BooksNULL
Expected Output
idnameparent_id
1ElectronicsNULL
2PC1
3Laptop2
4Desktop2
Model Answer
-- RECURSIVE lets category_tree reference itself
WITH RECURSIVE category_tree AS (

  -- Step 1: anchor row
  -- Start at id = 1 (Electronics)
  SELECT id, name, parent_id
  FROM categories
  WHERE id = 1

  UNION ALL  -- append each iteration

  -- Step 2: recursive term
  -- Find children of the previous iteration
  SELECT
    c.id, c.name, c.parent_id
  FROM categories c
  JOIN category_tree ct     -- self-reference
    ON c.parent_id = ct.id  -- match each child to its parent
)

-- Step 3: final output
SELECT * FROM category_tree;

/*
  Execution simulation:
  1. Anchor: WHERE id = 1 selects Electronics (id=1).
  2. Iteration 1: using Electronics (id=1) as parent ct selects PC (id=2), whose parent_id is 1.
  3. Iteration 2: using PC (id=2) as parent ct selects Laptop (id=3) and Desktop (id=4), whose parent_id is 2.
  4. Iteration 3: using Laptop and Desktop as parents finds no rows, so recursion stops.
  5. The selected rows 1, 2, 3, and 4 are combined and returned. Books (id=5) is excluded because it belongs to another tree.
*/
Explanation (table transitions & key points)
WITH RECURSIVE category_tree AS ( SELECT id, name, parent_id FROM categories WHERE id = 1 UNION ALL SELECT c.id, c.name, c.parent_id FROM categories c JOIN category_tree ct ON c.parent_id = ct.id ) SELECT * FROM category_tree;
LEGEND
Columns / keys under evaluation
Excluded / hidden data
1. Non-Recursive Term (Get the Starting Point)
WHERE id = 1This is the first step of WITH RECURSIVE. It selects the starting category, id=1 (Electronics), and keeps it as level 1 of the hierarchy.
1 / 4
idnameparent_id
1ElectronicsNULL
2PC1
3Laptop2
4Desktop2
5BooksNULL
Retrieved: level 1 (Electronics)
LEARNING POINTS
Eliminating the N+1 query problem: If application code loops through “get the parent category → use its id to get children → use those ids to continue,” deeper hierarchies cause an enormous number of database calls. WITH RECURSIVE lets the database traverse the entire tree and return it in one response, delivering a dramatic performance improvement.
Beware of infinite loops: If bad data creates a cycle—A's parent is B and B's parent is A—the recursion may never terminate, placing a heavy load on the database. In production, enforce data integrity or add safeguards that limit recursion depth.