Batch Processing — Learn SELECT through JOIN, aggregation, updates, and CTEs from the Basics
BasicSQL FundamentalsSELECT / WHERE / JOINAggregation and updatesCTEs and subqueriesAPIs and batch jobsPostgreSQL compatible5 questions
QUESTION 6
UPDATE + WHERE — A safe pattern for bulk status updates in batch jobs
UPDATEWHERESETStatus updates
Background

UPDATE changes column values in existing rows. The most important safety rule is to restrict the target rows with WHERE. Omitting WHERE updates every row in the table.

UPDATE table_name                   -- Target table
SET
  col1 = 'new_value',               -- Columns and values to update (comma-separated for multiple columns)
  col2 = NOW()                       -- Multiple columns can be updated at once
WHERE  id = 1;                       -- Required! Omitting it updates every row (a catastrophic bug)
UPDATE without WHERE is the biggest landmine: UPDATE users SET status = 'inactive' changes every user to inactive. Before running an UPDATE, first use SELECT with the same WHERE clause to verify the affected rows.

Common batch patterns include marking expired subscriptions inactive and invalidating expired tokens in bulk.

Problem

In the subscriptions table, find subscriptions whose end_date is today or earlier and whose status is 'active', then update their status to 'expired'. At the same time, set updated_at to the current time and use RETURNING to return the subscription_id and status of each updated row.

Tables used
▸ subscriptions (current date: 2024-06-01)
subscription_iduser_idstatusend_dateupdated_at
1U01active2024-05-202024-04-01
2U02active2024-06-302024-05-01
3U03active2024-05-312024-04-15
4U04expired2024-04-012024-04-02
5U05active2024-07-152024-05-10
Expected Output

Rows returned by RETURNING (the two active subscriptions with end_date ≤ 2024-06-01):

subscription_idstatus (after update)
1expired
3expired

IDs 2 and 5 have future end dates; ID 4 is already expired, so all three are excluded.

Model Answer
UPDATE subscriptions
SET
  status     = 'expired',
  updated_at = NOW()
WHERE  -- A carefully scoped WHERE clause prevents a full-table update
  end_date <= CURRENT_DATE     -- Expired rows
  AND status = 'active'       -- Active rows only (avoids needless updates)
RETURNING subscription_id, status;  -- Return updated rows

/*
  Logical execution order:
  1. FROM subscriptions    → Read rows
  2. WHERE                 → Filter rows
  3. SET status='expired'  → Update column values
  4. RETURNING             → Return updated rows
*/
Explanation (table transitions & key points)
UPDATE subscriptions SET status = 'expired', updated_at = NOW() WHERE end_date <= CURRENT_DATE AND status = 'active' RETURNING subscription_id, status;
LEGEND
Rows read / loaded
① FROM
FROM subscriptionsRead all five subscription rows. The current date is 2024-06-01.
1 / 4
subscription_iduser_idstatusend_dateupdated_at
1U01active2024-05-202024-04-01
2U02active2024-06-302024-05-01
3U03active2024-05-312024-04-15
4U04expired2024-04-012024-04-02
5U05active2024-07-152024-05-10
Read all 5 rows
LEARNING POINTS
Basics: UPDATE changes values in rows that already exist. Always pair UPDATE with a carefully reviewed WHERE clause; without it, every row in the table changes.
Standard expiration batch: A nightly job commonly runs WHERE end_date <= CURRENT_DATE AND status='active'. CRON plus this UPDATE and RETURNING lets the application log exactly how many subscriptions changed.
Ensuring idempotency: The AND status='active' predicate excludes rows already processed, so rerunning the batch produces the same result and is safe after a duplicate execution.
CURRENT_DATE vs NOW(): CURRENT_DATE contains a date without a time, while NOW() returns a timestamp. Use the former for date comparisons and the latter for update timestamps.
ANTI-PATTERNS
UPDATE without WHERE: UPDATE subscriptions SET status='expired' marks all five subscriptions expired. Before an UPDATE, run a matching SELECT COUNT(*) FROM subscriptions WHERE ... and review the count.
Typo in a SET column: SET stats = 'expired' fails if the column does not exist, but can silently change the wrong data if a similarly named column exists. Review column names before execution.
Practical column: Batch idempotency
Nightly jobs often stop partway because of network or infrastructure errors and are then rerun. A condition such as AND status = 'active' makes already-processed rows ineligible, so a retry is safe. When writing an UPDATE batch, always ask what happens if it stops halfway and starts again.
QUESTION 7
DELETE vs soft deletion — A safe API design for removing data
DELETEUPDATESoft deletionData preservation
Background

DELETE physically removes rows from a table. A soft delete keeps the row and records the deletion time in a deleted_at column. Soft deletion is common in API development because the data can be restored and audited.

-- Hard delete: permanently remove a row
DELETE FROM table_name             -- Target table
WHERE id = 1;                       -- Required; omitting it deletes every row

-- Soft delete: keep the row and record its deletion time
UPDATE table_name
SET    deleted_at = NOW()           -- Change NULL to the current timestamp
WHERE  id = 1
  AND  deleted_at IS NULL;          -- Only rows not yet deleted (idempotent)
Benefits of soft deletion: Data can be restored, deletion history remains available for auditing, and foreign-key references do not break. Queries for active data must include WHERE deleted_at IS NULL.
Problem

In the comments table, soft-delete the comment with comment_id=3 by recording the current time in deleted_at. Also write a query that returns only active comments, excluding rows that have been soft-deleted.

Tables used
▸ comments
comment_idpost_idbodydeleted_at
110Great articleNULL
210Very helpfulNULL
311Inappropriate commentNULL
411Thank you2024-05-01
Expected Output

Active comments after the soft delete (only rows where deleted_at IS NULL):

comment_idpost_idbodydeleted_at
110Great articleNULL
210Very helpfulNULL

ID 3 is deleted by this operation, and ID 4 was already deleted, so both are excluded.

Model Answer
-- ① Soft-delete comment 3 by recording the current time
UPDATE comments
SET    deleted_at = NOW()   -- Record the deletion time; do not remove the row
WHERE  comment_id = 3
  AND  deleted_at IS NULL;  -- Only undeleted rows

-- ② Return active comments whose deleted_at is NULL
SELECT
  comment_id,
  post_id,
  body
FROM   comments
WHERE  deleted_at IS NULL  -- Undeleted rows only
ORDER BY comment_id ASC;

/*
  Logical execution order for the SELECT:
  1. FROM comments               → Read rows
  2. WHERE deleted_at IS NULL    → Filter rows
  3. SELECT                      → Evaluate columns
  4. ORDER BY comment_id         → Sort and return
*/
Explanation (table transitions & key points)
UPDATE comments SET deleted_at = NOW() WHERE comment_id = 3 AND deleted_at IS NULL;
LEGEND
Rows read / loaded
① FROM
FROM comments before the soft deleteRead all four comments. ID 4 has already been soft-deleted.
1 / 3
comment_idbodydeleted_at
1Great articleNULL
2Very helpfulNULL
3Inappropriate commentNULL
4Thank you2024-05-01
all 4 rows read
SELECT comment_id, post_id, body FROM comments WHERE deleted_at IS NULL ORDER BY comment_id ASC;
LEGEND
Rows read / loaded
① FROM
FROM comments after the soft deleteRead the comments after IDs 3 and 4 have deletion timestamps.
1 / 3
comment_idpost_idbodydeleted_at
110Great articleNULL
210Very helpfulNULL
311Inappropriate comment2024-06-01
411Thank you2024-05-01
all 4 rows
LEARNING POINTS
Basics: DELETE physically removes data, while soft deletion records a deletion timestamp with UPDATE and keeps the row. This operational choice supports recovery and auditing.
Why soft deletion is common: It supports undo, preserves an audit trail, and avoids breaking foreign-key references. User and order data are often retained this way.
Filter every query: Once soft deletion is adopted, SELECT, UPDATE, and DELETE queries need WHERE deleted_at IS NULL unless deleted rows are explicitly intended. An ORM default scope or view can apply this rule consistently.
When hard deletion is appropriate: GDPR erasure requests and complete removal of PII may require physical deletion because a soft-deleted row still exists.
ANTI-PATTERNS
DELETE without WHERE: DELETE FROM comments removes every row. Check the intended count with SELECT COUNT(*) WHERE ... before running production DELETE statements.
Using a boolean instead of deleted_at: is_deleted = true/false loses when deletion occurred. A TIMESTAMPTZ deleted_at represents both active rows (NULL) and deleted rows (timestamp).
Practical column: Soft deletion and unique constraints
Soft deletion can prevent a user from registering again with the same email because the old row still occupies the unique constraint. Common schema-level solutions include a composite uniqueness strategy using email and deleted_at, or appending a deletion timestamp such as _deleted_<timestamp> to the archived value.
QUESTION 8
WITH (CTE) — Break complex batch queries into readable, reusable steps
WITHCTEQuery decompositionBatch processing
Background

A CTE (Common Table Expression) defines a temporary named result set with WITH that later parts of the query can reference. CTEs make complex batch queries easier to read by separating them into clear steps.

WITH cte1 AS (                 -- Define the first CTE
  SELECT col1, col2
  FROM   table_a
  WHERE  ...
),
cte2 AS (                      -- Separate CTEs with commas
  SELECT col1
  FROM   cte1                  -- Reference an earlier CTE
  WHERE  ...
)
SELECT *
FROM cte2;                      -- Finish with the main query
CTEs vs subqueries: Deeply nested subqueries are read from the inside out. Named CTEs present the same work as a top-to-bottom pipeline, which is especially useful in long batch SQL.
Problem

Identify users who placed at least two orders in the last 30 days as VIP customers. Using the orders and users tables, split the query into two CTE stages and return each VIP user's user_id, name, and order_count in descending order of order_count. Assume the current date is 2024-06-01.

Tables used
orders
order_iduser_idordered_at
1U012024-05-10
2U012024-05-20
3U022024-05-25
4U032024-04-01
5U012024-05-28
6U022024-05-29
7U032024-03-15
users
user_idname
U01Tanaka Taro
U02Sato Hanako
U03Suzuki Ichiro
Expected Output

Return VIP users with at least two orders in the last 30 days, sorted by order_count descending:

user_idnameorder_count
U01Tanaka Taro3
U02Sato Hanako2

U03 has no orders in the last 30 days and is excluded.

Model Answer
WITH
  recent_orders AS (               -- CTE 1: Count each user's orders in the last 30 days 
    SELECT
      user_id,
      COUNT(*) AS order_count
    FROM   orders
    WHERE  ordered_at >= CURRENT_DATE - INTERVAL '30 days'  -- last 30 days
    GROUP BY user_id
  ),
  vip_users AS (                   -- CTE 2: Keep users with at least two orders 
    SELECT user_id, order_count
    FROM   recent_orders            -- Read the result of the first CTE
    WHERE  order_count >= 2
  )
SELECT
  v.user_id,
  u.name,
  v.order_count
FROM       vip_users v
INNER JOIN users    u              -- join names to VIP users
  ON  v.user_id = u.user_id
ORDER BY v.order_count DESC;

/*
  Logical execution order:
  1. CTE recent_orders          → Build recent-order counts
  2. CTE vip_users              → Keep counts of at least two
  3. FROM vip_users v            → Read rows
  4. INNER JOIN users u          → Join matching rows
  5. SELECT                      → Evaluate columns
  6. ORDER BY order_count DESC   → Sort and return
*/
Explanation (table transitions & key points)
WITH recent_orders AS ( SELECT user_id, COUNT(*) AS order_count FROM orders WHERE ordered_at >= CURRENT_DATE - INTERVAL '30 days' GROUP BY user_id ), vip_users AS ( SELECT user_id, order_count FROM recent_orders WHERE order_count >= 2 ) SELECT v.user_id, u.name, v.order_count FROM vip_users v INNER JOIN users u ON v.user_id = u.user_id ORDER BY v.order_count DESC;
LEGEND
Rows read / loaded
Excluded / hidden data
① CTE: FROM + WHERE
recent_orders AS (FROM orders WHERE ordered_at >= 30 )Keep orders dated 2024-05-02 or later. U03's older orders are outside the 30-day window.
1 / 4
order_iduser_idordered_atwithin 30 days?
1U012024-05-10
2U012024-05-20
3U022024-05-25
4U032024-04-01
5U012024-05-28
6U022024-05-29
7U032024-03-15
7 rows → 5 recent rows
LEARNING POINTS
Basics: A WITH clause (CTE) defines a temporary named result set at the start of a query. Later clauses can read it like a table, which makes long SQL easier to organize.
Name CTEs for intent: Names such as recent_orders and vip_users tell the reader what each stage does and often remove the need for explanatory comments.
Reuse intermediate results: Define a repeated subquery once as a CTE and reference it from multiple places. This reduces duplicated code and improves maintainability.
Batch SQL structure: Separate extraction, filtering, and aggregation into named CTE stages, then build the final shape in the main query. During debugging, SELECT from each stage independently to inspect intermediate results.
ANTI-PATTERNS
Deeply nested subqueries: Three or four nested SELECT statements make the processing order difficult to understand. Refactor to CTEs when the nesting goes beyond a simple expression.
Carrying every column into a CTE: SELECT * FROM orders carries unused data into later stages. Select only the columns required by downstream logic for better readability and performance.
Practical column: CTE performance and materialized views
CTEs make SQL much easier to read, but repeated computation over very large data can still be expensive depending on the database version and plan. If an analytical batch takes minutes, consider storing intermediate results in a table or materialized view and indexing that stored result.
QUESTION 9
EXISTS / NOT EXISTS — Efficiently check for related data
EXISTSNOT EXISTSExistence checkCondition filter
Background

EXISTS evaluates to TRUE when its subquery returns at least one row and FALSE otherwise. It is frequently used in batch jobs to select rows that have related data or rows that have not yet been processed.

SELECT col1
FROM   table_a a
WHERE EXISTS (                 -- TRUE when the subquery returns a row
  SELECT 1                    -- The selected value is irrelevant
  FROM   table_b b
  WHERE  b.a_id = a.id       -- Correlate with the outer row
);

WHERE NOT EXISTS (...)            -- TRUE when the subquery returns no rows
EXISTS vs IN: EXISTS can stop after the first match and expresses an existence check directly. NOT EXISTS is also safe when the compared column may contain NULL.
Problem

From the users table, retrieve users who have never placed an order—that is, users with no matching row in orders. Return user_id and name, sorted by user_id ascending.

Tables used
users
user_idname
1Tanaka Taro
2Sato Hanako
3Suzuki Ichiro
4Yamada Jiro
5Ito Saburo
orders
order_iduser_idamount
10115000
10233200
10318800
Expected Output

Users who have never placed an order (no matching row in orders):

user_idname
2Sato Hanako
4Yamada Jiro
5Ito Saburo

User 1 has two orders and user 3 has one, so both are excluded.

Model Answer
SELECT
  u.user_id,
  u.name
FROM  users u
WHERE NOT EXISTS (              -- Keep only users with no orders 
  SELECT 1                      -- The value does not matter for an existence test
  FROM   orders o
  WHERE  o.user_id = u.user_id  -- Correlate orders with the outer user
)
ORDER BY u.user_id ASC;

/*
  Logical execution order:
  1. FROM users u               → Read rows
  2. NOT EXISTS (subquery) → Check for matching orders
  3. SELECT                     → Evaluate columns
  4. ORDER BY u.user_id         → Sort and return
*/
Explanation (table transitions & key points)
SELECT u.user_id, u.name FROM users u WHERE NOT EXISTS ( SELECT 1 FROM orders o WHERE o.user_id = u.user_id ) ORDER BY u.user_id ASC;
LEGEND
Rows read / loaded
① FROM
FROM users uRead all five users. The correlated NOT EXISTS subquery is evaluated for each user.
1 / 3
user_idname
1Tanaka Taro
2Sato Hanako
3Suzuki Ichiro
4Yamada Jiro
5Ito Saburo
Read all 5 rows
LEARNING POINTS
Basics: A subquery is a SELECT inside another SQL statement. EXISTS returns TRUE when that subquery returns at least one row, which makes it useful for checking related data.
Batch use cases: NOT EXISTS is common for finding users who have not received an email, or accounts that have not yet been billed. It expresses processed versus unprocessed state directly.
Why SELECT 1: EXISTS evaluates only whether a row exists, not the selected value. SELECT 1 is therefore conventional and communicates the intent more clearly than SELECT *.
NOT EXISTS vs LEFT JOIN: Both can produce the same anti-join result. PostgreSQL often optimizes them similarly, but NOT EXISTS makes the existence-check intent especially clear and is recommended here.
ANTI-PATTERNS
NULL with NOT IN: If the subquery contains one NULL, WHERE user_id NOT IN (...) can evaluate to UNKNOWN for every row and return nothing. NOT EXISTS is safer for nullable data.
Missing correlation: WHERE EXISTS (SELECT 1 FROM orders) is true for every user whenever orders has any row. The correlated predicate o.user_id = u.user_id is essential.
Practical column: Anti-joins (NOT EXISTS vs LEFT JOIN)
An anti-join finds data that does not exist in a related table. LEFT JOIN plus IS NULL is another valid form, and query planners often optimize both forms similarly. NOT EXISTS is frequently preferred in team code because its existence-check intent is explicit and easy to maintain.
QUESTION 10
CASE WHEN — Conditionally transform values in batch aggregates and API responses
CASE WHENTHEN/ELSEValue conversionAPI response
Background

CASE WHEN is SQL's if-else expression. It can be used in SELECT, WHERE, ORDER BY, GROUP BY, and many other contexts. It is common in APIs for converting stored values into labels and in batch aggregation for conditional counts.

-- Simple CASE: Branch on enumerated column values: 
CASE col1
  WHEN 'val1' THEN 'displayA'         -- col1 = 'val1'  
  WHEN 'val2' THEN 'displayB'         -- col1 = 'val2'  
  ELSE 'Other'                    -- When nothing matches (NULL if omitted)
END                                -- Always close CASE with END

-- Searched CASE: Branch on any condition; more flexible: 
CASE
  WHEN col2 > 100 THEN 'Large'         -- Comparison operators can be used
  WHEN col2 > 50  THEN 'Medium'         -- Evaluate top to bottom and return the first matching THEN
  ELSE 'Small'
END
CASE WHEN evaluation order: TRUE WHEN ... THEN rows
Problem

For each row in orders, assign a purchase rank based on amount: premium for 10000 or more, standard for 5000 or more, and basic otherwise. Also convert status codes to display labels. Return order_id, amount, rank_label, and status_label, sorted by ordered_at descending.

Tables used
orders
order_idamountstatusordered_at
112000completed2024-05-20
24800pending2024-05-18
37500cancelled2024-05-15
4500completed2024-05-10
55000pending2024-05-08
Expected Output

Expected output, with amount mapped to a rank and status mapped to an English label:

order_idamountrank_labelstatus_label
112000premiumCompleted
24800basicPending
37500standardCancelled
4500basicCompleted
55000standardPending
Model Answer
SELECT
  order_id,
  amount,
  CASE                              --  purchase rank 
    WHEN amount >= 10000 THEN 'premium'
    WHEN amount >=  5000 THEN 'standard'  -- Evaluate top to bottom (use the first match)
    ELSE                       'basic'
  END AS rank_label,
  CASE status                      -- Branch on a column value (simple CASE)
    WHEN 'completed' THEN 'Completed'
    WHEN 'pending'   THEN 'Pending'
    WHEN 'cancelled' THEN 'Cancelled'
    ELSE 'Unknown'                      -- Fallback for unexpected values
  END AS status_label

FROM   orders
ORDER BY ordered_at DESC;

/*
  Logical execution order:
  1. FROM orders                → Read rows
  2. SELECT                     → Evaluate rank_label and status_label
  3. ORDER BY ordered_at        → Sort and return
*/
Explanation (table transitions & key points)
SELECT order_id, amount, CASE WHEN amount >= 10000 THEN 'premium' WHEN amount >= 5000 THEN 'standard' ELSE 'basic' END AS rank_label, CASE status WHEN 'completed' THEN 'Completed' WHEN 'pending' THEN 'Pending' WHEN 'cancelled' THEN 'Cancelled' ELSE 'Unknown' END AS status_label FROM orders ORDER BY ordered_at DESC;
LEGEND
Rows read / loaded
① FROM
FROM ordersRead all five orders. The CASE expressions are evaluated for each row in SELECT.
1 / 4
order_idamountstatusordered_at
112000completed2024-05-20
24800pending2024-05-18
37500cancelled2024-05-15
4500completed2024-05-10
55000pending2024-05-08
Read all 5 rows
LEARNING POINTS
Basics: CASE WHEN is SQL's if-else expression. It can convert database values into display labels and can be used in SELECT, WHERE, ORDER BY, and GROUP BY clauses.
API response transformation: Store stable codes such as 'completed' in the database and use CASE to produce a response label. This can keep a simple one-query transformation close to the data.
Conditional aggregation: Expressions such as SUM(CASE WHEN status='completed' THEN amount ELSE 0 END) calculate a conditional total and are frequently combined with GROUP BY for KPI reports.
Always write ELSE: Omitting ELSE turns unmatched values into NULL, which can leak unexpected NULLs into an API response. Provide a fallback such as ELSE 'Unknown'.
ANTI-PATTERNS
Reversing WHEN order: In WHEN amount >= 5000 THEN 'standard' WHEN amount >= 10000 THEN 'premium', 12000 matches the first branch and never reaches premium. Order searched CASE conditions from most specific to least specific.
Forgetting END: Every CASE must have a matching END. Nested or repeated CASE expressions are easier to review when each END is aligned with its CASE.
Practical column: Should business logic live in the application or database?
CASE WHEN can keep conditional logic in SQL, but overusing it makes every specification change require SQL changes and can duplicate backend rules. A useful division is to keep filtering and aggregation conditions in the database while handling simple display labels and fine formatting in the application or BFF.