Batch Processing — Learn SELECT, JOIN, Aggregation, Updates, and CTEs from the Basics
BasicSQL fundamentalsSELECT / WHERE / JOINAggregation and updatesCTEs and subqueriesAPI and batchPostgreSQL-ready5 questions
QUESTION 1
SELECT + WHERE + ORDER BY — The basic API data-retrieval pattern
SELECTWHEREORDER BYAPIretrieve
Background

SELECT is the basic command for retrieving rows and columns from a table. WHERE filters rows, while ORDER BY controls their order.

SELECT   col1, col2                 -- List the columns to retrieve (* means all columns)
FROM     table_name               -- Source table
WHERE    col1 = 'value'           -- Filter condition (optional)
ORDER BY col2 DESC;               -- Sort order (ASC ascending, DESC descending; ASC is the default)
WHERE comparison operators: = (equal), != or <> (not equal), > < >= <= (inequalities), LIKE '%word%' (partial match), IN (a, b, c) (one of several values), and IS NULL (NULL check). Do not use = NULL.

A standard user-list API filters active users with WHERE and uses ORDER BY to return the newest records first.

Problem

From the users table, retrieve users whose status is 'active' and plan is 'premium', ordered by created_at from newest to oldest. Return only user_id, name, email, created_at.

Tables used
▸ users
user_idnameemailstatusplancreated_at
1Taro Tanakatanaka@ex.comactivepremium2024-03-10
2Hanako Satosato@ex.cominactivepremium2024-02-01
3Ichiro Suzukisuzuki@ex.comactivefree2024-04-05
4Jiro Yamadayamada@ex.comactivepremium2024-01-20
5Saburo Itoito@ex.comactivepremium2024-05-15
Expected Output

Expected output (active and premium only, created_at descending):

user_idnameemailcreated_at
5Saburo Itoito@ex.com2024-05-15
1Taro Tanakatanaka@ex.com2024-03-10
4Jiro Yamadayamada@ex.com2024-01-20

Users 2 (inactive) and 3 (free plan) are excluded.

Model Answer
SELECT
  user_id,
  name,
  email,
  created_at
FROM   users
WHERE  status = 'active'   -- Keep active users on the premium plan
  AND  plan   = 'premium'
ORDER BY created_at DESC;  -- Newest first

/*
  Logical execution order:
  1. FROM users               → Read rows
  2. WHERE status, plan       → Filter rows
  3. SELECT                   → Evaluate columns
  4. ORDER BY created_at DESC → Sort and return
*/
Explanation (table transitions & key points)
SELECT user_id, name, email, created_at FROM users WHERE status = 'active' AND plan = 'premium' ORDER BY created_at DESC;
LEGEND
Rows read / loaded
① FROM
FROM usersRead all five rows from users. Every row is still eligible at this stage.
1 / 3
user_idnamestatusplan
1Taro Tanakaactivepremium
2Hanako Satoinactivepremium
3Ichiro Suzukiactivefree
4Jiro Yamadaactivepremium
5Saburo Itoactivepremium
All 5 rows read
LEARNING POINTS
Execution order: SELECT names output columns, FROM chooses the source table, and WHERE filters rows. The logical order is FROM → WHERE → SELECT, even though SQL is written in a different order.
Common API pattern: Convert query parameters such as GET /users?status=active&plan=premium into bound WHERE parameters. Prepared statements keep the SQL safe.
AND / OR precedence: AND binds more tightly than OR. Write WHERE a=1 OR (b=2 AND c=3) with parentheses whenever that is the intended meaning.
Why SELECT * is unsafe in production: Adding or removing a column can silently change an API response and break clients. List the required columns explicitly.
ANTI-PATTERNS
Comparing NULL with =: WHERE deleted_at = NULL never matches. Use IS NULL or IS NOT NULL because NULL represents an unknown value.
Assuming order without ORDER BY: An RDBMS may return rows in any order when ORDER BY is omitted. APIs must state their ordering explicitly.
Soft deletes and missing WHERE predicates
Web services often mark records as deleted with deleted_at instead of physically removing them. Add AND deleted_at IS NULL whenever the endpoint must exclude withdrawn users; do not rely blindly on an ORM default scope.
QUESTION 2
INNER JOIN — Build an API response from related tables in one query
INNER JOINONtablejoinAPI response
Background

JOIN connects tables through a shared column, usually a foreign key. INNER JOIN returns only rows for which matching data exists in both tables.

SELECT   a.col1, b.col2
FROM     table_a a                  -- a is an alias for table_a
INNER JOIN table_b b               -- INNER is optional; JOIN alone means INNER JOIN
  ON   a.id = b.a_id;             -- Join condition (primary key to foreign key)
INNER JOIN vs LEFT JOIN: INNER JOIN keeps only matching rows. LEFT JOIN keeps every row from the left table and fills right-side columns with NULL when no match exists. Use LEFT JOIN to include users with no orders and INNER JOIN to include only users with orders.

JOIN is essential when an API must return orders with user names. Combining the data in one query avoids an N+1 query loop in the application.

Problem

Join the orders and users tables. For orders whose status is 'completed', return order_id, user_name, amount, ordered_at, ordered by ordered_at from newest to oldest.

Tables used
▸ orders
order_iduser_idamountstatusordered_at
10115000completed2024-05-10
10223200pending2024-05-12
10318800completed2024-05-15
10431500completed2024-05-08
10546200cancelled2024-05-11
▸ users
user_idname
1Taro Tanaka
2Hanako Sato
3Ichiro Suzuki
4Jiro Yamada
Expected Output

Expected output (completed orders only, ordered_at descending):

order_iduser_nameamountordered_at
103Taro Tanaka88002024-05-15
101Taro Tanaka50002024-05-10
104Ichiro Suzuki15002024-05-08

Orders 102 (pending) and 105 (cancelled) are excluded.

Model Answer
SELECT
  o.order_id,
  u.name   AS user_name,
  o.amount,
  o.ordered_at
FROM       orders o
INNER JOIN users  u            -- Join rows that exist in both tables
  ON  o.user_id = u.user_id
WHERE  o.status = 'completed'  -- Completed orders only
ORDER BY o.ordered_at DESC;

/*
  Logical execution order:
  1. FROM orders o              → Read rows
  2. INNER JOIN users u         → Join matching rows
  3. WHERE o.status             → Filter rows
  4. SELECT                     → Evaluate columns
  5. ORDER BY o.ordered_at DESC → Sort and return
*/
Explanation (table transitions & key points)
SELECT o.order_id, u.name AS user_name, o.amount, o.ordered_at FROM orders o INNER JOIN users u ON o.user_id = u.user_id WHERE o.status = 'completed' ORDER BY o.ordered_at DESC;
LEGEND
Rows read / loaded
① FROM / INNER JOIN
FROM orders o INNER JOIN users uPrepare five order rows and four user rows. INNER JOIN combines only rows with a matching key.
1 / 4
▸ orders (o) — source table
order_iduser_idamountstatusordered_at
10115000completed2024-05-10
10223200pending2024-05-12
10318800completed2024-05-15
10431500completed2024-05-08
10546200cancelled2024-05-11
▸ users (u) — jointable
user_idname
1Taro Tanaka
2Hanako Sato
3Ichiro Suzuki
4Jiro Yamada
Left: 5 rows / Right: 4 rows
LEARNING POINTS
JOIN fundamentals: JOIN combines separate tables through a shared key such as user_id. It is a core relational-database operation.
Avoiding N+1 queries: Fetching a user name inside an order loop creates one query per order. A single JOIN minimizes database round trips.
Use aliases deliberately: Qualify columns as table.column or alias.column to avoid ambiguity. Short aliases also make complex queries readable.
Choosing INNER vs LEFT: Use LEFT JOIN when users with zero orders must remain; use INNER JOIN when only users with matching orders belong in the response.
ANTI-PATTERNS
Forgetting the ON condition: FROM orders, users or JOIN users without ON creates a Cartesian product. Always state the relationship explicitly.
Putting the join condition in WHERE: The old comma-join style mixes relationship logic with filters. Prefer explicit INNER JOIN ... ON.
N+1 queries and API response time
Loading the order list and then fetching each user name separately can turn 100 orders into 101 queries. JOIN the required related data once, while avoiding unnecessarily wide joins that consume memory.
QUESTION 3
GROUP BY + HAVING — Build a KPI summary API with aggregate functions
GROUP BYHAVINGCOUNT/SUMKPIaggregate
Background

GROUP BY groups rows by selected column values. Aggregate functions such as COUNT, SUM, AVG, MAX, and MIN reduce each group to one row. HAVING then filters the aggregate results, much like WHERE filters rows before aggregation.

SELECT
  group_col,
  COUNT(*)      AS row_count,
  SUM(amount)   AS total_amount
FROM   table_name
WHERE  status = 'completed'  -- Filter individual rows before grouping
GROUP BY group_col                  -- Create one group per value
HAVING COUNT(*) >= 2;            -- Filter groups after aggregation
WHERE vs HAVING: WHERE filters input rows before grouping. HAVING filters groups after aggregate functions have been calculated. Aggregate conditions such as COUNT(*) >= 2 belong in HAVING.
Problem

For completed orders in orders, calculate order_count and total_amount for each user_id. Return only users with at least two orders, ordered by total_amount descending.

Tables used
▸ orders
order_iduser_idamountstatus
1U013000completed
2U015000completed
3U028000completed
4U032000cancelled
5U011500completed
6U024500pending
7U036000completed
8U032500completed
Expected Output

Expected output (completed orders, order_count ≥ 2, total_amount descending):

user_idorder_counttotal_amount
U0139500
U0328500

U02 has only one completed order (order_id 3), so HAVING excludes it.

Model Answer
SELECT
  user_id,
  COUNT(*)    AS order_count,
  SUM(amount) AS total_amount
FROM   orders
WHERE  status = 'completed'  -- Filter rows before aggregation
GROUP BY user_id
HAVING COUNT(*) >= 2          -- Keep groups with at least two orders
ORDER BY total_amount DESC;

/*
  Logical execution order:
  1. FROM orders                → Read rows
  2. WHERE status               → Filter rows
  3. GROUP BY user_id           → Form groups
  4. COUNT(*) / SUM(amount)     → Aggregate each group
  5. HAVING COUNT(*) >= 2       → Filter groups
  6. SELECT                     → Evaluate columns
  7. ORDER BY total_amount DESC → Sort and return
*/
Explanation (table transitions & key points)
SELECT user_id, COUNT(*) AS order_count, SUM(amount) AS total_amount FROM orders WHERE status = 'completed' GROUP BY user_id HAVING COUNT(*) >= 2 ORDER BY total_amount DESC;
LEGEND
Rows read / loaded
① FROM
FROM ordersRead all seven rows from orders before applying any filters.
1 / 6
order_iduser_idamountstatus
1U013000completed
2U015000completed
3U028000completed
4U032000cancelled
5U011500completed
6U024500pending
7U036000completed
8U032500completed
All 7 rows read
LEARNING POINTS
GROUP BY creates one row per group: Aggregate functions such as COUNT and SUM summarize the rows in each user group.
WHERE vs HAVING: WHERE filters individual input rows before grouping; HAVING filters aggregate results after grouping. Use HAVING for conditions such as COUNT(*) >= 2.
Aggregate KPI queries: Combining COUNT and SUM in one query is a compact way to produce per-user dashboards and API summaries.
Filter early when possible: Restricting a recent time window in WHERE reduces the rows that GROUP BY must process.
ANTI-PATTERNS
Using HAVING for row-level filters: Put status = 'completed' in WHERE so unwanted rows are removed before aggregation.
Mixing ordinary and aggregate columns without GROUP BY: SELECT user_id, COUNT(*) FROM orders is ambiguous. Every non-aggregate selected column must be grouped.
Aggregating very large tables
For tens of millions of log rows, grouping everything at once can exhaust memory and cause slow queries. Filter the relevant period first, then aggregate the smaller working set for a reliable KPI API.
QUESTION 4
LIMIT + OFFSET — Pagination basics for APIs and page-number implementations
LIMITOFFSETPaginationAPI implementation
Background

LIMIT sets the maximum number of rows to return, while OFFSET skips rows from the beginning. Together they provide the basic page-number pagination used by endpoints such as /items?page=2&per_page=10.

SELECT col1, col2
FROM   table_name
ORDER BY col1 ASC  -- Always define a stable order
LIMIT  10             -- Number of rows per page
OFFSET 10;            -- Rows to skip: (page - 1) × page size
Page formula: OFFSET = (page - 1) × per_page. For page 2 with three rows per page, OFFSET is 3.

Large offsets force the database to scan and discard many rows. For large production APIs, cursor pagination based on the last retrieved ID is usually more efficient.

Problem

From products, return page 2 with three rows per page. Select product_id, name, price and order the rows by price ascending.

Tables used
▸ products (7 rows)
product_idnameprice
1Pen100
2Notebook200
3Eraser80
4Ruler150
5Scissors300
6Glue120
7Paper clip50
Expected Output

Sort by price ascending and return page 2 (items 4–6):

Pageproduct_idnameprice
1 page7,3,1Paper clip,Eraser,Pen50,80,100
Page 2 ★6Glue120
4Ruler150
2Notebook200
Model Answer
SELECT
  product_id,
  name,
  price
FROM   products
ORDER BY price ASC  -- Lowest price first
LIMIT  3               -- Return three rows
OFFSET 3;              -- Skip page 1

/*
  Logical execution order:
  1. FROM products     → Read rows
  2. ORDER BY price    → Sort rows
  3. OFFSET 3          → Skip three rows
  4. LIMIT 3           → Return three rows
*/
Explanation (table transitions & key points)
SELECT product_id, name, price FROM products ORDER BY price ASC LIMIT 3 OFFSET 3;
LEGEND
Rows read / loaded
① FROM
FROM productsRead all seven product rows. LIMIT and OFFSET are applied after sorting.
1 / 3
product_idnameprice
1Pen100
2Notebook200
3Eraser80
4Ruler150
5Scissors300
6Glue120
7Paper clip50
All 7 rows read
LEARNING POINTS
LIMIT and OFFSET: LIMIT caps the result size; OFFSET skips rows before the result begins. Together they split a large result into pages.
API implementation: For GET /products?page=2&per_page=3, calculate OFFSET = (page - 1) * per_page = 3. A separate COUNT query commonly supplies the total.
LIMIT alone: LIMIT 1 can fetch the newest item and LIMIT 5 can fetch the top five after an explicit ORDER BY.
Cursor pagination: Large offsets become slow because the database scans and discards earlier rows. A condition such as WHERE id > :last_id ORDER BY id LIMIT 10 can use an index instead.
ANTI-PATTERNS
LIMIT without ORDER BY: LIMIT 10 does not define which ten rows are returned. The order is not guaranteed.
OFFSET under concurrent inserts: New rows inserted between page requests can shift the offset, causing duplicates or skipped rows. Cursor pagination is safer for changing data.
The OFFSET trap and the rise of cursor pagination
With OFFSET 100000, the database may read and discard 100,000 rows before returning a page. Cursor-based APIs use the last ID as a stable boundary and remain efficient as the dataset grows.
QUESTION 5
INSERT + RETURNING — Return the database-generated ID from a create API
INSERTRETURNINGPOST APICreate API
Background

INSERT adds a new row to a table. PostgreSQL's RETURNING clause returns columns from the inserted row like SELECT. It is the standard way for a create API to return a database-generated ID without a second query.

INSERT INTO table_name (col1, col2)
VALUES ('value1', 'value2')
RETURNING id, col1;  -- Return values from the inserted row
INSERT structure: Write the target table and columns, then provide the same number of values in the same order. RETURNING is optional, but it safely returns generated values from this exact INSERT.

SERIAL, BIGSERIAL, and GENERATED ALWAYS AS IDENTITY columns are generated during INSERT; the application does not need to choose an ID in advance.

Problem

Create a row in tasks with title='Batch processing implementation', status='pending', and created_at set to the current time. Return the database-generated id and created_at.

Tables used
▸ tasks (table definition)
ColumnTypeNotes
idBIGSERIALAuto-generated primary key (do not specify)
titleTEXTTask name
statusTEXT'pending'/'running'/'done'
created_atTIMESTAMPTZCreation timestamp
▸ tasks (data before insert)
idtitlestatuscreated_at
1User syncdone2024-05-01 09:00
2Email deliveryrunning2024-05-10 11:00
Expected Output

Values returned by RETURNING (the generated id and current timestamp):

idcreated_at
32024-06-01 14:30:00+09

The database generates id=3 automatically; the application does not supply it.

Model Answer
INSERT INTO tasks (
  title,
  status,
  created_at
)
VALUES (
  'Batch processing implementation',
  'pending',
  NOW()                  -- Current timestamp
)
RETURNING id, created_at;  -- Return values from the inserted row

/*
  Execution flow:
  1. INSERT INTO tasks       → Prepare a new row
  2. VALUES                  → Assign values to columns
  3. BIGSERIAL               → Generate id automatically
  4. RETURNING id, created_at → Return the inserted row's values
*/
Explanation (table transitions & key points)
INSERT INTO tasks ( title, status, created_at ) VALUES ( 'Batch processing implementation', 'pending', NOW() ) RETURNING id, created_at;
LEGEND
Rows read / loaded
① INSERT INTO
Current tasks tableThe table contains two rows before the INSERT. The id column is generated automatically.
1 / 3
idtitlestatuscreated_at
1User syncdone2024-05-01 09:00
2Email deliveryrunning2024-05-10 11:00
Currently 2 rows
LEARNING POINTS
INSERT adds a row: The column list and VALUES list must have the same order and number of elements.
Typical POST flow: The client sends POST, the server executes INSERT + RETURNING, and the generated ID is included in the response. No follow-up SELECT is needed.
BIGSERIAL: BIGSERIAL combines BIGINT with a sequence that increments automatically. It provides a much larger range than SERIAL for large services.
RETURNING *: Returning every column also exposes database defaults, which is convenient for create-response payloads.
ANTI-PATTERNS
Reading the ID with SELECT MAX(id): Another concurrent request can insert a row between your INSERT and SELECT. RETURNING returns the ID belonging to this statement safely.
Mismatched column and value counts: INSERT INTO tasks (title, status) VALUES ('test') supplies one value for two columns and fails. Keep both lists aligned.
Combining bulk inserts with RETURNING
Batch jobs should insert many rows in one statement, such as VALUES (...), (...), (...), instead of paying network overhead for one INSERT per row. RETURNING id can return all generated IDs for downstream work.