Batch Processing — Learn NULL, LIKE, EXISTS, window functions, and UPSERT from the Basics
BasicSQL FundamentalsDISTINCT / NULL / LIKEIN / EXISTS / date functionsWINDOW / UNIONTransactions / UPSERTPostgreSQL-ready5 questions
QUESTION 6
Window functions — Return rankings and running totals in one query with ROW_NUMBER, RANK, and SUM OVER
ROW_NUMBERRANKSUM OVERwindow functions
Background

Window functions assign a rank within a group or a running total to each row without aggregating rows. Unlike GROUP BY, they preserve the original row count while adding aggregate values.

ROW_NUMBER() OVER (               -- Assign consecutive numbers within a group (no duplicates)
  PARTITION BY grp_col           -- PARTITION BY: Group boundary (equivalent to GROUP BY)
  ORDER BY sort_col DESC         -- ORDER BY: Determine the order within the group
) AS rn

RANK()       OVER (...)           -- Equal values receive the same rank (the next rank is skipped: 1,1,3)
DENSE_RANK() OVER (...)           -- Equal values receive the same rank (no skip: 1,1,2)
SUM(col)     OVER (               -- Running total (accumulated row by row within the partition)
  PARTITION BY grp_col
  ORDER BY sort_col
)
The key difference from GROUP BY:GROUP BY reduces the row count by aggregating. Window functions preserve the row count and attach aggregate values to each row. Use them when you want every row plus a within-group rank.
Problem

From the sales table, calculate the sales ranking within each product category (ties receive the same rank) and the running sales total within each category. Sort by category, rank, and running total.

Tables used
▸ sales
sale_idcategoryproductamount
1BeveragesCoffee5000
2BeveragesTea3000
3BeveragesJuice3000
4FoodBread8000
5FoodCake6000
6FoodCookies4000
Expected Output

Expected output (RANK: equal sales share a rank; running totals accumulate in descending amount order):

categoryproductamountrank_in_catrunning_total
BeveragesCoffee500015000
BeveragesTea300028000
BeveragesJuice3000211000
FoodBread800018000
FoodCake6000214000
FoodCookies4000318000

Tea and Juice in Beverages both total 3000, so they share rank 2 (rank 4 is skipped)

Model Answer
SELECT
  category,
  product,
  amount,
  RANK() OVER (            -- Rank within each category
    PARTITION BY category  -- Partition by category
    ORDER BY amount DESC   -- descending amount
  ) AS rank_in_cat,        -- Equal values share a rank; the next rank is skipped (1,1,3)
  SUM(amount) OVER (               -- Running total within each category
    PARTITION BY category
    ORDER BY amount DESC
    ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW  -- Range from the first row through the current row
  ) AS running_total                 -- running total

FROM     sales
ORDER BY category, rank_in_cat;  -- Ascending category, then rank within each category

/*
  Logical execution order (window-function query):
  1. FROM sales                     → Read rows
  2. PARTITION BY category          → partition the windows
  3. RANK() OVER (...)              → Evaluate the window function (preserve the row count)
  4. SUM(amount) OVER (...)         → Evaluate the window function (preserve the row count)
  5. SELECT                         → Evaluate columns
  6. ORDER BY category, rank_in_cat → Sort and return
*/
Explanation (table transitions & key points)
SELECT category, product, amount, RANK() OVER ( PARTITION BY category ORDER BY amount DESC ) AS rank_in_cat, SUM(amount) OVER ( PARTITION BY category ORDER BY amount DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS running_total FROM sales ORDER BY category, rank_in_cat;
LEGEND
Rows read / loaded
① FROM
FROM salesRead all six rows from the sales table. The next step partitions the windows by category with PARTITION BY category.
1 / 4
sale_idcategoryproductamount
1BeveragesCoffee5000
2BeveragesTea3000
3BeveragesJuice3000
4FoodBread8000
5FoodCake6000
6FoodCookies4000
All 6 rows read
LEARNING POINTS
Core idea: The fundamental difference between window functions and GROUP BY: GROUP BY aggregates each category into one row, reducing the row count. Window functions add ranks and running values as columns to the original rows without changing the count. Use window functions when you want the original data plus ranks or running totals.
Typical use in a ranking API: To return only the top three products by sales within each category, calculate the rank and filter it in a CTE or subquery. Trying to achieve the same result with GROUP BY and JOIN instead of window functions makes the query extremely complex.
Meaning of ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: This specifies the range from the first row through the current row as the window frame. It is often the default for running totals, but writing it explicitly makes the intent clearer.
Unique sequence numbers with ROW_NUMBER: Use ROW_NUMBER when equal values must still receive different sequence numbers. It is commonly combined with a filter for deduplication or for selecting each user's latest order.
ANTI-PATTERNS
Filtering a window-function result directly in WHERE: WHERE RANK() OVER (...) <= 3 causes an error. Window functions can be used only in SELECT and ORDER BY. To filter, calculate the window value in a CTE or subquery and put WHERE in the outer query.
Confusing RANK and DENSE_RANK: RANK skips the next rank after a tie (1,1,3), while DENSE_RANK does not (1,1,2). If you use RANK to get the top three, a large tie can mean that rank 3 does not exist. Choose the function that fits the use case.
Practical column: Window functions greatly expand SQL expressiveness
Window functions may look difficult to SQL beginners, but once learned they let you write month-over-month calculations, moving averages over the latest N rows, and first-order identification per user in one query. In practice, ROW_NUMBER + CTE is especially common for retrieving the latest row in each group. Intermediate SQL users who rely only on GROUP BY may cut their query length by more than half after learning window functions.
QUESTION 7
Three-table JOIN — Build a detailed API query by joining orders, products, and categories
INNER JOINLEFT JOINThree tablesAPI design
Background

Most production APIs need to join multiple tables. JOINs can be chained, so it is important to understand both join types and join order.

SELECT  a.col, b.col, c.col
FROM       table_a a             -- Starting table
INNER JOIN table_b b             -- Join only rows that match on both sides
  ON a.b_id = b.id
LEFT JOIN  table_c c             -- Keep a rows even without a match in c (fill with NULL)
  ON b.c_id = c.id;
How to choose between INNER JOIN and LEFT JOIN: When the joined table always has data, use INNER JOIN. When rows with no joined data must remain, use LEFT JOIN. With LEFT JOIN, filtering rows where the joined table is NULL with WHERE c.id IS NULL returns only rows with no match.
Problem

Join the three tables order_items (order line items), products (products), and categories (categories), and attach product and category names to each order item. Include products without a category and show category_name as 'Uncategorized'.

Tables used
▸ order_items
item_idorder_idproduct_idqtyprice
1101P012500
2101P0211200
3102P033300
4102P011500
▸ products
product_idnamecategory_id
P01CoffeeC01
P02SandwichC02
P03New Product ANULL
▸ categories
category_idname
C01Beverages
C02Food
Expected Output

Expected output (products without a category are shown as 'Uncategorized'):

item_idorder_idproduct_namecategory_nameqtysubtotal
1101CoffeeBeverages21000
2101SandwichFood11200
3102New Product AUncategorized3900
4102CoffeeBeverages1500
Model Answer
SELECT
  oi.item_id,
  oi.order_id,
  p.name         AS product_name,
  COALESCE(c.name, 'Uncategorized') AS category_name, -- Uncategorized = a product with no LEFT JOIN match
  oi.qty,
  oi.price * oi.qty AS subtotal        -- Subtotal = unit price × quantity

FROM       order_items oi
INNER JOIN products   p             -- Products are required, so use INNER JOIN
  ON oi.product_id = p.product_id
LEFT JOIN  categories c             -- Keep products without categories (LEFT JOIN)
  ON p.category_id = c.category_id

ORDER BY oi.item_id;

/*
  Logical execution order (three-table JOIN query):
  1. FROM order_items oi          → Read rows
  2. INNER JOIN products p ON ... → Join (matching rows only)
  3. LEFT JOIN categories c ON ...→ Join (keep every row from the left table)
  4. SELECT                       → Evaluate columns (subtotal)
  5. ORDER BY oi.item_id          → Sort and return
*/
Explanation (table transitions & key points)
SELECT oi.item_id, oi.order_id, p.name AS product_name, COALESCE(c.name, 'Uncategorized') AS category_name, oi.qty, oi.price * oi.qty AS subtotal FROM order_items oi INNER JOIN products p ON oi.product_id = p.product_id LEFT JOIN categories c ON p.category_id = c.category_id ORDER BY oi.item_id;
LEGEND
Rows read / loaded
① FROM order_items
FROM order_items oiRead all order_items (4 rows). INNER JOIN products on product_id, then LEFT JOIN categories on category_id.
1 / 4
item_idorder_idproduct_idqtyprice
1101P012500
2101P0211200
3102P033300
4102P011500
order_items: 4 rows
LEARNING POINTS
Core idea: JOINs can be chained: You can JOIN table C to table B after joining B to table A. Continue writing JOIN clauses after FROM; tables are joined from left to right in the order FROM → JOIN1 → JOIN2 ... .
Typical order-detail API design: An e-commerce order-detail API (GET /orders/:id/items) commonly joins four or five tables such as order_items, products, categories, and users. A reliable pattern is to INNER JOIN required tables first and use LEFT JOIN for optional fields.
Naming aliases for joined tables: With three or more tables, short aliases such as oi (order_items), p (products), and c (categories) are common. If column names collide, qualify the column with its alias.
Interaction between WHERE and LEFT JOIN: If you add WHERE c.name = 'Beverages' for a column from the right table of a LEFT JOIN, the NULL row (New Product A) is removed and the query behaves like an INNER JOIN. Take care not to defeat the purpose of LEFT JOIN.
ANTI-PATTERNS
Mistyping an ON condition (Cartesian product): Omitting one ON condition in a three-table join creates a Cartesian product with that table. Check that every JOIN has an ON condition (joining N tables requires N−1 ON conditions).
Joining unnecessary tables: Adding a JOIN only for columns not used in SELECT increases join cost. Always ask whether the same result is possible without it, and remove unnecessary JOINs.
Practical column: JOIN order and the query planner
SQL specifies a join order, but the database query planner (optimizer) automatically chooses the best execution order. With appropriate indexes, the query can run quickly without hand-tuning JOIN order. For complex queries joining more than five tables, however, the planner may fail to find the best plan, so consider splitting the work with CTEs or temporary tables.
QUESTION 8
UNION / UNION ALL — Combine multiple queries vertically to build a multi-source report
UNION ALLUNIONVertical combinationBatch reporting
Background

UNION combines the results of multiple SELECT statements vertically. UNION ALL combines every row, including duplicates, while UNION without ALL removes duplicate rows.

SELECT col1, col2 FROM table_a     -- upper SELECT
UNION ALL                           -- Combine all rows vertically, including duplicates (faster than UNION)
SELECT col1, col2 FROM table_b;    -- Lower SELECT (match its column count and types to the upper SELECT)

SELECT col1, col2 FROM table_a
UNION                               -- Combine rows vertically after removing duplicates (DISTINCT runs internally)
SELECT col1, col2 FROM table_b;
UNION constraints: The SELECT statements must have the same number of columns and compatible types, or the query errors. Column names come from the first SELECT. Cast values when types differ.

Use JOIN to place different tables side by side (add columns), and UNION to stack them vertically (add rows).

Problem

The system has two notification tables: email_notifications and push_notifications. Combine notifications from both tables into one result, return the list in descending sent_at order, and add a notification type ('email'/'push') to each row.

Tables used
▸ email_notifications
iduser_idsubjectsent_at
1U01Order confirmation2024-05-10 09:00
2U02Announcement2024-05-12 14:00
3U01Shipping notification2024-05-15 11:00
▸ push_notifications
iduser_idmessagesent_at
1U01Coupon available!2024-05-11 10:00
2U03New product arrival2024-05-14 16:00
Expected Output

Expected output (Combine both tables, ordered by sent_at descending):

notification_typeuser_idcontentsent_at
emailU01Shipping notification2024-05-15 11:00
pushU03New product arrival2024-05-14 16:00
emailU02Announcement2024-05-12 14:00
pushU01Coupon available!2024-05-11 10:00
emailU01Order confirmation2024-05-10 09:00
Model Answer
SELECT
  'email'  AS notification_type,  -- Add the type as a fixed value
  user_id,
  subject  AS content,            -- Normalize the column name to content
  sent_at
FROM  email_notifications

UNION ALL                         -- Stack vertically without deduplication (fast)
SELECT
  'push'   AS notification_type,  -- Add the type as a fixed value
  user_id,
  message  AS content,            -- Match the column name used by email
  sent_at
FROM  push_notifications

ORDER BY sent_at DESC;  -- Sort the entire UNION result newest first

/*
  Logical execution order (UNION ALL query):
  1. Upper SELECT from email_notifications → Read rows
  2. Lower SELECT from push_notifications  → Read rows
  3. UNION ALL                      → combine the sets
  4. ORDER BY sent_at DESC          → Sort and return
*/
Explanation (table transitions & key points)
SELECT 'email' AS notification_type, user_id, subject AS content, sent_at FROM email_notifications UNION ALL SELECT 'push' AS notification_type, user_id, message AS content, sent_at FROM push_notifications ORDER BY sent_at DESC;
LEGEND
Rows read / loaded
① Upper SELECT
SELECT 'email' AS notification_type, ... FROM email_notificationsRead three rows from email_notifications and add the fixed string 'email' as notification_type. Normalize subject as content.
1 / 4
notification_typeuser_idcontentsent_at
emailU01Order confirmation2024-05-10 09:00
emailU02Announcement2024-05-12 14:00
emailU01Shipping notification2024-05-15 11:00
email: 3 rows
LEARNING POINTS
Core idea: Difference between JOIN and UNION: JOIN expands horizontally by adding columns and linking tables with relationship keys. UNION stacks vertically by adding rows and combining result sets with the same shape. Choose based on whether you need more columns or more rows.
Why UNION ALL is recommended: UNION without ALL sorts internally to remove duplicates and is slower than UNION ALL. Use UNION ALL whenever duplicates are impossible or acceptable.
Identifying the source with a fixed string: Adding a fixed-string column such as 'email' AS notification_type lets you identify the source table after combining the data. This pattern is common in batch reports and audit-log generation.
Use exactly one ORDER BY at the end: When combining SELECTs with UNION ALL, write one ORDER BY after the final SELECT. ORDER BY clauses inside individual SELECTs are ignored during UNION and may be syntax errors depending on the database.
ANTI-PATTERNS
Mismatched column counts or types: Different column counts or types in the two UNION SELECTs cause an error. If types differ, cast them explicitly with CAST(amount AS TEXT) or ::TEXT so they match.
ORDER BY inside each SELECT: SELECT ... FROM a ORDER BY sent_at UNION ALL SELECT ... FROM b can be a syntax error or behave unexpectedly. Write ORDER BY only after the final SELECT.
Practical column: Multi-tenant, multi-source data integration
UNION ALL is powerful when one API must return multiple data sources. It can combine delivery histories for a notification-history API or order, review, and comment actions into a chronological activity feed. As the number of tables grows, however, the number of UNION SELECTs grows too; consider redesigning around a unified event-log table.
QUESTION 9
Transactions — Process multiple updates atomically with BEGIN, COMMIT, and ROLLBACK
BEGINCOMMITROLLBACKACID
Background

A transaction treats multiple SQL statements as one unit of work. It starts with BEGIN, commits with COMMIT when everything succeeds, and rolls back all changes with ROLLBACK if an error occurs (atomicity).

BEGIN;                              -- Start the transaction (START TRANSACTION also works)

  UPDATE table_a SET col = val;    -- operation1
  INSERT INTO table_b VALUES (...); -- operation2
  -- If an error occurs here ↓

COMMIT;                             -- Commit all operations (persist them in the database)
-- Or --
ROLLBACK;                           -- Cancel all operations (return to the pre-BEGIN state)
ACID properties: A = atomicity (all succeed or all fail), C = consistency (constraints are always satisfied), I = isolation (transactions do not affect one another), and D = durability (a COMMIT survives power loss). Transactions provide these ACID guarantees.
Problem

Implement the e-commerce order flow (① INSERT an order row into orders, ② INSERT a line item into order_items, and ③ UPDATE inventory in products) as one atomic transaction. If any one of the three operations fails, make sure all of them are rolled back.

Tables to modify
▸ orders (order header)
order_iduser_idtotal_amountstatus
(NEW)U011500pending
▸ products (stock before processing)
product_idnamestock
P01Coffee100
Expected Output

State of each table after a successful transaction (after COMMIT):

TableChange
ordersA new order row (order_id=1001, user_id=U01, total=1500) is added
order_itemsA line item (order_id=1001, product_id=P01, qty=3, price=500) is added
productsP01 stock changes from 100 → 97 (three units removed)

If an error occurs partway through, ROLLBACK returns every table to its pre-BEGIN state

Model Answer
BEGIN;  -- Start the transaction (uncommitted until COMMIT)

-- ① Add the order header to orders
INSERT INTO orders (order_id, user_id, total_amount, status)
VALUES (1001, 'U01', 1500, 'pending');

-- ② Add the order line item to order_items
INSERT INTO order_items (order_id, product_id, qty, price)
VALUES (1001, 'P01', 3, 500);

-- ③ Decrease products inventory by the purchased quantity
UPDATE products
SET   stock = stock - 3    -- Decrease stock by 3 (relative update)
WHERE product_id = 'P01';  -- WHERE is required (without it, every row is updated)

COMMIT;  -- All operations succeeded → commit permanently (visible to other sessions)

/*
  Logical execution order (success path):
  1. BEGIN               → Start the transaction
  2. INSERT orders       → Insert rows (uncommitted)
  3. INSERT order_items  → Insert rows (uncommitted)
  4. UPDATE products     → Update the row (uncommitted)
  5. COMMIT              → Commit the changes permanently
  */
Explanation (table transitions & key points)
BEGIN; INSERT INTO orders ... INSERT INTO order_items ... UPDATE products SET ... COMMIT; ROLLBACK;
LEGEND
Rows read / loaded
① BEGIN
BEGIN; — Start the transactionBEGIN starts the transaction. Every INSERT/UPDATE after this is uncommitted (awaiting COMMIT), and other sessions cannot see these changes.
1 / 4
stepoperationordersorder_itemsproducts.stock
STARTBEGINunchangedunchanged100
Transaction started
LEARNING POINTS
Core idea: The transaction all-or-nothing principle: Operations between BEGIN and COMMIT either all succeed or all fail (ROLLBACK). This prevents partial states such as an order being created without reducing stock, or stock being reduced without an order row.
Transaction management in APIs: In Node.js and similar frameworks, the basic pattern is to put BEGIN through COMMIT in try-catch and call ROLLBACK in the catch block. Explicit transaction control can still be necessary when using an ORM.
Implicit transactions: PostgreSQL automatically commits a single SQL statement run without BEGIN as an internal transaction (autocommit). Use explicit BEGIN through COMMIT only when multiple SQL statements must be atomic.
Partial rollback with SAVEPOINT: Set SAVEPOINT sp1; partway through a transaction, then use ROLLBACK TO SAVEPOINT sp1; to undo only the changes up to that point. It is useful for retrying part of a long transaction.
ANTI-PATTERNS
Leaving a transaction open for too long: Leaving a transaction open after BEGIN without COMMIT or ROLLBACK keeps row locks on updated tables and blocks other queries, stalling the API. Keep transactions limited to the necessary operations and commit or roll them back immediately.
Mixing application logic into a transaction: HTTP requests or external API calls between BEGIN and the database operations keep locks held. Prepare data outside the transaction and keep only database operations inside it.
Practical column: Designing payment processing and transactions
Payment processing in an e-commerce site often follows “① charge through an external payment API → ② record the order in the database,” but the external API call cannot be inside the transaction because HTTP calls cannot be rolled back. In practice, record the order as "pending" first, change it to "completed" after payment succeeds, and change it to "failed" on failure; this status pattern provides idempotency. Transactions guarantee atomicity for database operations, but consistency with external services needs a separate design.
QUESTION 10
INSERT ON CONFLICT (UPSERT) — Make batch processing safe with idempotent data registration
INSERTON CONFLICTUPSERTidempotency
Background

UPSERT (INSERT + UPDATE) performs “INSERT if absent, UPDATE if present” in one statement. In PostgreSQL, use INSERT ... ON CONFLICT.

INSERT INTO table_name (col1, col2)
VALUES ('val1', 'val2')
ON CONFLICT (unique_col)            -- Specify the column whose unique constraint may be violated
DO UPDATE SET                       -- The UPDATE operation to run on conflict
  col2 = EXCLUDED.col2;            -- EXCLUDED = the new value that was about to be inserted

-- When nothing should be done on conflict:
ON CONFLICT (unique_col) DO NOTHING; -- Ignore the error and leave the existing row unchanged
Why UPSERT matters: Batch jobs may be rerun or retried. UPSERT prevents duplicate data and “row already exists” errors, enabling an idempotent operation that produces the same result every time.
Problem

Synchronize profile data obtained from an external API into the user_profiles table. INSERT a new row when user_id is absent; when it exists, update only name and updated_at (do not change email).

Table definition and current data
▸ user_profiles (table to UPSERT)
user_id (UNIQUE)nameemailupdated_at
U01Taro Tanakatanaka@example.com2024-04-01
U02Hanako Satosato@example.com2024-04-15
▸ Sync data from the external API
user_idnameemail
U01Taro Tanaka (renamed)tanaka@example.com
U03Ichiro Suzukisuzuki@example.com
Expected Output

user_profiles after UPSERT:

user_idnameemailupdated_at
U01Taro Tanaka (renamed) ← updatedtanaka@example.com (unchanged)2024-06-01 (updated)
U02Hanako Sato (unchanged)sato@example.com2024-04-15 (unchanged)
U03Ichiro Suzuki ← newsuzuki@example.com2024-06-01 (new)

U01: existing row → name updated and email retained / U02: absent from sync data, so unchanged / U03: new row → INSERT

Model Answer
-- Sync U01: existing row → update name and updated_at (email is unchanged)
INSERT INTO user_profiles (user_id, name, email, updated_at)
VALUES (
  'U01',
  'Taro Tanaka (renamed)',
  'tanaka@example.com',
  NOW()                               -- Set the sync timestamp
)
ON CONFLICT (user_id)               -- What to do when user_id conflicts (an existing row is present)
DO UPDATE SET                       -- Run UPDATE when a conflict occurs
  name       = EXCLUDED.name,      -- EXCLUDED: keyword for referencing the new value that was about to be inserted
  updated_at = EXCLUDED.updated_at; -- Leaving email out of SET preserves its original value

-- Sync U03: new row → INSERT (no ON CONFLICT occurs)
INSERT INTO user_profiles (user_id, name, email, updated_at)
VALUES (
  'U03',
  'Ichiro Suzuki',
  'suzuki@example.com',
  NOW()
)
ON CONFLICT (user_id)               -- Same UPSERT syntax, but no conflict → normal INSERT
DO UPDATE SET
  name       = EXCLUDED.name,
  updated_at = EXCLUDED.updated_at;

/*
  Logical execution order (UPSERT / U01 case):
  1. INSERT INTO user_profiles → Check the unique constraint
  2. Detect a unique-constraint violation → Conflict with the existing row
  3. ON CONFLICT DO UPDATE     → Update the existing row

  Logical execution order (UPSERT / U03 case):
  1. INSERT INTO user_profiles → Check the unique constraint
  2. No conflict                  → No constraint is violated
  3. normal INSERT             → Insert rows
*/
Explanation (table transitions & key points)
INSERT INTO user_profiles (user_id, name, email, updated_at) VALUES ('U01', 'Taro Tanaka (renamed)', 'tanaka@example.com', NOW()) ON CONFLICT (user_id) DO UPDATE SET name = EXCLUDED.name, updated_at = EXCLUDED.updated_at;
LEGEND
Rows read / loaded
① INSERT attempt (U01)
INSERT INTO user_profiles VALUES ('U01', ...)We attempt to INSERT U01 (Taro Tanaka, renamed). The database checks the UNIQUE constraint on user_id='U01'. U01 exists in the existing rows, so the ON CONFLICT clause fires.
1 / 3
user_idnameemailupdated_at
U01 (existing)Taro Tanakatanaka@example.com2024-04-01
U02 (existing)Hanako Satosato@example.com2024-04-15
user_profiles before UPSERT (2 rows)
INSERT INTO user_profiles (user_id, name, email, updated_at) VALUES ('U03', 'Ichiro Suzuki', 'suzuki@example.com', NOW()) ON CONFLICT (user_id) DO UPDATE SET name = EXCLUDED.name, updated_at = EXCLUDED.updated_at;
LEGEND
Rows read / loaded
① INSERT attempt (U03)
INSERT INTO user_profiles VALUES ('U03', ...)We attempt to INSERT U03 (Ichiro Suzuki). The database checks the UNIQUE constraint on user_id='U03'. U03 does not exist, so there is no conflict and it is processed as a normal INSERT.
1 / 2
user_idnameemailupdated_at
U01Taro Tanaka (renamed)tanaka@example.com2024-06-01
U02Hanako Satosato@example.com2024-04-15
State after U01 UPSERT (2 rows)
LEARNING POINTS
Core idea: What is UPSERT (UPDATE + INSERT)?: UPSERT performs “update if present, insert if absent” in one SQL statement. It removes the two-query application logic of checking with SELECT first and then choosing INSERT or UPDATE. Syntax differs by SQL dialect (MySQL uses INSERT ... ON DUPLICATE KEY UPDATE; PostgreSQL uses ON CONFLICT ... DO UPDATE).
Idempotency and batch processing: Idempotency means producing the same result no matter how many times an operation runs. Batch jobs may be rerun after network failures or server restarts; UPSERT enables an idempotent batch where the second run does not produce a duplicate-INSERT error.
The EXCLUDED keyword: In DO UPDATE SET, write EXCLUDED.column_name to reference the new value being inserted. user_profiles.column_name references the existing row. For example, SET click_count = user_profiles.click_count + EXCLUDED.click_count can add to the existing value.
Choosing DO NOTHING: ON CONFLICT DO NOTHING ignores the error and keeps the existing row unchanged. Use it for master-data initialization when only the first INSERT should take effect, or when duplicate inserts are possible but should not become errors.
ANTI-PATTERNS
Check-Then-Act: SELECT before INSERT/UPDATE: The logic IF EXISTS(SELECT ...) THEN UPDATE ELSE INSERT can encounter a duplicate-key error when another thread inserts between SELECT and INSERT in a multithreaded environment (a TOCTOU race). UPSERT is safer because it handles the operation atomically in one statement.
No unique constraint on the ON CONFLICT target: ON CONFLICT (user_id) errors unless user_id has a UNIQUE or primary-key constraint. Always add a database-level unique constraint to columns used with ON CONFLICT.
Practical column: Designing data-sync batches with external services
For a data-sync batch that periodically imports data from an external API or another system, UPSERT is usually the most stable choice. DELETE-all then INSERT-all (TRUNCATE + INSERT) risks another API reading an empty table during processing. UPSERT applies delta updates, preserving existing data while applying only changes. Updating updated_at = NOW() also records when synchronization occurred and helps investigate incidents.