Duplicate Data Management — Learn DISTINCT, EXCEPT, INTERSECT, and Representative-Row Selection from the Basics
BasicDuplicate data managementDISTINCTEXCEPT / INTERSECTSet operationsPostgreSQL-ready5 questions
QUESTION 6
Explore Combination Types with Multi-Column DISTINCT — How Does the Result Change as Columns Increase?
DISTINCTMultiple columnsCombinationsProjection
Background

In Q1, we applied DISTINCT to one column, as in SELECT DISTINCT user_id. When you list multiple columns in SELECT, DISTINCT treats all of them together as one key when checking for duplicates.

SELECT DISTINCT user_id              -- check only user_id → Q1
SELECT DISTINCT user_id, category     -- check the (user_id, category) pair
SELECT DISTINCT user_id, category, dt -- check the 3-column combination (even looser)
Adding columns makes “unique” less strict: The more columns you include, the more varied the combinations become, so the condition for treating rows as “different” becomes looser. With fewer columns, the condition is stricter and fewer rows remain.
Problem

purchase_logs records the same user purchasing the same category multiple times. Return the distinct combinations of categories that each user has purchased.

Table used
▸ purchase_logs
order_iduser_idcategoryamount
1101Books1500
2101Electronics3000
3102Books800
4101Books2000
5102Electronics5000
6103Books1200
7103Books900

Note: (101, Books) and (103, Books) each appear twice. Deduplicate by the (user_id, category) combination, not by user_id alone.

Expected Output
user_idcategory
101Books
101Electronics
102Books
102Electronics
103Books
Model Answer
SELECT DISTINCT  -- remove duplicates by the 2-column combination
  user_id,
  category
FROM purchase_logs
ORDER BY user_id, category;

/*
  Logical evaluation order:
  1. FROM purchase_logs          → read the rows
  2. SELECT user_id, category    → project 2 columns
  3. DISTINCT                    → remove duplicate combinations
  4. ORDER BY user_id, category  → sort and output
  */
Explanation (table transitions & key points)
SELECT DISTINCT user_id, category FROM purchase_logs ORDER BY user_id, category;
LEGEND
Rows read / loaded
① Source data
FROM purchase_logsRead the 7 rows of purchase_logs. Looking at both user_id and category, (101, Books) is duplicated twice (order_id=1,4), as is (103, Books) (order_id=6,7).
1 / 4
order_iduser_idcategoryamount
1101Books1500
2101Electronics3000
3102Books800
4101Books2000
5102Electronics5000
6103Books1200
7103Books900
purchase_logs: 7 rows (duplicate user_id/category combinations)
LEARNING POINTS
MULTI-COLUMN DISTINCT
Multi-column DISTINCT — deduplicate by the full column combination
Adding columns loosens the condition for treating rows as “different,” so more rows remain
DISTINCT (2 columns): 7 rows → 5 combinations
All selected columns become one combined key: DISTINCT checks duplicates using the “combination” of every column in the SELECT clause as one key. With DISTINCT user_id, category, pairs such as (101, Books) / (101, Electronics) / (102, Books) … remain separate unless the pair matches. This is why adding columns tends to loosen the uniqueness criterion and increase the number of result rows.
“One-column DISTINCT” vs. “two-column DISTINCT” — how the results differ: SELECT DISTINCT user_id returns only 3 combinations (101/102/103), while SELECT DISTINCT user_id, category returns 5 combinations. A different category remains a separate row even when user_id is the same. Decide what should count as “unique” first, then SELECT only those columns to get accurate results.
ANTI-PATTERNS
Adding a unique column prevents duplicates from disappearing: SELECT DISTINCT order_id, user_id, category includes a primary-key-like unique column (order_id), so every row becomes a different combination and DISTINCT has no effect. To remove duplicates, select only the key columns you want to compare. If you need other information, use a subquery or JOIN.
Writing DISTINCT(user_id) makes the parentheses meaningless: SELECT DISTINCT(user_id), category treats the parentheses as ordinary grouping, so DISTINCT ultimately applies to the (user_id, category) combination. This unintentionally makes the check two-column and can produce more rows than expected. DISTINCT is a keyword, so avoid putting parentheses immediately after it.
Field Notes
Multi-column DISTINCT is natural when you first clarify what kinds of combinations you want to list. Examples: categories purchased by users (e-commerce recommendations), tag/category pairs in use (admin screens), and assignee/project combinations (work-hour tracking). A common production pitfall is that duplicates remain after adding a column; in many cases, the cause is an unnecessary column being included. Build the habit of stating which column combination should be unique before writing SQL; it prevents bugs.
QUESTION 7
Get Data Present on Only One Side — Detect Differences with EXCEPT
EXCEPTSet operationsSet differenceDifference detection
Background

Q5 introduced UNION, a set operation that “adds” two lists. EXCEPT (called MINUS in Oracle) is the operation that “subtracts”: it removes rows in the right SELECT result from the left SELECT result.

SELECT email FROM A
EXCEPT             -- keep rows present in A but not in B (set difference A − B)
SELECT email FROM B;
Order changes the result: A EXCEPT B and B EXCEPT A produce different results. Unlike UNION, EXCEPT is non-commutative. Always keep track of which side you subtract from. EXCEPT also removes duplicates internally, so two identical values on the A side become one.
Problem

all_subscribers stores every newsletter subscriber, while unsubscribed stores the opt-out list. Return the email addresses of the current active subscribers who have not unsubscribed.

Table used
▸ all_subscribers
email
tanaka@ex.com
sato@ex.com
suzuki@ex.com
yamada@ex.com
▸ unsubscribed
email
sato@ex.com
yamada@ex.com

Note: subtract the 2 rows in unsubscribed (sato, yamada) from the 4 rows in all_subscribers. Order the result with ORDER BY email.

Expected Output
email
suzuki@ex.com
tanaka@ex.com
Model Answer
SELECT email FROM all_subscribers
EXCEPT                       -- subtract the right side from the left (set difference A − B)
SELECT email FROM unsubscribed
ORDER BY email;

/*
  Logical evaluation order:
  1. SELECT email FROM all_subscribers  → get the upper list of 4 rows
  2. SELECT email FROM unsubscribed     → get the lower list of 2 rows
  3. EXCEPT                             → remove rows from the upper list that appear below (set difference: A − B)
  4. ORDER BY email                     → sort email in ascending order
  */
Explanation (table transitions & key points)
SELECT email FROM all_subscribers EXCEPT SELECT email FROM unsubscribed ORDER BY email;
LEGEND
Rows read / loaded
① Upper list — all_subscribers
SELECT email FROM all_subscribersThis is the list of all newsletter subscribers (4 rows). EXCEPT removes rows that also appear in the opt-out list.
1 / 4
email
tanaka@ex.com
sato@ex.com
suzuki@ex.com
yamada@ex.com
all_subscribers: 4 rows
LEARNING POINTS
SET DIFFERENCE
Set difference A − B — remove rows in the right list from the left list
The reverse of UNION (adding): subtract an exclusion list and keep only active rows
all_subs(4) EXCEPT unsubscribed(2) = 2 rows
How EXCEPT works: check whether each A row exists in B, then remove it: EXCEPT compares each row in A with all of B and removes rows present in B. It also performs deduplication internally, so even two identical emails on the A side become one result. To preserve duplicates, use EXCEPT ALL (PostgreSQL-compatible).
Choosing between EXCEPT and NOT IN: The same result can be obtained with WHERE email NOT IN (SELECT email FROM unsubscribed), but EXCEPT makes the intent clearer and is easier to read. However, NOT IN requires care with NULL (a NULL in the subquery can easily cause a bug that excludes every row). EXCEPT handles rows containing NULL correctly, making it safer for difference detection.
The three core set operations — UNION / EXCEPT / INTERSECT: In Q5 we learned UNION (set union: A ∪ B), and in Q7 EXCEPT (set difference: A − B). Q9 covers INTERSECT (set intersection: A ∩ B). Together, these operations express adding, subtracting, and taking common elements entirely in SQL.
ANTI-PATTERNS
Confusing A EXCEPT B with B EXCEPT A: EXCEPT is an order-dependent, non-commutative operation. all_subscribers EXCEPT unsubscribed (get active subscribers) and unsubscribed EXCEPT all_subscribers (on the opt-out list but absent from the subscriber roster = detect inconsistent data) have completely different meanings. Clarify which side is the base and which side is subtracted before writing the query.
Adding ORDER BY to each SELECT: As with UNION, EXCEPT does not allow ORDER BY on each SELECT. Put one ORDER BY at the end of the entire statement. The two SELECTs must also have matching column counts and types, so the combined SELECTs must have the same number, order, and types of columns.
Field Notes
Difference detection is a frequent real-world use case. Examples: generating an active list excluding unsubscribers, detecting users who were present last month but absent this month, and finding master-data items that have not appeared in an activity table. EXCEPT is clearer in intent and NULL-safe compared with NOT IN / NOT EXISTS. However, it is available in PostgreSQL/SQL Server but unsupported in MySQL (older 5.x), so check the target database before using it.
QUESTION 8
Retrieve Every Duplicate Row — Identify the Actual Rows with a Subquery + IN
SubqueriesIN clauseDuplicate-row retrievalGROUP BY + HAVING
Background

In Q3, GROUP BY + HAVING COUNT(*) > 1 returned a list of “duplicate keys.” However, it only tells you which values are duplicated; it cannot return the specific rows (or other column information such as order_id or amount). Combining a subquery with IN lets you retrieve every row whose key is duplicated.

WHERE (user_id, product_id) IN (
  SELECT user_id, product_id
  FROM orders
  GROUP BY user_id, product_id
  HAVING COUNT(*) > 1   -- list only duplicated combinations
)
The standard two-step flow from Q3 to Q8: First use GROUP BY + HAVING to confirm “which keys are duplicated” (Q3), then use a subquery + IN to retrieve “all rows with those keys” (Q8). This is a common practical workflow.
Problem

The orders table contains duplicate (user_id, product_id) combinations. Return every order row whose combination is duplicated, including both rows in each duplicate pair.

Table used
▸ orders
order_iduser_idproduct_idamount
1101P0013000
2102P0021500
3101P0013000
4103P0032000
5102P0021500
6103P0015000

Note: the duplicated combinations are (101, P001) and (102, P002). Each has two rows, so the result contains 4 rows in total.

Expected Output
order_iduser_idproduct_idamount
1101P0013000
3101P0013000
2102P0021500
5102P0021500
Model Answer
SELECT
  order_id, user_id, product_id, amount
FROM orders
WHERE (user_id, product_id) IN (
  SELECT   user_id, product_id
  FROM     orders
  GROUP BY user_id, product_id
  HAVING   COUNT(*) > 1   -- list only duplicated combinations
)
ORDER BY user_id, product_id, order_id;

/*
  Logical evaluation order:
  1. Subquery GROUP BY + HAVING                 → extract duplicate keys
  2. FROM orders                                → read all rows in the outer query
  3. WHERE (...) IN                             → keep only rows with duplicate keys
  4. SELECT                                    → project the columns
  5. ORDER BY user_id, product_id, order_id     → sort and output
  */
Explanation (table transitions & key points)
SELECT order_id, user_id, product_id, amount FROM orders WHERE (user_id, product_id) IN ( SELECT user_id, product_id FROM orders GROUP BY user_id, product_id HAVING COUNT(*) > 1 ) ORDER BY user_id, product_id, order_id;
LEGEND
Rows read / loaded
① Source data
FROM ordersRead the 6 rows of orders. order_id=1 and 3 are duplicates of (101, P001), and order_id=2 and 5 are duplicates of (102, P002). The goal is to retrieve all of these rows.
1 / 4
order_iduser_idproduct_idamount
1101P0013000
2102P0021500
3101P0013000
4103P0032000
5102P0021500
6103P0015000
orders: 6 rows (duplicates present)
LEARNING POINTS
DUPLICATE ROW RETRIEVAL
Duplicate-row retrieval — detect duplicate keys with a subquery → filter rows with IN
An extension of Q3 (duplicate-key list): retrieve the specific row details too
GROUP BY + HAVING → key list → WHERE IN → retrieve all rows
The two-step flow of Q3 (detect duplicate keys) and Q8 (retrieve duplicate rows): Q3’s GROUP BY + HAVING COUNT(*) > 1 returns only the keys—“which emails are duplicated.” In practice, you often also need other columns (order date, amount, and so on), so the standard next step is to pull every row with those duplicate keys using a subquery + IN.
Row-tuple comparison with multiple columns as the IN key: WHERE (user_id, product_id) IN (...) Comparing multiple columns together with IN is called row-tuple comparison. It is available in PostgreSQL / MySQL 5.5+. With this pattern, you do not need to write one condition per row; you can compare the subquery’s multi-row result directly.
An equivalent rewrite with JOIN: The same result can be obtained with INNER JOIN (subquery) dup ON orders.user_id = dup.user_id AND orders.product_id = dup.product_id. For large datasets, JOIN may be easier for the optimizer to optimize, but IN is often more readable.
ANTI-PATTERNS
Stopping at Q3 without checking the actual rows: After finding duplicate keys with GROUP BY + HAVING, stopping with “we know how many duplicates there are” means which rows to delete (keep the newest or the oldest?) cannot be decided. Handle duplicates safely in this order: detect → inspect all rows → choose rows to delete → DELETE.
Excluding every row when a subquery returns NULL: This applies only when using NOT IN, but if the subquery result contains NULL, WHERE key NOT IN (..., NULL, ...) becomes false for every row. IN (the positive form) is safe here, but when using NOT IN, use WHERE key NOT IN (SELECT key FROM ... WHERE key IS NOT NULL) and make a habit of adding IS NOT NULL.
Field Notes
Retrieving every duplicate row is an essential pattern before correcting data. Examples: investigating duplicate orders (checking whether a customer accidentally ordered the same product twice), detecting double payments (two transactions with the same amount and timestamp), and cleansing before adding a unique constraint (confirming which rows to keep and delete). After identifying every row with this query, use Q4’s ROW_NUMBER to split them into “rows to keep (rn=1)” and “rows to delete (rn>1)” to complete the deduplication workflow.
QUESTION 9
Take the Common Part of Two Lists — Get the Set Intersection with INTERSECT
INTERSECTSet operationsSet intersectionCommon data
Background

INTERSECT (set intersection) returns only the common part of two SELECT results. Together with Q5’s UNION (set union) and Q7’s EXCEPT (set difference), it forms the three core set operations.

SELECT email FROM A
INTERSECT        -- keep rows present in both A and B (set intersection A ∩ B)
SELECT email FROM B;

-- Set-operation summary
UNION      -- A ∪ B: add (remove duplicates)
INTERSECT  -- A ∩ B: take the common part
EXCEPT     -- A − B: subtract
INTERSECT also removes duplicates internally: Even if one list contains the same value twice, it appears only once in the INTERSECT result. Use INTERSECT ALL to preserve duplicates (PostgreSQL-compatible).
Problem

Return the email addresses of users who signed up for both the spring and summer campaigns.

Table used
▸ campaign_spring
email
tanaka@ex.com
sato@ex.com
suzuki@ex.com
yamada@ex.com
▸ campaign_summer
email
sato@ex.com
kimura@ex.com
tanaka@ex.com
suzuki@ex.com

Note: sato, suzuki, and tanaka are the 3 people in both campaigns. Exclude yamada, who is only in spring, and kimura, who is only in summer.

Expected Output
email
sato@ex.com
suzuki@ex.com
tanaka@ex.com
Model Answer
SELECT email FROM campaign_spring
INTERSECT                    -- keep only rows present in both (set intersection A ∩ B)
SELECT email FROM campaign_summer
ORDER BY email;

/*
  Logical evaluation order:
  1. SELECT email FROM campaign_spring  → get the upper list of 4 rows
  2. SELECT email FROM campaign_summer  → get the lower list of 4 rows
  3. INTERSECT                          → keep only rows present in both (A ∩ B)
  4. ORDER BY email                     → sort email in ascending order
  */
Explanation (table transitions & key points)
SELECT email FROM campaign_spring INTERSECT SELECT email FROM campaign_summer ORDER BY email;
LEGEND
Rows read / loaded
① Upper list — campaign_spring
SELECT email FROM campaign_springThis is the spring campaign signup list (4 rows). Because yamada appears only in spring, INTERSECT excludes it.
1 / 4
email
tanaka@ex.com
sato@ex.com
suzuki@ex.com
yamada@ex.com
campaign_spring: 4 rows
LEARNING POINTS
SET INTERSECTION
Set intersection A ∩ B — return only rows present in both lists
Together with UNION (union) and EXCEPT (difference), this completes the three core set operations
spring(4) ∩ summer(4) = 3 common rows
The three core set operations: SQL set operations consist of UNION (set union: A ∪ B), INTERSECT (set intersection: A ∩ B), and EXCEPT (set difference: A − B). All remove duplicates internally; adding ALL (UNION ALL / INTERSECT ALL / EXCEPT ALL) preserves duplicates. Write ORDER BY only once, at the end of the entire statement.
INTERSECT versus INNER JOIN: The same result can be obtained with SELECT s.email FROM campaign_spring s INNER JOIN campaign_summer u ON s.email = u.email. JOIN is more flexible because it can reference other columns, but INTERSECT communicates the “present in both or not” purpose clearly and is readable. INTERSECT removes duplicates automatically, whereas JOIN can multiply duplicate rows.
INTERSECT is symmetric (commutative): Unlike EXCEPT, INTERSECT is commutative (A ∩ B = B ∩ A). spring INTERSECT summer and summer INTERSECT spring return the same result. The column count and types must still match, and ORDER BY still belongs once at the end.
ANTI-PATTERNS
Creating duplicates with INNER JOIN: When INNER JOIN replaces INTERSECT, duplicate rows on one side also multiply the JOIN result. For example, if sato appears twice in spring, the JOIN also returns two sato rows. INTERSECT removes them automatically, but JOIN requires a separate DISTINCT.
INTERSECT is unavailable in older MySQL versions: MySQL versions below 8.0.31 do not support INTERSECT. In that case, use IN + subquery (the Q8 pattern) or INNER JOIN as alternatives. Check the production database version before using it. PostgreSQL supports both INTERSECT and EXCEPT as standard set operations.
Field Notes
Finding common members is often used to reconcile multiple data sources. Examples: identifying heavy users who joined both campaigns (segment delivery to overlapping customers), checking master records present in both old and new systems (post-migration consistency checks), and extracting repeat customers who purchased both last month and this month (retention analysis). Mastering the three set operations (UNION/INTERSECT/EXCEPT) lets you express complex membership filtering in simple SQL.
QUESTION 10
Choose a Representative Row with GROUP BY + MIN — Deduplicate Without a Window Function
MIN / MAXGROUP BYDeduplicationSubquery + IN
Background

In Q4, we used ROW_NUMBER() (a window function) to choose one representative row per email. With MIN(id) + GROUP BY, you can achieve the same deduplication without a window function. It also works on older databases and has a simple structure.

-- Q4 approach (window function)
ROW_NUMBER() OVER (PARTITION BY email ORDER BY id ASC) = 1

-- Q10 approach (aggregation + subquery)
WHERE id IN (SELECT MIN(id) FROM t GROUP BY email)
-- switch to MAX(id) when you want to keep the latest row
MIN keeps the row with the smallest ID: MIN(id) chooses the row with the smallest contact_id as the representative. Use MIN to keep the oldest registration and MAX(id) to keep the newest. Q4’s ROW_NUMBER can specify any priority with ORDER BY, so it is more flexible when a compound priority is needed.
Problem

The contacts table contains multiple registrations with the same email address. Without using ROW_NUMBER(), return one row per email, keeping only the first row (the smallest contact_id).

Table used
▸ contacts
contact_idemailnamesource
1tanaka@ex.comIchiro Tanakaweb
2sato@ex.comHanako Satoweb
3tanaka@ex.comTaro Tanakaapp
4suzuki@ex.comJiro Suzukiweb
5sato@ex.comMiko Satosns
6yamada@ex.comKen Yamadaweb

Note: tanaka@ex.com (id=1,3) and sato@ex.com (id=2,5) are duplicated. Use MIN(contact_id) to keep the row with the smallest ID for each email.

Expected Output
contact_idemailnamesource
1tanaka@ex.comIchiro Tanakaweb
2sato@ex.comHanako Satoweb
4suzuki@ex.comJiro Suzukiweb
6yamada@ex.comKen Yamadaweb
Model Answer
SELECT
  c.contact_id, c.email, c.name, c.source
FROM contacts c
WHERE c.contact_id IN (
  SELECT MIN(contact_id)       -- choose the smallest ID for each email
  FROM   contacts
  GROUP BY email
)
ORDER BY c.contact_id;

/*
  Logical evaluation order:
  1. Subquery GROUP BY email       → extract the smallest ID for each email
  2. FROM contacts c               → read all rows in the outer query
  3. WHERE contact_id IN (...)     → keep rows with the smallest IDs
  4. SELECT                        → project the columns
  5. ORDER BY c.contact_id         → sort and output
  */
Explanation (table transitions & key points)
SELECT c.contact_id, c.email, c.name, c.source FROM contacts c WHERE c.contact_id IN ( SELECT MIN(contact_id) FROM contacts GROUP BY email ) ORDER BY c.contact_id;
LEGEND
Rows read / loaded
① Source data
FROM contactsRead the 6 rows of contacts. Using email as the duplicate key, tanaka@ex.com (id=1,3) and sato@ex.com (id=2,5) are duplicated. Choose the row with the smallest ID for each email.
1 / 4
contact_idemailnamesource
1tanaka@ex.comIchiro Tanakaweb
2sato@ex.comHanako Satoweb
3tanaka@ex.comTaro Tanakaapp
4suzuki@ex.comJiro Suzukiweb
5sato@ex.comMiko Satosns
6yamada@ex.comKen Yamadaweb
contacts: 6 rows (duplicate emails)
LEARNING POINTS
AGGREGATE DEDUPLICATION
Aggregate deduplication with MIN + GROUP BY — choose a representative row without a window function
An alternative to Q4 (ROW_NUMBER): remove duplicate emails in three simple steps
GROUP BY email → MIN(id) → WHERE IN → one representative row
Q4 (ROW_NUMBER) vs. Q10 (MIN + GROUP BY) — which should you choose?: ROW_NUMBER offers the flexibility to choose a representative row using any priority, such as ORDER BY created_at DESC. MIN/MAX only supports the fixed choice of keeping the row with the smallest or largest ID. “Keep the oldest registration (MIN)” and “keep the newest registration (MAX)” are simple cases where Q10 is sufficient. For compound conditions such as “prefer the newer date, then the larger ID on the same date,” use Q4’s ROW_NUMBER.
How the subquery + IN pattern works: The subquery returns a scalar list {1, 2, 4, 6}, and the outer query’s WHERE contact_id IN (1, 2, 4, 6) matches that list. Q8 used IN with a multi-column tuple, whereas Q10 uses the simpler one-column scalar form. This pattern—build a condition list in a subquery and filter with IN—applies broadly.
Applying the pattern to DELETE — remove everything except MIN(id): To actually clean duplicate data, write DELETE FROM contacts WHERE contact_id NOT IN (SELECT MIN(contact_id) FROM contacts GROUP BY email). NOT IN deletes the inverse of the subquery result (everything except the representatives). For safety, inspect the deletion targets with SELECT before running DELETE.
ANTI-PATTERNS
Assuming that MIN(id) also returns the other columns: If you write SELECT email, MIN(contact_id), name FROM contacts GROUP BY email, PostgreSQL and most MySQL modes raise an error because name is neither aggregated nor included in GROUP BY. To get the other columns (name, source) from the row selected by MIN(id), as in Q10, use an outer query with JOIN/IN on contacts to retrieve the complete row in two stages.
Misunderstanding the result when contact_id is not sequential: MIN(contact_id) returns the row with the smallest ID, but if contact_id is not sequential (for example, a UUID or a sequence with gaps), it is not necessarily the oldest registration. To keep the oldest, JOIN using MIN(created_at); to keep the newest, MAX(created_at) is accurate. Rely on IDs alone only when sequential IDs are a valid assumption.
Field Notes
Deduplication with MIN/MAX + GROUP BY is useful in older environments without window functions and whenever simplicity is the priority. Examples: consolidating duplicate registrations for the same email to the first row (customer-master cleansing), matching duplicate records for the same customer to the smallest ID, and checking targets before DELETE. Combining the patterns from Q1–Q10 yields a complete cleansing workflow: detect duplicates (Q3/Q8) → choose representative rows (Q4/Q10) → delete them (NOT IN + DELETE).