Duplicate Data Management — Learn DISTINCT, ROW_NUMBER, and UNION from the Basics
BasicDuplicate dataDISTINCTROW_NUMBERUNIONPostgreSQL-ready5 questions
QUESTION 1
Deduplicate Rows — Get a Unique List with SELECT DISTINCT
DISTINCTDeduplicationProjectionORDER BY
Background

It is common to need a list of "types" from a table where identical rows appear many times. DISTINCT bundles rows whose combination of columns selected by SELECT is duplicated, returning only unique rows.

-- DISTINCT checks duplicates using the combination of columns selected
SELECT DISTINCT user_id FROM access_logs;        -- only the user_id values remain
SELECT DISTINCT user_id, page FROM access_logs;  -- checks the (user_id, page) pair
DISTINCT applies to all selected columns: DISTINCT checks duplicates using the combination of every column listed in the SELECT clause, not just the one column immediately after it. The more columns you add, the looser the "unique" condition becomes, and the more rows remain.
Problem

The access_logs table records multiple visits by the same users. Return a deduplicated list of user IDs who visited at least once.

Table used
▸ access_logs
log_iduser_idpage
1101/home
2102/home
3101/mypage
4101/home
5103/home
6102/mypage

Note: user_id=101 appears 3 times and 102 appears twice. Remove duplicates and return only the unique user_id values.

Expected Output
user_id
101
102
103
Model Answer
SELECT DISTINCT  -- collapse duplicate user_id values
  user_id
FROM access_logs
ORDER BY user_id;

/*
  Execution order (logical evaluation order in SQL):
  1. FROM access_logs  → read all 6 rows
  2. SELECT user_id    → project only the user_id column
  3. DISTINCT          → collapse duplicate user_id values
  4. ORDER BY user_id  → output user_id in ascending order
  */
Explanation (table transitions & key points)
SELECT DISTINCT user_id FROM access_logs ORDER BY user_id;
LEGEND
Rows read / loaded
① Source data
FROM access_logsRead the 6 rows of access_logs. The user_id column contains duplicate values: 101 appears 3 times and 102 appears twice.
1 / 4
log_iduser_idpage
1101/home
2102/home
3101/mypage
4101/home
5103/home
6102/mypage
access_logs: 6 rows (duplicate user_id values)
LEARNING POINTS
6 RowsDISTINCT3 Unique
DISTINCT
Remove duplicate rows
Make rows unique by the combination of SELECT columns
DISTINCT works at the "selected-column combination" level:DISTINCT checks duplicates using all columns listed in the SELECT clause as one combined key.SELECT DISTINCT user_id checks only user_id, while SELECT DISTINCT user_id, page checks the (user_id, page) pair.The latter produces "unique pages visited by each user", so the number of output rows can change.
DISTINCT and GROUP BY are related:SELECT DISTINCT user_id FROM access_logs and SELECT user_id FROM access_logs GROUP BY user_id return the same result.For simple deduplication without aggregates such as COUNT or SUM, DISTINCT is concise and readable; use GROUP BY when aggregation is needed.
ANTI-PATTERNS
Writing DISTINCT(user_id) as if DISTINCT were a function:DISTINCT is a keyword, not a function.SELECT DISTINCT(user_id), page still treats the parentheses as ordinary grouping, so DISTINCT ultimately applies to both user_id and page.The query unintentionally deduplicates by the (user_id, page) pair, producing more rows than expected.
Adding a unique column so duplicates never disappear:SELECT DISTINCT log_id, user_id includes a primary-key-like unique column such as log_id, every combination becomes unique and deduplication has no effect.When removing duplicates, select only the key columns you want to compare.
Field Notes
"A list with duplicates removed" appears constantly in real work. Examples include a site's unique visitors, a deduplicated customer master, and the tags or categories actually in use. A common production accident is an unexpected column mixed into SELECT, making DISTINCT ineffective; first clarify which key defines a duplicate, then choose the columns.
QUESTION 2
Count Without Duplicates — COUNT(*) vs COUNT(DISTINCT column)
COUNT(DISTINCT)AggregationGROUP BYUnique count
Background

"Page views" and "visitors" are different measures. No matter how many times the same person visits, they count as one visitor. COUNT has three forms, each counting a different target.

COUNT(*)                  -- number of rows (counts duplicates and NULLs)
COUNT(user_id)            -- rows where user_id is not NULL
COUNT(DISTINCT user_id)   -- number of distinct user_id values
PV vs UU:COUNT(*) counts total page views (PV), COUNT(DISTINCT user_id) counts unique users (UU).Repeated visits by the same user collapse to one person with COUNT(DISTINCT). Because PV is meant to count the access-log rows themselves, COUNT(*) is generally used instead of COUNT(user_id).
Problem

access_logs and return, for each page, the "total access count (pv)" and "unique user count (uu)".

Table used
▸ access_logs
log_iduser_idpage
1101/home
2102/home
3101/home
4103/home
5101/products
6102/products
7102/products

Note: /home has 4 accesses, but user_id=101 is duplicated, so there are 3 unique users.

Expected Output
pagepvuu
/home43
/products32
Model Answer
SELECT
  page,
  COUNT(*)                AS pv,  -- total access count (number of rows)
  COUNT(DISTINCT user_id) AS uu   -- unique visitor count
FROM access_logs
GROUP BY page
ORDER BY page;

/*
  Execution order (logical evaluation order in SQL):
  1. FROM access_logs            → read all 7 rows
  2. GROUP BY page               → aggregate into 2 groups by page
  3. COUNT(*)                    → count the rows (PV) in each group
  4. COUNT(DISTINCT user_id)     → count the distinct user_id values (UU) in each group
  5. SELECT page, pv, uu         → project 3 columns
  6. ORDER BY page               → output page in ascending order
*/
Explanation (table transitions & key points)
SELECT page, COUNT(*) AS pv, COUNT(DISTINCT user_id) AS uu FROM access_logs GROUP BY page ORDER BY page;
LEGEND
Rows read / loaded
① Source data
FROM access_logsRead the 7 rows of access_logs. It contains duplicate rows where the same user_id visits the same page multiple times.
1 / 4
log_iduser_idpage
1101/home
2102/home
3101/home
4103/home
5101/products
6102/products
7102/products
access_logs: 7 rows
LEARNING POINTS
COUNT(*)Total: 4vsDISTINCTUnique: 3
COUNT(DISTINCT)
Count after deduplication
Distinguish PV (total) from UU (unique)
The three COUNT forms differ:COUNT(*) counts the existence of rows (every row, including NULLs).COUNT(user_id) counts rows where user_id is not NULL.COUNT(DISTINCT user_id) counts distinct values. Confusing these three leads to aggregation mistakes: mixing up PV, non-NULL row counts, and UU.
COUNT(DISTINCT) performs internal deduplication:COUNT(DISTINCT) may perform internal sorting or hashing to make values unique, so it tends to be heavier than COUNT(*). For large data where an approximate unique count is acceptable, remember that approximate aggregation (such as a PostgreSQL extension) may be an option.
ANTI-PATTERNS
Mind portability when using COUNT(DISTINCT a, b) with multiple columns:To count unique (user_id, page) pairs, MySQL allows COUNT(DISTINCT user_id, page), but PostgreSQL cannot take multiple columns directly; wrap them in a row expression such as COUNT(DISTINCT (user_id, page)). Syntax differs by database.
Forgetting GROUP BY turns the whole table into one group:If you forget GROUP BY page when you need page-level results, the whole table is aggregated as one group and returns only totals with no page distinction. A requirement phrased as "per ..." is a GROUP BY signal. As a rule, every non-aggregated column in SELECT should also appear in GROUP BY.
Field Notes
Distinguishing PV from UU is the foundation of web analytics. Examples include DAU / MAU (daily/monthly active users), unique purchasers (separate from total orders), and the number of distinct SKUs handled. When a report asks for a "count", first confirm whether it means total (COUNT(*)) or unique (COUNT(DISTINCT)) to prevent aggregation errors.
QUESTION 3
Detect Duplicate Data — GROUP BY + HAVING COUNT(*) > 1
HAVINGDuplicate detectionData qualityCOUNT(*)
Background

Detecting duplicate data—such as an email address registered twice—is common in real work. Bundle equal values with GROUP BY, then keep only groups with "two or more rows" using HAVING COUNT(*) > 1.

GROUP BY email              -- bundle rows with the same email
HAVING COUNT(*) > 1     -- two or more rows in a group = duplicate
WHERE vs HAVING:WHERE filters individual rows before aggregation, while HAVING filters groups after aggregation.COUNT(*) such as COUNT(*) cannot be used in WHERE; they can be used in HAVING.
Problem

users table, find email addresses registered multiple times and their registration counts (cnt).

Table used
▸ users
user_idnameemail
1Tanakatanaka@example.com
2Satosato@example.com
3Tanakatanaka@example.com
4Suzukisuzuki@example.com
5Satosato@example.com
6Yamadayamada@example.com

Note: tanaka@example.com and sato@example.com each appear twice. Do not include addresses that appear only once.

Expected Output
emailcnt
sato@example.com2
tanaka@example.com2
Model Answer
SELECT
  email,
  COUNT(*) AS cnt
FROM users
GROUP BY email
HAVING COUNT(*) > 1  -- keep only groups with two or more rows (duplicates)
ORDER BY email;

/*
  Execution order (logical evaluation order in SQL):
  1. FROM users           → read the rows
  2. GROUP BY email       → group by email
  3. HAVING COUNT(*) > 1  → keep only duplicate groups
  4. SELECT email, cnt    → project 2 columns
  5. ORDER BY email       → sort and output
  */
Explanation (table transitions & key points)
SELECT email, COUNT(*) AS cnt FROM users GROUP BY email HAVING COUNT(*) > 1 ORDER BY email;
LEGEND
Rows read / loaded
① Source data
FROM usersRead the 6 rows of users. The email column shows tanaka@example.com and sato@example.com twice each, suggesting duplicate registrations.
1 / 4
user_idnameemail
1Tanakatanaka@example.com
2Satosato@example.com
3Tanakatanaka@example.com
4Suzukisuzuki@example.com
5Satosato@example.com
6Yamadayamada@example.com
users: 6 rows
LEARNING POINTS
GroupsCOUNT>1FilterDupes
DUPLICATE DETECTION
Detect duplicate data
GROUP BY + HAVING COUNT(*) > 1
Why HAVING rather than WHERE:WHERE evaluates one row at a time before aggregation, so the COUNT(*) value does not exist yet. The condition "a group has at least two rows" can only be checked after groups are formed, so use HAVING, which operates after aggregation. It helps to remember the order as WHERE → GROUP BY → HAVING.
Identifying "which rows are duplicates" is a separate step:This query returns the duplicate value (email) and its count, but it does not decide which user_id row to keep or delete. To identify deletion targets one row at a time, use the row numbering with ROW_NUMBER() covered in Q4.
ANTI-PATTERNS
Writing WHERE COUNT(*) > 1 causes an error:Aggregate functions cannot be used in a WHERE clause. WHERE COUNT(*) > 1 is a syntax error, so put aggregate filters in HAVING. Conversely, conditions that filter rows before aggregation (for example, excluding withdrawn users) belong in WHERE.
Missing duplicates across a multi-column combination by checking only one column:For composite duplicates such as "same name and same date of birth", group by multiple columns with GROUP BY name, birth_date. Looking at only email or only name can treat different people as duplicates or miss real duplicates. Define the duplicate key—the combination of columns that makes rows identical—first.
Field Notes
Duplicate detection is the first step in data cleansing. Examples include pre-checking before adding a UNIQUE constraint (adding it fails when existing data contains duplicates), investigating double registrations or double charges, and finding duplicate candidates before entity resolution. HAVING COUNT(*) > 1 is a classic health-check query for asking whether data is clean, so make this pattern familiar.
QUESTION 4
Keep One Row per Duplicate — Retain the Representative with ROW_NUMBER()
ROW_NUMBERCTEDeduplicationPARTITION BY
Background

After detecting duplicates, the next step is elimination. When you want to keep exactly one representative row, ROW_NUMBER() is the tool. Use PARTITION BY for the duplicate key, ORDER BY for the priority that decides which row to keep, and retain only rows where rn = 1.

ROW_NUMBER() OVER (
  PARTITION BY email            -- key used to identify duplicates
  ORDER BY created_at DESC     -- prefer the newest row (keep it)
) AS rn                          -- rn=1 is the representative row in each group
Why DISTINCT is not enough:DISTINCT can remove only rows where every column matches exactly. It cannot handle partial duplicates where email is the same but name or created_at differs. ROW_NUMBER enables key-based deduplication, such as keeping one row for each email.
Problem

customers contains customers registered multiple times with the same email. Keep only the newest registration (the one with the latest created_at) for each email and return the unique customer list.

Table used
▸ customers
customer_idemailnamecreated_at
1tanaka@ex.comTanaka2024-01-10
2sato@ex.comSato2024-01-12
3tanaka@ex.comTanaka T2024-02-05
4suzuki@ex.comSuzuki2024-01-20
5sato@ex.comSato S2024-03-01

Note: tanaka@ex.com and sato@ex.com each appear twice. Keep the newer one. Solve it with CTE + ROW_NUMBER.

Expected Output
customer_idemailnamecreated_at
3tanaka@ex.comTanaka T2024-02-05
4suzuki@ex.comSuzuki2024-01-20
5sato@ex.comSato S2024-03-01
Model Answer
WITH ranked AS (
  SELECT
    customer_id, email, name, created_at,
    ROW_NUMBER() OVER (
      PARTITION BY email
      ORDER BY created_at DESC    -- the newest row gets rn=1
    ) AS rn
  FROM customers
)
SELECT
  customer_id, email, name, created_at
FROM ranked
WHERE rn = 1                  -- keep only one representative row per email
ORDER BY customer_id;

/*
  Execution order (logical evaluation order in SQL):
  1. CTE(ranked)              → read customers
  2. ROW_NUMBER() OVER (...)  → number rows newest-first within each email
  3. FROM ranked              → reference the derived table
  4. WHERE rn = 1             → keep the newest row for each email
  5. SELECT ...               → project the columns
  6. ORDER BY customer_id     → sort and output
  */
Explanation (table transitions & key points)
WITH ranked AS ( SELECT customer_id, email, name, created_at, ROW_NUMBER() OVER ( PARTITION BY email ORDER BY created_at DESC ) AS rn FROM customers ) SELECT customer_id, email, name, created_at FROM ranked WHERE rn = 1 ORDER BY customer_id;
LEGEND
Rows read / loaded
① CTE — source data
FROM customersRead the 5 rows of customers. Treating email as the duplicate key, tanaka@ex.com (2 rows) and sato@ex.com (2 rows) are duplicated. Select the newest row from each duplicate set.
1 / 4
customer_idemailnamecreated_at
1tanaka@ex.comTanaka2024-01-10
2sato@ex.comSato2024-01-12
3tanaka@ex.comTanaka T2024-02-05
4suzuki@ex.comSuzuki2024-01-20
5sato@ex.comSato S2024-03-01
customers: 5 rows (duplicate email values)
LEARNING POINTS
Partitionrn=1rn=2WHERE rn=1Top 1
DEDUPLICATION
Deduplication that keeps one representative
PARTITION BY + ORDER BY + WHERE rn=1
PARTITION BY and ORDER BY have different roles:PARTITION BY defines what counts as a duplicate (the key), while ORDER BY defines which duplicate to keep (the priority).For "keep the newest", use ORDER BY created_at DESC; for "keep the oldest", use ASC; for "keep the highest priority", sort by any scoring column. Designing these two clauses lets you express deduplication rules freely.
Why DISTINCT / GROUP BY cannot replace it:DISTINCT removes only exact full-row duplicates, and GROUP BY cannot return other non-aggregated columns such as name alongside the grouping key. ROW_NUMBER is the only way to keep the entire row while reducing duplicates for a specified key. That is why ROW_NUMBER is the go-to pattern for deduplication.
Window functions cannot be used directly in WHERE:WHERE ROW_NUMBER() OVER(...) = 1 is invalid. Window functions are computed at the SELECT stage, so first create rn as a column in a CTE or subquery, then filter with WHERE rn = 1 outside it.
ANTI-PATTERNS
Confusing ROW_NUMBER with RANK / DENSE_RANK:If two rows have the same created_at, RANK() gives both rank 1, leaving a duplicate. To guarantee exactly one row, use ROW_NUMBER; for a deterministic tie-breaker, add a column such as ORDER BY created_at DESC, customer_id DESC.
Omitting ORDER BY makes the retained row undefined:If you write only PARTITION BY and omit ORDER BY, which row becomes rn=1 is allowed to change from one execution to another. There must be a retention rule such as "keep the newest", so always specify ORDER BY.
Field Notes
ROW_NUMBER deduplication is one of the most common patterns in data cleansing and entity resolution. To actually delete duplicate rows, delete "non-representatives (rn>1)" with a query such as DELETE FROM customers WHERE customer_id IN (SELECT customer_id FROM ranked WHERE rn > 1). As a safety rule, inspect the rn>1 rows with SELECT first instead of deleting immediately.
QUESTION 5
Control Duplicates When Combining Lists — UNION vs UNION ALL
UNIONSet operationsUNION ALLDeduplication
Background

Set operations stack multiple lists vertically. UNION stacks them and then automatically removes duplicate rows, while UNION ALL keeps every row, including duplicates.

SELECT email FROM campaign_a
UNION          -- remove duplicates (equivalent to DISTINCT)
SELECT email FROM campaign_b;

-- UNION ALL keeps every duplicate row (faster because there is no removal step)
Align the columns:Every SELECT in a UNION must have the same number, order, and data types of columns. As with DISTINCT, duplicates are determined by the combination of all columns.
Problem

Combine the applicant emails from two campaigns (campaign_a and campaign_b) into one deduplicated list.

Table used
▸ campaign_a
email
tanaka@ex.com
sato@ex.com
suzuki@ex.com
▸ campaign_b
email
sato@ex.com
yamada@ex.com
tanaka@ex.com

Note: tanaka@ex.com and sato@ex.com applied to both campaigns. Collapse the duplicates into one row.

Expected Output
email
sato@ex.com
suzuki@ex.com
tanaka@ex.com
yamada@ex.com
Model Answer
SELECT email FROM campaign_a
UNION                       -- combine both lists and remove duplicates
SELECT email FROM campaign_b
ORDER BY email;

/*
  Execution order (logical evaluation order in SQL):
  1. SELECT FROM campaign_a  → read the upper list
  2. SELECT FROM campaign_b  → read the lower list
  3. UNION                   → stack vertically and remove duplicates
  4. ORDER BY email          → sort the combined result
  */
Explanation (table transitions & key points)
SELECT email FROM campaign_a UNION SELECT email FROM campaign_b ORDER BY email;
LEGEND
Rows read / loaded
① Upper list — campaign_a
SELECT email FROM campaign_aThese are the 3 applicant emails from the first campaign. Stack this list vertically with campaign_b.
1 / 5
email
tanaka@ex.com
sato@ex.com
suzuki@ex.com
campaign_a: 3 rows
LEARNING POINTS
ABUNIONUniqueA ∪ B
SET OPERATION
UNION vs UNION ALL
Deduplicate (UNION) or keep every row (UNION ALL)
The essential difference between UNION and UNION ALL:UNION performs deduplication after stacking, so it incurs internal sorting or hashing costs. UNION ALL is faster because it returns every row as-is without a removal step. When you know duplicates cannot occur or want to keep them, UNION ALL is the performance best practice.
Duplicate detection uses the "combination of all columns":Like DISTINCT, UNION determines duplicates using the combination of every selected column. This example is simple because it selects only email; with multiple columns, only rows where all columns match are duplicates. Every SELECT must also have matching column counts, order, and types.
Use ORDER BY only once, at the end:Sort the entire UNION by writing ORDER BY once after the final SELECT. You cannot attach ORDER BY to each SELECT. The same set-operation family also includes INTERSECT for the intersection and EXCEPT for the difference.
ANTI-PATTERNS
Using UNION for everything and paying unnecessary costs:Using UNION even when the data is known to contain no duplicates (for example, stacking month-partitioned tables) pays the sorting cost of unnecessary deduplication every time. Worse, using UNION to stack detail rows can remove legitimately separate rows just because their values match. UNION ALL is correct for stacking detail.
Mismatched column counts or types cause errors; adding ORDER BY to each SELECT:UNION raises an error when SELECT statements have different column counts or incompatible types. It is also invalid syntax to put ORDER BY on an intermediate SELECT. Always sort once at the very end of the complete result.
Field Notes
Combining lists is a standard real-world task. Examples include merging e-commerce and store customer lists without duplicates, consolidating masters from multiple data sources, and stacking tables split by month or year. The rule is simple: use UNION when merging rosters and removing duplicates; use UNION ALL when stacking detail as-is. Choosing based on whether duplicates should disappear balances correctness and performance.