It is common to need a list of "types" from a table where identical rows appear many times. DISTINCT bundles rows whose combination of columns selected by SELECT is duplicated, returning only unique rows.
-- DISTINCT checks duplicates using the combination of columns selected SELECT DISTINCT user_id FROM access_logs; -- only the user_id values remain SELECT DISTINCT user_id, page FROM access_logs; -- checks the (user_id, page) pair
The access_logs table records multiple visits by the same users. Return a deduplicated list of user IDs who visited at least once.
| log_id | user_id | page |
|---|---|---|
| 1 | 101 | /home |
| 2 | 102 | /home |
| 3 | 101 | /mypage |
| 4 | 101 | /home |
| 5 | 103 | /home |
| 6 | 102 | /mypage |
Note: user_id=101 appears 3 times and 102 appears twice. Remove duplicates and return only the unique user_id values.
| user_id |
|---|
| 101 |
| 102 |
| 103 |
SELECT DISTINCT -- collapse duplicate user_id values user_id FROM access_logs ORDER BY user_id; /* Execution order (logical evaluation order in SQL): 1. FROM access_logs → read all 6 rows 2. SELECT user_id → project only the user_id column 3. DISTINCT → collapse duplicate user_id values 4. ORDER BY user_id → output user_id in ascending order */
LEGEND
① Source data
FROM access_logsRead the 6 rows of access_logs. The user_id column contains duplicate values: 101 appears 3 times and 102 appears twice.| log_id | user_id | page |
|---|---|---|
| 1 | 101 | /home |
| 2 | 102 | /home |
| 3 | 101 | /mypage |
| 4 | 101 | /home |
| 5 | 103 | /home |
| 6 | 102 | /mypage |
SELECT DISTINCT user_id checks only user_id, while SELECT DISTINCT user_id, page checks the (user_id, page) pair.The latter produces "unique pages visited by each user", so the number of output rows can change.SELECT DISTINCT user_id FROM access_logs and SELECT user_id FROM access_logs GROUP BY user_id return the same result.For simple deduplication without aggregates such as COUNT or SUM, DISTINCT is concise and readable; use GROUP BY when aggregation is needed.DISTINCT(user_id) as if DISTINCT were a function:DISTINCT is a keyword, not a function.SELECT DISTINCT(user_id), page still treats the parentheses as ordinary grouping, so DISTINCT ultimately applies to both user_id and page.The query unintentionally deduplicates by the (user_id, page) pair, producing more rows than expected.SELECT DISTINCT log_id, user_id includes a primary-key-like unique column such as log_id, every combination becomes unique and deduplication has no effect.When removing duplicates, select only the key columns you want to compare."Page views" and "visitors" are different measures. No matter how many times the same person visits, they count as one visitor. COUNT has three forms, each counting a different target.
COUNT(*) -- number of rows (counts duplicates and NULLs) COUNT(user_id) -- rows where user_id is not NULL COUNT(DISTINCT user_id) -- number of distinct user_id values
COUNT(*) counts total page views (PV), COUNT(DISTINCT user_id) counts unique users (UU).Repeated visits by the same user collapse to one person with COUNT(DISTINCT). Because PV is meant to count the access-log rows themselves, COUNT(*) is generally used instead of COUNT(user_id).access_logs and return, for each page, the "total access count (pv)" and "unique user count (uu)".
| log_id | user_id | page |
|---|---|---|
| 1 | 101 | /home |
| 2 | 102 | /home |
| 3 | 101 | /home |
| 4 | 103 | /home |
| 5 | 101 | /products |
| 6 | 102 | /products |
| 7 | 102 | /products |
Note: /home has 4 accesses, but user_id=101 is duplicated, so there are 3 unique users.
| page | pv | uu |
|---|---|---|
| /home | 4 | 3 |
| /products | 3 | 2 |
SELECT page, COUNT(*) AS pv, -- total access count (number of rows) COUNT(DISTINCT user_id) AS uu -- unique visitor count FROM access_logs GROUP BY page ORDER BY page; /* Execution order (logical evaluation order in SQL): 1. FROM access_logs → read all 7 rows 2. GROUP BY page → aggregate into 2 groups by page 3. COUNT(*) → count the rows (PV) in each group 4. COUNT(DISTINCT user_id) → count the distinct user_id values (UU) in each group 5. SELECT page, pv, uu → project 3 columns 6. ORDER BY page → output page in ascending order */
LEGEND
① Source data
FROM access_logsRead the 7 rows of access_logs. It contains duplicate rows where the same user_id visits the same page multiple times.| log_id | user_id | page |
|---|---|---|
| 1 | 101 | /home |
| 2 | 102 | /home |
| 3 | 101 | /home |
| 4 | 103 | /home |
| 5 | 101 | /products |
| 6 | 102 | /products |
| 7 | 102 | /products |
COUNT(*) counts the existence of rows (every row, including NULLs).COUNT(user_id) counts rows where user_id is not NULL.COUNT(DISTINCT user_id) counts distinct values. Confusing these three leads to aggregation mistakes: mixing up PV, non-NULL row counts, and UU.COUNT(*). For large data where an approximate unique count is acceptable, remember that approximate aggregation (such as a PostgreSQL extension) may be an option.COUNT(DISTINCT user_id, page), but PostgreSQL cannot take multiple columns directly; wrap them in a row expression such as COUNT(DISTINCT (user_id, page)). Syntax differs by database.Detecting duplicate data—such as an email address registered twice—is common in real work. Bundle equal values with GROUP BY, then keep only groups with "two or more rows" using HAVING COUNT(*) > 1.
GROUP BY email -- bundle rows with the same email HAVING COUNT(*) > 1 -- two or more rows in a group = duplicate
WHERE filters individual rows before aggregation, while HAVING filters groups after aggregation.COUNT(*) such as COUNT(*) cannot be used in WHERE; they can be used in HAVING.users table, find email addresses registered multiple times and their registration counts (cnt).
| user_id | name | |
|---|---|---|
| 1 | Tanaka | tanaka@example.com |
| 2 | Sato | sato@example.com |
| 3 | Tanaka | tanaka@example.com |
| 4 | Suzuki | suzuki@example.com |
| 5 | Sato | sato@example.com |
| 6 | Yamada | yamada@example.com |
Note: tanaka@example.com and sato@example.com each appear twice. Do not include addresses that appear only once.
| cnt | |
|---|---|
| sato@example.com | 2 |
| tanaka@example.com | 2 |
SELECT email, COUNT(*) AS cnt FROM users GROUP BY email HAVING COUNT(*) > 1 -- keep only groups with two or more rows (duplicates) ORDER BY email; /* Execution order (logical evaluation order in SQL): 1. FROM users → read the rows 2. GROUP BY email → group by email 3. HAVING COUNT(*) > 1 → keep only duplicate groups 4. SELECT email, cnt → project 2 columns 5. ORDER BY email → sort and output */
LEGEND
① Source data
FROM usersRead the 6 rows of users. The email column shows tanaka@example.com and sato@example.com twice each, suggesting duplicate registrations.| user_id | name | |
|---|---|---|
| 1 | Tanaka | tanaka@example.com |
| 2 | Sato | sato@example.com |
| 3 | Tanaka | tanaka@example.com |
| 4 | Suzuki | suzuki@example.com |
| 5 | Sato | sato@example.com |
| 6 | Yamada | yamada@example.com |
HAVING, which operates after aggregation. It helps to remember the order as WHERE → GROUP BY → HAVING.ROW_NUMBER() covered in Q4.WHERE COUNT(*) > 1 is a syntax error, so put aggregate filters in HAVING. Conversely, conditions that filter rows before aggregation (for example, excluding withdrawn users) belong in WHERE.GROUP BY name, birth_date. Looking at only email or only name can treat different people as duplicates or miss real duplicates. Define the duplicate key—the combination of columns that makes rows identical—first.HAVING COUNT(*) > 1 is a classic health-check query for asking whether data is clean, so make this pattern familiar.After detecting duplicates, the next step is elimination. When you want to keep exactly one representative row, ROW_NUMBER() is the tool. Use PARTITION BY for the duplicate key, ORDER BY for the priority that decides which row to keep, and retain only rows where rn = 1.
ROW_NUMBER() OVER ( PARTITION BY email -- key used to identify duplicates ORDER BY created_at DESC -- prefer the newest row (keep it) ) AS rn -- rn=1 is the representative row in each group
customers contains customers registered multiple times with the same email. Keep only the newest registration (the one with the latest created_at) for each email and return the unique customer list.
| customer_id | name | created_at | |
|---|---|---|---|
| 1 | tanaka@ex.com | Tanaka | 2024-01-10 |
| 2 | sato@ex.com | Sato | 2024-01-12 |
| 3 | tanaka@ex.com | Tanaka T | 2024-02-05 |
| 4 | suzuki@ex.com | Suzuki | 2024-01-20 |
| 5 | sato@ex.com | Sato S | 2024-03-01 |
Note: tanaka@ex.com and sato@ex.com each appear twice. Keep the newer one. Solve it with CTE + ROW_NUMBER.
| customer_id | name | created_at | |
|---|---|---|---|
| 3 | tanaka@ex.com | Tanaka T | 2024-02-05 |
| 4 | suzuki@ex.com | Suzuki | 2024-01-20 |
| 5 | sato@ex.com | Sato S | 2024-03-01 |
WITH ranked AS ( SELECT customer_id, email, name, created_at, ROW_NUMBER() OVER ( PARTITION BY email ORDER BY created_at DESC -- the newest row gets rn=1 ) AS rn FROM customers ) SELECT customer_id, email, name, created_at FROM ranked WHERE rn = 1 -- keep only one representative row per email ORDER BY customer_id; /* Execution order (logical evaluation order in SQL): 1. CTE(ranked) → read customers 2. ROW_NUMBER() OVER (...) → number rows newest-first within each email 3. FROM ranked → reference the derived table 4. WHERE rn = 1 → keep the newest row for each email 5. SELECT ... → project the columns 6. ORDER BY customer_id → sort and output */
LEGEND
① CTE — source data
FROM customersRead the 5 rows of customers. Treating email as the duplicate key, tanaka@ex.com (2 rows) and sato@ex.com (2 rows) are duplicated. Select the newest row from each duplicate set.| customer_id | name | created_at | |
|---|---|---|---|
| 1 | tanaka@ex.com | Tanaka | 2024-01-10 |
| 2 | sato@ex.com | Sato | 2024-01-12 |
| 3 | tanaka@ex.com | Tanaka T | 2024-02-05 |
| 4 | suzuki@ex.com | Suzuki | 2024-01-20 |
| 5 | sato@ex.com | Sato S | 2024-03-01 |
PARTITION BY defines what counts as a duplicate (the key), while ORDER BY defines which duplicate to keep (the priority).For "keep the newest", use ORDER BY created_at DESC; for "keep the oldest", use ASC; for "keep the highest priority", sort by any scoring column. Designing these two clauses lets you express deduplication rules freely.WHERE ROW_NUMBER() OVER(...) = 1 is invalid. Window functions are computed at the SELECT stage, so first create rn as a column in a CTE or subquery, then filter with WHERE rn = 1 outside it.RANK() gives both rank 1, leaving a duplicate. To guarantee exactly one row, use ROW_NUMBER; for a deterministic tie-breaker, add a column such as ORDER BY created_at DESC, customer_id DESC.DELETE FROM customers WHERE customer_id IN (SELECT customer_id FROM ranked WHERE rn > 1). As a safety rule, inspect the rn>1 rows with SELECT first instead of deleting immediately.Set operations stack multiple lists vertically. UNION stacks them and then automatically removes duplicate rows, while UNION ALL keeps every row, including duplicates.
SELECT email FROM campaign_a UNION -- remove duplicates (equivalent to DISTINCT) SELECT email FROM campaign_b; -- UNION ALL keeps every duplicate row (faster because there is no removal step)
Combine the applicant emails from two campaigns (campaign_a and campaign_b) into one deduplicated list.
| tanaka@ex.com |
| sato@ex.com |
| suzuki@ex.com |
| sato@ex.com |
| yamada@ex.com |
| tanaka@ex.com |
Note: tanaka@ex.com and sato@ex.com applied to both campaigns. Collapse the duplicates into one row.
| sato@ex.com |
| suzuki@ex.com |
| tanaka@ex.com |
| yamada@ex.com |
SELECT email FROM campaign_a UNION -- combine both lists and remove duplicates SELECT email FROM campaign_b ORDER BY email; /* Execution order (logical evaluation order in SQL): 1. SELECT FROM campaign_a → read the upper list 2. SELECT FROM campaign_b → read the lower list 3. UNION → stack vertically and remove duplicates 4. ORDER BY email → sort the combined result */
LEGEND
① Upper list — campaign_a
SELECT email FROM campaign_aThese are the 3 applicant emails from the first campaign. Stack this list vertically with campaign_b.| tanaka@ex.com |
| sato@ex.com |
| suzuki@ex.com |
UNION performs deduplication after stacking, so it incurs internal sorting or hashing costs. UNION ALL is faster because it returns every row as-is without a removal step. When you know duplicates cannot occur or want to keep them, UNION ALL is the performance best practice.ORDER BY once after the final SELECT. You cannot attach ORDER BY to each SELECT. The same set-operation family also includes INTERSECT for the intersection and EXCEPT for the difference.ORDER BY on an intermediate SELECT. Always sort once at the very end of the complete result.