When there are many anomalies, first rank them to determine which are most anomalous and inspect only the top entries. The three ranking window functions differ in how they handle equal values (ties).
ROW_NUMBER() OVER (ORDER BY x DESC) -- 1,2,3,4 … always unique (even ties are numbered; order is nondeterministic) RANK() OVER (ORDER BY x DESC) -- 1,1,3,4 … ties share a rank; gaps occur DENSE_RANK() OVER (ORDER BY x DESC) -- 1,1,2,3 … ties share a rank; no gaps
OVER (), with neither PARTITION nor ORDER, treats the entire table as one window and attaches the same aggregate (for example, the overall average) to every row. Unlike GROUP BY, it does not collapse rows. Use that as the baseline for measuring each row's deviation.From daily_sales, calculate the deviation from the full-period average and its absolute value (abs_dev), then assign ROW_NUMBER, RANK, and DENSE_RANK in descending abs_dev order. Return dt, amount, deviation, abs_dev, rn, rnk, dense_rnk, ordered by descending abs_dev and ascending dt for ties. Round deviation and abs_dev to one decimal place.
| dt | amount |
|---|---|
| 2024-01-01 | 120 |
| 2024-01-02 | 130 |
| 2024-01-03 | 125 |
| 2024-01-04 | 50 |
| 2024-01-05 | 200 |
| 2024-01-06 | 122 |
| 2024-01-07 | 128 |
| 2024-01-08 | 125 |
The overall average is 125. The sharp drop on 01-04 (50) and surge on 01-05 (200) have deviations of the same magnitude (±75). This tie highlights the difference among the three ranking functions.
Expected output (descending abs_dev, ascending dt):
| dt | amount | deviation | abs_dev | rn | rnk | dense_rnk |
|---|---|---|---|---|---|---|
| 2024-01-04 | 50 | -75.0 | 75.0 | 1 | 1 | 1 |
| 2024-01-05 | 200 | 75.0 | 75.0 | 2 | 1 | 1 |
| 2024-01-01 | 120 | -5.0 | 5.0 | 3 | 3 | 2 |
| 2024-01-02 | 130 | 5.0 | 5.0 | 4 | 3 | 2 |
| 2024-01-06 | 122 | -3.0 | 3.0 | 5 | 5 | 3 |
| 2024-01-07 | 128 | 3.0 | 3.0 | 6 | 5 | 3 |
| 2024-01-03 | 125 | 0.0 | 0.0 | 7 | 7 | 4 |
| 2024-01-08 | 125 | 0.0 | 0.0 | 8 | 7 | 4 |
For the two tied 75.0 rows, rn becomes 1,2; rnk becomes 1,1 (then 3); and dense_rnk becomes 1,1 (then 2).
WITH dev AS ( SELECT dt, amount, amount - AVG(amount) OVER () AS deviation, -- Deviation from the full-period average ABS(amount - AVG(amount) OVER ()) AS abs_dev -- Absolute deviation (anomaly score) FROM daily_sales ) SELECT dt, amount, ROUND(deviation, 1) AS deviation, ROUND(abs_dev, 1) AS abs_dev, ROW_NUMBER() OVER (ORDER BY abs_dev DESC, dt) AS rn, -- Unique sequence (break ties by dt) RANK() OVER (ORDER BY abs_dev DESC) AS rnk, -- Equal values share a rank; gaps occur DENSE_RANK() OVER (ORDER BY abs_dev DESC) AS dense_rnk -- Equal values share a rank; no gaps FROM dev ORDER BY abs_dev DESC, dt; /* Execution order (logical SQL evaluation order): 1. FROM daily_sales → Read the rows 2. AVG(amount) OVER () → Evaluate the window function (preserve row count) 3. deviation / ABS() → Format the values 4. ROW_NUMBER / RANK / DENSE_RANK → Evaluate the window functions (preserve row count) 5. SELECT / ORDER BY abs_dev DESC, dt → Sort and return the output */
LEGEND
① FROM daily_sales (8 rows)
FROM daily_salesRead eight days of sales. The 50 drop on 01-04 and the 200 surge on 01-05 are anomaly candidates. Rank their deviation magnitudes to determine which is more anomalous.| dt | amount |
|---|---|
| 01-01 | 120 |
| 01-02 | 130 |
| 01-03 | 125 |
| 01-04 | 50 |
| 01-05 | 200 |
| 01-06 | 122 |
| 01-07 | 128 |
| 01-08 | 125 |
WHERE rn <= 3 returns exactly three rows; WHERE rnk <= 3 can return four or more rows because it includes ties (here rnk is 1,1,3,3, so it returns four). dense_rnk <= 3 returns the top three abs_dev groups: six rows. Choose for the intended outcome.AVG(amount) OVER () attaches a baseline to every row at once. Unlike GROUP BY, it preserves row count, so deviations can be calculated while retaining each row.ORDER BY abs_dev DESC, dt.rn <= N can keep one and miss another. Use RANK when all tied entries must be kept.ORDER BY abs_dev DESC LIMIT 3 handles ties nondeterministically and arbitrarily drops boundary events. A ranking function makes the intended treatment of ties explicit.RANK() OVER (ORDER BY score DESC), and has humans review only the top entries. Adding PARTITION BY date also returns the worst anomaly per day (see PARTITION BY in Q8).z-scores and day-over-day changes are strong for one large point anomaly, but they miss gradual drift: a slightly high value every day. Accumulating (running total) each day's deviation from the target reveals the buildup of small biases.
SUM(amount - 100) OVER ( ORDER BY dt ROWS UNBOUNDED PRECEDING -- = ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) -- Running total from the first row through the current row -- With only ORDER BY and no frame, the default is RANGE UNBOUNDED PRECEDING …
ORDER BY but no frame, its default is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. If multiple rows have the same dt, RANGE adds tied values together. Explicitly use ROWS UNBOUNDED PRECEDING for row-by-row accumulation (the same ROWS vs. RANGE issue as Q1).From daily_sales (with a daily target of 100), calculate each day's target deviation (daily_dev = amount − 100) and its cumulative value (cum_dev); assign drift_flag 'drift_alert' when cum_dev exceeds 20 and 'normal' otherwise. Return dt, amount, daily_dev, cum_dev, drift_flag in ascending dt order.
| dt | amount |
|---|---|
| 2024-01-01 | 100 |
| 2024-01-02 | 102 |
| 2024-01-03 | 101 |
| 2024-01-04 | 103 |
| 2024-01-05 | 105 |
| 2024-01-06 | 104 |
| 2024-01-07 | 106 |
| 2024-01-08 | 108 |
Every day is within a few percent of the previous day and looks normal in isolation. But every value remains above target, so the deviations accumulate.
Expected output (ascending dt):
| dt | amount | daily_dev | cum_dev | drift_flag |
|---|---|---|---|---|
| 2024-01-01 | 100 | 0 | 0 | normal |
| 2024-01-02 | 102 | +2 | 2 | normal |
| 2024-01-03 | 101 | +1 | 3 | normal |
| 2024-01-04 | 103 | +3 | 6 | normal |
| 2024-01-05 | 105 | +5 | 11 | normal |
| 2024-01-06 | 104 | +4 | 15 | normal |
| 2024-01-07 | 106 | +6 | 21 | drift_alert |
| 2024-01-08 | 108 | +8 | 29 | drift_alert |
Although no isolated value is anomalous, cumulative deviation reaches 21 on 01-07 and crosses the threshold of 20. This detects a sustained shift that point-anomaly detection misses.
SELECT dt, amount, amount - 100 AS daily_dev, -- Daily deviation from the target (100) SUM(amount - 100) OVER ( -- Cumulative deviation (running total) ORDER BY dt ROWS UNBOUNDED PRECEDING -- First row through current row ) AS cum_dev, CASE WHEN SUM(amount - 100) OVER (ORDER BY dt ROWS UNBOUNDED PRECEDING) > 20 THEN 'drift_alert' ELSE 'normal' END AS drift_flag FROM daily_sales ORDER BY dt; /* Execution order (logical SQL evaluation order): 1. FROM daily_sales → Read the rows 2. amount - 100 → Format the values 3. SUM(...) OVER (...) → Evaluate the window function (preserve row count) 4. CASE WHEN cum_dev > 20 → Format the values 5. SELECT / ORDER BY dt → Sort and return the output */
LEGEND
① FROM daily_sales (8 rows)
FROM daily_salesSales are slightly above the target of 100 every day. Day-over-day changes are within a few percent, so point-anomaly detection finds nothing.| dt | amount |
|---|---|
| 01-01 | 100 |
| 01-02 | 102 |
| 01-03 | 101 |
| 01-04 | 103 |
| 01-05 | 105 |
| 01-06 | 104 |
| 01-07 | 106 |
| 01-08 | 108 |
SUM(x) OVER (ORDER BY dt), with no explicit frame, accumulates from UNBOUNDED PRECEDING through CURRENT ROW. In contrast, SUM(x) OVER (), with neither a frame nor ORDER BY, is the total across all rows. State the intended behavior explicitly.amount - AVG(amount) OVER (ORDER BY dt ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) instead of using a fixed 100 yields cumulative deviation from a moving average, adapting to series whose trend changes.SUM(x) OVER () attaches the all-row total, not a cumulative value, to every row and hides drift. A cumulative sum always needs ORDER BY.cum_dev > 20 misses accumulated negative drift. Use ABS(cum_dev) or two-sided thresholds when downward movement must also be monitored.Measuring anomalies against a company-wide average fails when stores of different sizes are mixed. A high sale at a small store can appear normal against a large store's baseline, and vice versa. Measure anomalies within the same peer group. PARTITION BY splits a window by group, attaching each row's own group aggregate.
AVG(amount) OVER (PARTITION BY store_id) -- Average for each store STDDEV(amount) OVER (PARTITION BY store_id) -- Standard deviation for each store -- Unlike GROUP BY, rows are not collapsed; group aggregates run alongside each row
PARTITION BY store_id is verbose. Define it once with WINDOW w AS (PARTITION BY store_id), then reuse it as AVG(amount) OVER w (the same mechanism as the WINDOW clause in Q2).From store_sales, calculate each store's average and standard deviation, then calculate each row's z-score = (amount − store average) / store standard deviation; assign status 'anomaly' when |z| is at least 1.5 and 'normal' otherwise. Return store_id, dt, amount, store_avg, store_std, z_score, status, ordered by store_id and then dt. Round average, standard deviation, and z-score to two decimal places.
| store_id | dt | amount |
|---|---|---|
| A | 2024-01-01 | 100 |
| A | 2024-01-02 | 105 |
| A | 2024-01-03 | 95 |
| A | 2024-01-04 | 100 |
| A | 2024-01-05 | 300 |
| B | 2024-01-01 | 500 |
| B | 2024-01-02 | 510 |
| B | 2024-01-03 | 490 |
| B | 2024-01-04 | 505 |
| B | 2024-01-05 | 495 |
Store A is normally near 100, with 300 on 01-05 standing out. Store B is consistently near 500. Using the company-wide average (=320) makes A's ordinary values look abnormally low and makes A's 300 less conspicuous. The baseline must be within each store.
Expected output (ascending store_id, ascending dt):
| store_id | dt | amount | store_avg | store_std | z_score | status |
|---|---|---|---|---|---|---|
| A | 2024-01-01 | 100 | 140.00 | 89.51 | -0.45 | normal |
| A | 2024-01-02 | 105 | 140.00 | 89.51 | -0.39 | normal |
| A | 2024-01-03 | 95 | 140.00 | 89.51 | -0.50 | normal |
| A | 2024-01-04 | 100 | 140.00 | 89.51 | -0.45 | normal |
| A | 2024-01-05 | 300 | 140.00 | 89.51 | +1.79 | anomaly |
| B | 2024-01-01 | 500 | 500.00 | 7.91 | 0.00 | normal |
| B | 2024-01-02 | 510 | 500.00 | 7.91 | +1.26 | normal |
| B | 2024-01-03 | 490 | 500.00 | 7.91 | -1.26 | normal |
| B | 2024-01-04 | 505 | 500.00 | 7.91 | +0.63 | normal |
| B | 2024-01-05 | 495 | 500.00 | 7.91 | -0.63 | normal |
Under A's baseline (average 140, SD 89.5), only 01-05 (300) is anomalous at z=+1.79. B uses a different baseline (average 500, SD 7.9), so every row is normal. Each segment has its own anomaly scale.
SELECT store_id, dt, amount, ROUND(AVG(amount) OVER w, 2) AS store_avg, -- Average within the store ROUND(STDDEV(amount) OVER w, 2) AS store_std, -- Standard deviation within the store ROUND( (amount - AVG(amount) OVER w) / NULLIF(STDDEV(amount) OVER w, 0) -- Avoid division by zero when SD=0 , 2) AS z_score, CASE WHEN ABS( (amount - AVG(amount) OVER w) / NULLIF(STDDEV(amount) OVER w, 0) ) >= 1.5 THEN 'anomaly' ELSE 'normal' END AS status FROM store_sales WINDOW w AS (PARTITION BY store_id) -- Split and reuse the window for each store ORDER BY store_id, dt; /* Execution order (logical SQL evaluation order): 1. FROM store_sales → Read the rows 2. WINDOW w (PARTITION BY store_id) → Define the windows 3. AVG/STDDEV OVER w → Evaluate window functions (preserve row count) 4. (amount-avg)/NULLIF(std,0) → Calculate z-score 5. CASE WHEN ABS(z) is at least 1.5 → Decide anomaly status 6. SELECT / ORDER BY store_id, dt → Sort and return the output */
LEGEND
① FROM store_sales (10 rows)
FROM store_salesTwo stores of different sizes are mixed together. Store A is a small store near 100, while B is a large store near 500. A single baseline would hide anomalies behind this size difference.| store_id | dt | amount |
|---|---|---|
| A | 01-01 | 100 |
| A | 01-02 | 105 |
| A | 01-03 | 95 |
| A | 01-04 | 100 |
| A | 01-05 | 300 |
| B | 01-01 | 500 |
| B | 01-02 | 510 |
| B | 01-03 | 490 |
| B | 01-04 | 505 |
| B | 01-05 | 495 |
GROUP BY collapses rows and returns only aggregates, whereas PARTITION BY keeps every row and attaches its group aggregate. Anomaly detection that must retain the original detail needs the latter (Q9 covers GROUP BY).WINDOW w AS (...) and refer to it as AVG(...) OVER w. This centralizes the definition and prevents inconsistent PARTITION, ORDER, or frame changes.ORDER BY, the implicit frame is RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING, so average and SD cover the entire store. Adding ORDER BY dt changes the default to RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, producing a cumulative average instead of the store-wide average. Omit ORDER BY for full-window aggregates or explicitly specify the full frame.NULLIF(stddev, 0) so the result becomes NULL.PARTITION BY store_id for stores, PARTITION BY dow for weekdays, or PARTITION BY product_category for products. In practice, run several partition axes in parallel to see which context contains the deviation. Add ORDER BY dt and a ROWS BETWEEN ... frame to PARTITION BY store_id to naturally extend it to a store-level moving average (Q1).Duplicate records—such as the same transaction being counted twice—are one of the most common data-quality anomalies. A single row cannot reveal them; you must count how often the same key occurs. The essential tools are GROUP BY (collapse rows to aggregate) and HAVING (filter the aggregated result).
WHERE -- Filter individual rows before aggregation (COUNT cannot be used here) GROUP BY txn_id -- Collapse identical txn_id values into one group HAVING COUNT(*) > 1 -- Filter each group after aggregation
WHERE filters individual rows before grouping; HAVING filters aggregates such as COUNT and SUM after grouping. A key that occurs two or more times can only be identified after aggregation, so it belongs in HAVING.From txn_log, extract only transactions whose txn_id is recorded two or more times (duplicates), calculating their occurrence count and total amount. Return txn_id, occurrences, total_amount, ordered by descending occurrences and ascending txn_id.
| txn_id | dt | amount |
|---|---|---|
| T-101 | 2024-01-01 | 1200 |
| T-102 | 2024-01-01 | 800 |
| T-101 | 2024-01-02 | 1200 |
| T-103 | 2024-01-02 | 500 |
| T-104 | 2024-01-03 | 950 |
| T-102 | 2024-01-03 | 800 |
| T-101 | 2024-01-03 | 1200 |
T-101 occurs three times and T-102 twice, so both are duplicates. T-103 and T-104 occur once and are normal. The same transaction ID recorded at the same amount across several days suggests double counting.
Expected output (descending occurrences, ascending txn_id):
| txn_id | occurrences | total_amount |
|---|---|---|
| T-101 | 3 | 3600 |
| T-102 | 2 | 1600 |
T-103 and T-104, which occur only once, are removed by HAVING COUNT(*) > 1; only the two duplicates remain. total_amount is the amount counted more than once (the overstatement).
SELECT txn_id, COUNT(*) AS occurrences, -- Number of rows in the group = occurrence count SUM(amount) AS total_amount -- Amount total within the group FROM txn_log GROUP BY txn_id -- Collapse each txn_id into one group HAVING COUNT(*) > 1 -- After aggregation, keep only groups occurring twice or more ORDER BY occurrences DESC, txn_id; /* Execution order (logical SQL evaluation order): 1. FROM txn_log → Read the rows 2. (no WHERE) → Filter rows 3. GROUP BY txn_id → Group rows 4. COUNT(*) / SUM(amount) → Evaluate aggregate functions 5. HAVING COUNT(*) > 1 → Filter groups 6. SELECT / ORDER BY → Sort and return the output */
LEGEND
① FROM txn_log (7 rows)
FROM txn_logThere are seven transaction-log rows. Duplicates are not visible one row at a time. Counting how often each txn_id occurs is what reveals double counting.| txn_id | dt | amount |
|---|---|---|
| T-101 | 01-01 | 1200 |
| T-102 | 01-01 | 800 |
| T-101 | 01-02 | 1200 |
| T-103 | 01-02 | 500 |
| T-104 | 01-03 | 950 |
| T-102 | 01-03 | 800 |
| T-101 | 01-03 | 1200 |
WHERE filters individual rows before GROUP BY, while HAVING filters COUNT, SUM, and other aggregate values after GROUP BY. Occurrence count cannot be known before aggregation, so WHERE COUNT(*) > 1 is an error; HAVING is correct.PARTITION BY kept all rows, while GROUP BY aggregates each group into one row. Use GROUP BY when only a group property, such as the existence of duplicates, is needed; use PARTITION BY when detail must be retained for row-by-row comparison.txn_id values duplicates. To define a duplicate as the same ID on the same day, use a composite key such as GROUP BY txn_id, dt. The GROUP BY clause is the definition of what counts as a duplicate.WHERE COUNT(*) > 1 is evaluated before aggregation and produces a syntax error. Put aggregate filters in HAVING. Conversely, use WHERE for a date restriction when checking duplicates only within a certain period; both can be combined.SELECT txn_id, dt, COUNT(*) ... GROUP BY txn_id leaves dt indeterminate and errors in most databases (MySQL can silently return an arbitrary value, which is dangerous). SELECT only GROUP BY keys or aggregate functions.ROW_NUMBER() OVER (PARTITION BY txn_id ORDER BY dt) (combining Q6 and Q8) and delete rows other than rn = 1 as duplicates. Remember the division of responsibility: GROUP BY for detection, window functions for removal (deduplication).The final anomaly-detection step is to aggregate findings into alerts. “For each category, how many days were anomalous and what percentage of all days? Notify only categories with enough anomalous days.” COUNT(*) FILTER (WHERE ...) plus GROUP BY ... HAVING writes this combination of conditional counting and group filtering in one query (a synthesis of Q6–Q9).
COUNT(*) FILTER (WHERE amount > 200) -- Count only rows matching the condition COUNT(*) -- Count all rows (the denominator) -- Within one group, aggregate all rows and anomalies simultaneously
COUNT(*) FILTER (WHERE condition) counts only matching rows within a group. A table-level WHERE reduces the denominator too, whereas FILTER obtains both the denominator (all rows) and numerator (anomalies) in one query.From metrics, daily measurements by category, define amount > 200 as anomalous and aggregate each category's total days (total_days), anomalous days (anomaly_days), and anomaly rate percent (anomaly_pct). Keep only categories with at least two anomalous days as notification targets. Return category, total_days, anomaly_days, anomaly_pct, ordered by descending anomaly_days and ascending category. Round pct to one decimal place.
| category | dt | amount |
|---|---|---|
| web | 2024-01-01 | 100 |
| web | 2024-01-02 | 250 |
| web | 2024-01-03 | 300 |
| auth | 2024-01-01 | 260 |
| auth | 2024-01-02 | 240 |
| auth | 2024-01-03 | 220 |
| api | 2024-01-01 | 120 |
| api | 2024-01-02 | 280 |
| api | 2024-01-03 | 130 |
| cdn | 2024-01-01 | 90 |
| cdn | 2024-01-02 | 100 |
| cdn | 2024-01-03 | 95 |
Anomalies (amount>200) are web: 2 days, auth: 3, api: 1, and cdn: 0. Filtering to at least two anomalous days leaves only web and auth as notification targets.
Expected output (descending anomaly_days, ascending category):
| category | total_days | anomaly_days | anomaly_pct |
|---|---|---|---|
| auth | 3 | 3 | 100.0 |
| web | 3 | 2 | 66.7 |
api (one anomalous day) and cdn (zero) are excluded by HAVING anomaly_days >= 2. auth is anomalous every day and takes top priority, followed by web: a prioritized list of categories that need notification.
SELECT category, COUNT(*) AS total_days, -- Total days in the category (denominator) COUNT(*) FILTER (WHERE amount > 200) AS anomaly_days, -- Count only anomalous days ROUND( 100.0 * COUNT(*) FILTER (WHERE amount > 200) / COUNT(*) -- Anomaly rate = anomalous days / total days , 1) AS anomaly_pct FROM metrics GROUP BY category -- Aggregate by category HAVING COUNT(*) FILTER (WHERE amount > 200) >= 2 -- Notify only when at least two days are anomalous ORDER BY anomaly_days DESC, category; /* Execution order (logical SQL evaluation order): 1. FROM metrics → Read the rows 2. GROUP BY category → Group rows 3. COUNT(*) and COUNT(*) FILTER(...) → Evaluate aggregate functions 4. ROUND(100.0*anomaly/denominator,1) → Format the value 5. HAVING COUNT(*) FILTER(...) >= 2 → Filter groups 6. SELECT / ORDER BY anomaly_days DESC → Sort and return the output */
LEGEND
① FROM metrics (12 rows)
FROM metricsThere are three daily measurements for each of four categories. Define amount>200 as an anomaly. Raw detail does not show which category needs action, so aggregate by category.| category | dt | amount |
|---|---|---|
| web | 01-01 | 100 |
| web | 01-02 | 250 |
| web | 01-03 | 300 |
| auth | 01-01 | 260 |
| auth | 01-02 | 240 |
| auth | 01-03 | 220 |
| api | 01-01 | 120 |
| api | 01-02 | 280 |
| api | 01-03 | 130 |
| cdn | 01-01 | 90 |
| cdn | 01-02 | 100 |
| cdn | 01-03 | 95 |
WHERE amount > 200 leaves only anomalous rows even in the denominator, so no rate can be calculated. COUNT(*) FILTER (WHERE ...) sits beside total COUNT(*) and calculates the anomaly rate at the same time.100.0: COUNT(*) FILTER(...) / COUNT(*) can truncate to 0 when both operands are integers. Multiply by floating-point 100.0 first, or cast with CAST(... AS numeric), before dividing.HAVING anomaly_days >= 2 cannot reference SELECT's output alias. HAVING is logically evaluated before that alias is assigned, so repeat HAVING COUNT(*) FILTER (WHERE ...) >= 2. ORDER BY comes later, so its alias is allowed.... WHERE amount > 200 GROUP BY category aggregates only anomalous rows, loses the denominator, and makes every category appear 100%. Use FILTER (or CASE aggregation) when the denominator must remain.FILTER works in PostgreSQL and SQLite but not MySQL. Substitute SUM(CASE WHEN amount > 200 THEN 1 ELSE 0 END) or COUNT(CASE WHEN amount > 200 THEN 1 END) for the same behavior. Use CASE aggregation when portability is required.