The first step in anomaly detection is to establish a baseline for the normal range. Instead of using a simple average over the entire period, a moving average over the most recent N days provides a dynamic baseline that accounts for trends.
AVG(amount) OVER ( ORDER BY dt ROWS BETWEEN 2 PRECEDING AND CURRENT ROW ) -- ROWS BETWEEN 2 PRECEDING AND CURRENT ROW -- = current row + two preceding rows → the latest three rows (three days) -- The first two rows have smaller denominators because fewer rows are available
ROWS BETWEEN defines the frame by the physical number of rows. RANGE BETWEEN defines it by differences between values, so its behavior changes when multiple rows share the same dt. Row-based ROWS BETWEEN is generally used for time-series anomaly detection.From the daily_sales table, calculate each date's sales amount and the moving average over the most recent three days, including the current day, as moving_avg. Return dt, amount, moving_avg in ascending dt order. Round moving_avg to one decimal place.
| dt | amount |
|---|---|
| 2024-01-01 | 100 |
| 2024-01-02 | 120 |
| 2024-01-03 | 110 |
| 2024-01-04 | 130 |
| 2024-01-05 | 500 |
| 2024-01-06 | 115 |
| 2024-01-07 | 125 |
The value 500 on 2024-01-05 is an obvious anomaly (a spike). The moving average reveals the surrounding "normal band."
Expected output (ascending dt):
| dt | amount | moving_avg |
|---|---|---|
| 2024-01-01 | 100 | 100.0 |
| 2024-01-02 | 120 | 110.0 |
| 2024-01-03 | 110 | 110.0 |
| 2024-01-04 | 130 | 120.0 |
| 2024-01-05 | 500 | 246.7 |
| 2024-01-06 | 115 | 248.3 |
| 2024-01-07 | 125 | 246.7 |
Because only one or two rows are available for the first two frames, their averages use one and two rows, respectively.
SELECT dt, amount, ROUND( AVG(amount) OVER ( -- Window aggregate function ORDER BY dt -- Sort by date to determine the frame ROWS BETWEEN 2 PRECEDING -- From two rows before AND CURRENT ROW -- Through the current row = latest three rows ), 1 -- Round to one decimal place ) AS moving_avg FROM daily_sales ORDER BY dt; /* Execution order (logical SQL evaluation order): 1. FROM daily_sales → Read the rows 2. AVG(amount) OVER (...) → Evaluate the window function (preserve row count) 3. ROUND(..., 1) → Format the value 4. SELECT → Evaluate the columns (moving_avg) 5. ORDER BY dt → Sort and return the output */
LEGEND
① FROM daily_sales (7 rows)
FROM daily_salesRead the seven rows from daily_sales. The amount=500 on 2024-01-05 is an obvious spike. The goal is to quantify how far it lies outside the normal pattern with a moving average.| dt | amount |
|---|---|
| 01-01 | 100 |
| 01-02 | 120 |
| 01-03 | 110 |
| 01-04 | 130 |
| 01-05 | 500 |
| 01-06 | 115 |
| 01-07 | 125 |
ROWS BETWEEN 6 PRECEDING AND CURRENT ROW produces a seven-day moving average, while ROWS BETWEEN 29 PRECEDING AND CURRENT ROW produces a 30-day moving average. Changing N alone adjusts the degree of smoothing.AVG FILTER that excludes outliers.RANGE BETWEEN 2 PRECEDING AND CURRENT ROW means that dt differs from the current row's value by no more than 2. When dates are duplicated, unintended rows may enter the frame. Always use ROWS BETWEEN for a row-count-based frame.GROUP BY dt and OVER (ORDER BY dt) appear in the same query, GROUP BY executes first. For a table with multiple events per day, create a daily aggregate CTE before applying the window function.moving_avg and moving_stddev as features is standard practice.A moving average alone does not tell you how much variation is normal. Combining it with a moving standard deviation, moving_stddev, creates a dynamic threshold band: normal range = moving average ± k × moving standard deviation.
STDDEV(amount) OVER ( ORDER BY dt ROWS BETWEEN 2 PRECEDING AND CURRENT ROW ) -- STDDEV = sample standard deviation (denominator N-1) -- A one-row frame → NULL (standard deviation requires at least two points) -- STDDEV_POP = population standard deviation (denominator N)
STDDEV() is the unbiased sample standard deviation with Bessel's correction. A one-row frame makes the denominator zero, so it returns NULL. It is important either to replace NULL with 0 using COALESCE(STDDEV(...), 0) or to exclude the first row from the decision logic.From the daily_sales table, calculate each date's sales amount, the moving average over the latest three days as moving_avg, and the moving standard deviation as moving_stddev. Return dt, amount, moving_avg, moving_stddev in ascending dt order. Round the results to two decimal places.
| dt | amount |
|---|---|
| 2024-01-01 | 100 |
| 2024-01-02 | 120 |
| 2024-01-03 | 110 |
| 2024-01-04 | 130 |
| 2024-01-05 | 500 |
| 2024-01-06 | 115 |
| 2024-01-07 | 125 |
Expected output (ascending dt):
| dt | amount | moving_avg | moving_stddev |
|---|---|---|---|
| 2024-01-01 | 100 | 100.00 | NULL |
| 2024-01-02 | 120 | 110.00 | 14.14 |
| 2024-01-03 | 110 | 110.00 | 10.00 |
| 2024-01-04 | 130 | 120.00 | 10.00 |
| 2024-01-05 | 500 | 246.67 | 219.62 |
| 2024-01-06 | 115 | 248.33 | 218.08 |
| 2024-01-07 | 125 | 246.67 | 219.45 |
dt=2024-01-01 is NULL because the STDDEV frame contains only one row. Compared with stddev≈10 during the normal period, it expands more than twentyfold to ≈220 during the spike period.
SELECT dt, amount, ROUND(AVG(amount) OVER w, 2) AS moving_avg, ROUND(STDDEV(amount) OVER w, 2) AS moving_stddev -- Sample standard deviation (N-1) FROM daily_sales WINDOW w AS ( -- Share a named window definition ORDER BY dt ROWS BETWEEN 2 PRECEDING AND CURRENT ROW ) ORDER BY dt; /* Execution order (logical SQL evaluation order): 1. FROM daily_sales → Read the rows 2. WINDOW w AS (...) → Define the named window 3. AVG(amount) OVER w → Evaluate the window function (preserve row count) 4. STDDEV(amount) OVER w → Evaluate the window function (preserve row count) 5. ROUND(..., 2) → Format the values 6. SELECT / ORDER BY dt → Evaluate the columns, sort, and return the output */
LEGEND
① FROM daily_sales (7 rows)
FROM daily_salesUse the same table as Q1. This time, calculate variability (standard deviation) as well as the moving average to express the lower and upper bounds of the normal range numerically.| dt | amount |
|---|---|
| 01-01 | 100 |
| 01-02 | 120 |
| 01-03 | 110 |
| 01-04 | 130 |
| 01-05 | 500 |
| 01-06 | 115 |
| 01-07 | 125 |
OVER (ORDER BY dt ROWS BETWEEN ...) is redundant and error-prone. Define it once with WINDOW w AS (...) and reference it with OVER w, so changing the frame requires an edit in only one place.STDDEV, an alias for STDDEV_SAMP, is the unbiased sample standard deviation with denominator N-1. Use it when estimating a population from a sample. STDDEV_POP uses denominator N and applies when the observed values constitute the entire population, such as all transactions from one day. By convention, anomaly detection commonly uses STDDEV.(amount - avg) / stddev then becomes NULL, preventing the anomaly flag from being set correctly. Guard with NULLIF(STDDEV(...), 0) or COALESCE, or exclude the first row with WHERE.amount / 0 and raise an error. Always use NULLIF(STDDEV(...), 0) so that a zero denominator becomes NULL.A z-score, or standardized score, is a dimensionless measure of how many standard deviations the current value lies from the average. In anomaly detection, |z-score| > 2–3 is commonly used as an anomaly threshold.
-- Definition of a z-score z = (current value - average) / standard deviation -- SQL implementation based on a moving window (amount - AVG(amount) OVER w) / NULLIF(STDDEV(amount) OVER w, 0) -- Guard division by zero with NULL -- |z| > 2 → outer 4.6% of a normal distribution → investigate -- |z| > 3 → outer 0.3% of a normal distribution → rule-of-thumb anomaly
NULLIF(x, 0) returns NULL when x is 0. Simply wrap the denominator of a potentially zero-dividing expression in NULLIF to prevent a division-by-zero error. Because any value divided by NULL yields NULL, a subsequent CASE WHEN can assign the flag.From the daily_sales table, use the moving average and moving standard deviation over the latest five days to calculate each day's z-score, then assign anomaly_flag as 'anomaly' when |z| ≥ 2 and 'normal' otherwise. Return dt, amount, moving_avg, moving_stddev, z_score, anomaly_flag in ascending dt order. Round z_score to two decimal places.
| dt | amount |
|---|---|
| 2024-01-01 | 100 |
| 2024-01-02 | 105 |
| 2024-01-03 | 98 |
| 2024-01-04 | 102 |
| 2024-01-05 | 110 |
| 2024-01-06 | 400 |
| 2024-01-07 | 108 |
| 2024-01-08 | 103 |
Expected output (ascending dt):
| dt | amount | moving_avg | moving_stddev | z_score | anomaly_flag |
|---|---|---|---|---|---|
| 2024-01-01 | 100 | 100.00 | NULL | NULL | normal |
| 2024-01-02 | 105 | 102.50 | 3.54 | 0.71 | normal |
| 2024-01-03 | 98 | 101.00 | 3.61 | -0.83 | normal |
| 2024-01-04 | 102 | 101.25 | 2.99 | 0.25 | normal |
| 2024-01-05 | 110 | 103.00 | 4.69 | 1.49 | normal |
| 2024-01-06 | 400 | 163.00 | 132.56 | 1.79 | normal |
| 2024-01-07 | 108 | 163.60 | 132.24 | -0.42 | normal |
| 2024-01-08 | 103 | 164.60 | 131.64 | -0.47 | normal |
While the spike remains in the five-day window, stddev expands sharply, making it difficult for z_score to reach 2. This is the spike-contamination problem. In practice, a LAG-style frame that uses the window before the spike is effective; see the explanation.
WITH stats AS ( SELECT dt, amount, ROUND(AVG(amount) OVER w, 2) AS moving_avg, ROUND(STDDEV(amount) OVER w, 2) AS moving_stddev FROM daily_sales WINDOW w AS ( ORDER BY dt ROWS BETWEEN 4 PRECEDING AND CURRENT ROW -- Latest five days ) ) SELECT dt, amount, moving_avg, moving_stddev, ROUND( (amount - moving_avg) / NULLIF(moving_stddev, 0), -- Convert division by zero to NULL 2 ) AS z_score, CASE WHEN ABS( (amount - moving_avg) / NULLIF(moving_stddev, 0) ) >= 2 THEN 'anomaly' ELSE 'normal' END AS anomaly_flag FROM stats ORDER BY dt; /* Execution order (logical SQL evaluation order): 1. Define CTE stats → Evaluate window functions (preserve row count) 2. Evaluate outer query → Evaluate columns (z_score, anomaly_flag) 3. ORDER BY dt → Sort and return the output */
LEGEND
① FROM daily_sales + WINDOW (8 rows)
CTE stats: FROM + AVG/STDDEV OVER wRead the eight rows from daily_sales.| dt | amount |
|---|---|
| 01-01 | 100 |
| 01-02 | 105 |
| 01-03 | 98 |
| 01-04 | 102 |
| 01-05 | 110 |
| 01-06 | 400 |
| 01-07 | 108 |
| 01-08 | 103 |
ROWS BETWEEN 5 PRECEDING AND 1 PRECEDING, the five preceding days excluding the current day, so the current value is evaluated against the prior normal range.STDDEV = NULL because denominator N-1=0. Second, when every value in the frame is identical, STDDEV = 0. NULLIF(STDDEV(...), 0) safely guards both cases, producing NULL.SELECT (amount - AVG(amount) OVER w) / NULLIF(STDDEV(amount) OVER w, 0) produces the same result, but the identical expression must then be repeated in CASE WHEN. Make it a habit to calculate stats in a CTE first and reference them from the outer query.ROWS BETWEEN N PRECEDING AND 1 PRECEDING, evaluating today's value against the normal range through yesterday. A common design runs this query daily in a dbt (data build tool) or Apache Airflow batch and sends rows where anomaly_flag = 'anomaly' to Slack or email. Start with this simple z-score baseline before moving to a more complex model.The interquartile range, or IQR, is a robust anomaly-detection method that resists outliers. While a z-score is contaminated by the outlier itself, IQR uses only the middle 50% of the data, enabling it to calculate a stable threshold even for real-world data with many spikes.
-- Calculate the first quartile (Q1) and third quartile (Q3) PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY amount) AS q1 PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY amount) AS q3 -- IQR = Q3 - Q1 -- Outlier bounds (Tukey's fences) -- Lower bound: Q1 - 1.5 * IQR -- Upper bound: Q3 + 1.5 * IQR
WITHIN GROUP (ORDER BY col). PERCENTILE_CONT(0.5) is equivalent to the median. PERCENTILE_DISC, the discrete form, returns an observed value, while PERCENTILE_CONT, the continuous form, returns a linearly interpolated value.From the daily_sales table, calculate Q1, Q3, and IQR over the entire period, then assign outlier_flag as 'outlier' when an amount lies outside [Q1 − 1.5 × IQR, Q3 + 1.5 × IQR] and 'normal' when it lies inside. Return dt, amount, q1, q3, iqr, lower_fence, upper_fence, outlier_flag in ascending dt order.
| dt | amount |
|---|---|
| 2024-01-01 | 100 |
| 2024-01-02 | 105 |
| 2024-01-03 | 98 |
| 2024-01-04 | 102 |
| 2024-01-05 | 110 |
| 2024-01-06 | 400 |
| 2024-01-07 | 108 |
| 2024-01-08 | 103 |
Expected output (ascending dt):
| dt | amount | q1 | q3 | iqr | lower_fence | upper_fence | outlier_flag |
|---|---|---|---|---|---|---|---|
| 2024-01-01 | 100 | 101.50 | 108.50 | 7.00 | 91.00 | 119.00 | normal |
| 2024-01-02 | 105 | 101.50 | 108.50 | 7.00 | 91.00 | 119.00 | normal |
| 2024-01-03 | 98 | 101.50 | 108.50 | 7.00 | 91.00 | 119.00 | normal |
| 2024-01-04 | 102 | 101.50 | 108.50 | 7.00 | 91.00 | 119.00 | normal |
| 2024-01-05 | 110 | 101.50 | 108.50 | 7.00 | 91.00 | 119.00 | normal |
| 2024-01-06 | 400 | 101.50 | 108.50 | 7.00 | 91.00 | 119.00 | outlier |
| 2024-01-07 | 108 | 101.50 | 108.50 | 7.00 | 91.00 | 119.00 | normal |
| 2024-01-08 | 103 | 101.50 | 108.50 | 7.00 | 91.00 | 119.00 | normal |
q1, q3, and iqr have the same value on every row because they are aggregates over the entire period. The value 400 is an outlier because it greatly exceeds upper_fence=119. Although the spike itself affects Q3, its influence is limited.
WITH fences AS ( SELECT PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY amount) AS q1, PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY amount) AS q3 FROM daily_sales ) SELECT s.dt, s.amount, f.q1, f.q3, f.q3 - f.q1 AS iqr, f.q1 - 1.5 * (f.q3 - f.q1) AS lower_fence, -- Tukey's lower fence f.q3 + 1.5 * (f.q3 - f.q1) AS upper_fence, -- Tukey's upper fence CASE WHEN s.amount < f.q1 - 1.5 * (f.q3 - f.q1) OR s.amount > f.q3 + 1.5 * (f.q3 - f.q1) THEN 'outlier' ELSE 'normal' END AS outlier_flag FROM daily_sales s CROSS JOIN fences f -- Join the same q1/q3 to every row (one-row CTE) ORDER BY s.dt; /* Execution order (logical SQL evaluation order): 1. Define CTE fences → Evaluate aggregate functions (Q1, Q3) 2. FROM daily_sales s → Read the rows 3. CROSS JOIN fences f → Join them (Cartesian product) 4. SELECT → Evaluate columns (lower_fence, upper_fence, flag) 5. ORDER BY s.dt → Sort and return the output */
LEGEND
① CTE fences — Sort ascending and prepare to aggregate
WITHIN GROUP (ORDER BY amount)Sort all eight daily_sales rows in ascending order. The 25th percentile, Q1, lies between the second and third values; the 75th percentile, Q3, lies between the sixth and seventh values.| Rank | dt | amount (ascending) |
|---|---|---|
| 1 | 01-03 | 98 |
| 2 | 01-01 | 100 |
| 3 | 01-04 | 102 |
| 4 | 01-08 | 103 |
| 5 | 01-02 | 105 |
| 6 | 01-07 | 108 |
| 7 | 01-05 | 110 |
| 8 | 01-06 | 400 |
PERCENTILE_CONT linearly interpolates between two adjacent values and can return a non-integer such as 101.25. PERCENTILE_DISC returns an observed value such as 102. As a rule, use PERCENTILE_CONT for continuous quantities such as sales and prices, and PERCENTILE_DISC for discrete quantities such as ranks and rating scores.PERCENTILE_CONT(0.25) WITHIN GROUP (...) is an aggregate that collapses an entire group into one row. A windowed form such as PERCENTILE_CONT(0.25) WITHIN GROUP (...) OVER (PARTITION BY ...) is also possible, but omitting WITHIN GROUP's ORDER BY causes an error.Production anomaly detection must monitor not only absolute magnitude but also the percentage change from the previous day, or rate of change. A change from 100 → 500 is a +400% spike: the percentage change is large even when the absolute value is small. Conversely, a large absolute value can be normal when it remains stable.
-- Calculate day-over-day percentage change (amount - LAG(amount) OVER (ORDER BY dt)) / NULLIF(LAG(amount) OVER (ORDER BY dt), 0) * 100.0 -- Convert to a percentage -- If the previous value is 0, NULLIF converts it to NULL → prevents division by zero -- * 100.0 (a floating-point value) prevents integer division
LAG, a past reference, for day-over-day change and LEAD, a future reference, for change against the following day. The standard formula is rate of change = (current − previous) / |previous| × 100. To detect a sharp drop, either take the absolute value with ABS() or define a separate negative threshold, such as −50% or lower.From the daily_sales table, calculate each day's percentage change from the previous day as pct_change, then assign alert_type as 'spike' for a change of +50% or more, 'drop' for −30% or less, and 'normal' otherwise. Return dt, amount, prev_amount, pct_change, alert_type in ascending dt order. Round pct_change to one decimal place.
| dt | amount |
|---|---|
| 2024-01-01 | 100 |
| 2024-01-02 | 110 |
| 2024-01-03 | 105 |
| 2024-01-04 | 108 |
| 2024-01-05 | 400 |
| 2024-01-06 | 60 |
| 2024-01-07 | 112 |
| 2024-01-08 | 115 |
01-05: +270% spike. 01-06: −85% drop. 01-07: +87% spike.
Expected output (ascending dt):
| dt | amount | prev_amount | pct_change | alert_type |
|---|---|---|---|---|
| 2024-01-01 | 100 | NULL | NULL | normal |
| 2024-01-02 | 110 | 100 | 10.0 | normal |
| 2024-01-03 | 105 | 110 | -4.5 | normal |
| 2024-01-04 | 108 | 105 | 2.9 | normal |
| 2024-01-05 | 400 | 108 | 270.4 | spike |
| 2024-01-06 | 60 | 400 | -85.0 | drop |
| 2024-01-07 | 112 | 60 | 86.7 | spike |
| 2024-01-08 | 115 | 112 | 2.7 | normal |
WITH with_prev AS ( SELECT dt, amount, LAG(amount) OVER (ORDER BY dt) AS prev_amount -- Get the previous day's amount FROM daily_sales ) SELECT dt, amount, prev_amount, ROUND( (amount - prev_amount) / NULLIF(prev_amount, 0) * 100.0, -- Percentage change (division-by-zero guard) 1 ) AS pct_change, CASE WHEN (amount - prev_amount) / NULLIF(prev_amount, 0) >= 0.5 -- +50% or more THEN 'spike' WHEN (amount - prev_amount) / NULLIF(prev_amount, 0) <= -0.3 -- -30% or less THEN 'drop' ELSE 'normal' END AS alert_type FROM with_prev ORDER BY dt; /* Execution order (logical SQL evaluation order): 1. Define CTE with_prev → Evaluate the window function (preserve row count) 2. Evaluate outer query → Evaluate columns (pct_change, flag) 3. ORDER BY dt → Sort and return the output */
LEGEND
① FROM daily_sales (8 rows)
FROM daily_salesRead the eight rows from daily_sales.| dt | amount |
|---|---|
| 01-01 | 100 |
| 01-02 | 110 |
| 01-03 | 105 |
| 01-04 | 108 |
| 01-05 | 400 |
| 01-06 | 60 |
| 01-07 | 112 |
| 01-08 | 115 |
segment_id column, use LAG(amount) OVER (PARTITION BY segment_id ORDER BY dt) so the previous-day reference does not cross segment boundaries. PARTITION BY creates a wall between segments and guarantees independent rate-of-change calculations.108 / 108 performs integer division and produces 1. * 100.0 forces a cast to floating point, which is necessary to obtain decimal values such as 0.2963.... Casting with ::float or ::numeric has the same effect.LAG(amount) OVER (ORDER BY dt) orders every row as one sequence even when categories are mixed. The row after category A's final row can be category B's first row and flow into prev_amount, producing a meaningless rate of change. Always add PARTITION BY category when there are multiple categories.CASE WHEN pct >= 0.5 THEN 'spike' WHEN pct <= -0.3 THEN 'drop', +270% is captured by the first WHEN. If the spike and drop definitions overlap—for example, both use |pct| >= 0.5—the earlier WHEN wins. Design the condition order to reflect your intent.WHERE alert_type != 'normal' to a Slack webhook, PagerDuty, or a Datadog monitor, and the data team no longer needs to inspect a dashboard manually every morning. A layered anomaly-detection pipeline combining rate of change, z-score, and IQR is a standard architecture for production data platforms.