SQL Batch Processing — Applied CTEs, Window Functions

ADVBatch ProcessingCTEWindow FunctionsRunning Totals and PivotsAPI PracticePostgreSQL/BigQuery Compatible5 questions
1 / 5 · In progress 0 / 5 completed
QUESTION 1

CTE + GROUP BY — Aggregate a Daily Sales Summary for an API

WITHCTEGROUP BYDaily Batch
Background

A CTE (Common Table Expression) is a temporary named query defined as WITH name AS (...). In batch processing, a standard pattern is to split the three stages—aggregation, filtering, and formatting—into CTEs.

WITH aggregate_name AS (
  -- ① Define the aggregation (GROUP BY) here
  SELECT col1, SUM(col2) AS total
  FROM   table_name
  GROUP BY col1         -- Aggregate rows for each col1 value
)
-- ② Filter and sort the aggregated result
SELECT *
FROM   aggregate_name
WHERE  total > 10000  -- A CTE lets WHERE reference the aggregated total
ORDER BY col1;          -- Sort using the default ASC order
Why use a CTE:You normally cannot write WHERE SUM(...) and must use HAVING. Moving the aggregation into a CTE lets you treat its result as an ordinary column. As batch SQL grows, naming and separating processing stages becomes increasingly valuable.
Problem

From the orders table below, calculate the total sales, order count, and average order amount for each order_date.

Return only dates whose total sales are at least 50,000 yen, sorted by total sales in descending order.

Source table
▸ orders
order_idorder_dateamount
12024-04-0120000
22024-04-0135000
32024-04-0260000
42024-04-0215000
52024-04-0380000
62024-04-0340000
72024-04-0412000
82024-04-0418000
Expected Output
order_datetotal_salesorder_countavg_order
2024-04-03120000260000
2024-04-0275000237500
2024-04-0155000227500
Model Answer
-- WITH: split processing into a CTE
WITH daily_stats AS (
  -- ① Aggregate orders by date
  SELECT
    order_date,
    SUM(amount)  AS total_sales,
    COUNT(*)     AS order_count,
    AVG(amount)  AS avg_order
  FROM orders
  GROUP BY order_date
)
-- ② Filter and sort the CTE result
SELECT
  order_date,
  total_sales,
  order_count,
  avg_order
FROM daily_stats
WHERE total_sales >= 50000
ORDER BY total_sales DESC;

/*
  Evaluation order:
  1. CTE daily_stats            → aggregate by date (SUM/COUNT/AVG)
  2. Outer WHERE                  → filter by the total_sales threshold
  3. ORDER BY total_sales DESC  → return rows in descending order
*/
Explanation (table transitions & key points)
WITH daily_stats AS ( SELECT order_date, SUM(amount) AS total_sales, COUNT(*) AS order_count, AVG(amount) AS avg_order FROM orders GROUP BY order_date ) SELECT order_date, total_sales, order_count, avg_order FROM daily_stats WHERE total_sales >= 50000 ORDER BY total_sales DESC;
LEGEND
Rows read / loaded
① FROM
FROM ordersRead all eight rows from orders. In the next step, GROUP BY collects rows by date.
1 / 5
order_idorder_dateamount
12024-04-0120,000
22024-04-0135,000
32024-04-0260,000
42024-04-0215,000
52024-04-0380,000
62024-04-0340,000
72024-04-0412,000
82024-04-0418,000
Read all 8 rows
LEARNING POINTS
Evaluation order:SQL is evaluated in the order FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY. Splitting processing into a CTE and an outer query makes this order easier to see.
COUNT(*) vs. COUNT(col):COUNT(*) counts every row, including rows containing NULL, while COUNT(col) excludes NULL values in that column. Use the former when counting row existence itself.
ANTI-PATTERNS
Aggregate function directly in WHERE:WHERE SUM(...) fails because of SQL evaluation order. Use HAVING or an outer WHERE around a CTE.
ORDER BY inside a CTE:Sorting inside a CTE does not guarantee output order. Apply ORDER BY in the outer SELECT.
Practical note
Organizing CTEs as a pipeline—1. raw-data processing, 2. master-data joins, 3. aggregation, and 4. formatting—lets you run each stage independently and greatly improves operational maintainability.
QUESTION 2

LAG() Window Function — Calculate Prior-Month Differences and Growth Rates

LAGWindow FunctionsMonth-over-MonthBatch Report
Background

LAG() is a window function that brings a value from one row earlier (or n rows earlier) onto the current row. It is essential for month-over-month and day-over-day calculations.

LAG(value_column, 1, 0) OVER (
  PARTITION BY group_column   -- Reference the previous row independently within each group
  ORDER BY     sort_column       -- This order determines which row is previous
) AS previous_value

The arguments are LAG(column, offset, default). The offset defaults to 1 (one row earlier), and the default value is used when no previous row exists, as on the first row.

OVER and PARTITION BY:OVER() lets a window function calculate values without collapsing rows. If PARTITION BY is omitted, all rows form one group.
Problem

From the monthly_sales table below, calculate each month’s previous-month sales, difference from the previous month, and month-over-month growth rate (%).

Source table
▸ monthly_sales
monthsales
2024-01100000
2024-02130000
2024-03120000
2024-04160000
2024-05145000
Expected Output
monthsalesprev_salesdiffgrowth_rate
2024-011000000NULLNULL
2024-02130000100000+3000030.00
2024-03120000130000-10000-7.69
2024-04160000120000+4000033.33
2024-05145000160000-15000-9.38
Model Answer
-- WITH: split processing into a CTE
WITH sales_with_prev AS (
  -- ① Use LAG to place previous-month sales on the current row
  SELECT
    month,
    sales,
    LAG(sales, 1, 0) OVER (
      ORDER BY month
    ) AS prev_sales
  FROM monthly_sales
)
-- ② Calculate the difference and growth rate
SELECT
  month,
  sales,
  prev_sales,
  CASE
    WHEN prev_sales = 0 THEN NULL
    ELSE sales - prev_sales
  END AS diff,
  CASE
    WHEN prev_sales = 0 THEN NULL
    ELSE ROUND((sales - prev_sales)::numeric / prev_sales * 100, 2)
  END AS growth_rate
FROM sales_with_prev
ORDER BY month;

/*
  Evaluation order:
  1. CTE sales_with_prev  → attach previous-month sales with LAG
  2. Outer CASE             → calculate the difference and growth rate
  3. ORDER BY month       → return rows in ascending month order
*/
Explanation (table transitions & key points)
WITH sales_with_prev AS ( SELECT month, sales, LAG(sales, 1, 0) OVER ( ORDER BY month ) AS prev_sales FROM monthly_sales ) SELECT month, sales, prev_sales, CASE WHEN prev_sales = 0 THEN NULL ELSE sales - prev_sales END AS diff, CASE WHEN prev_sales = 0 THEN NULL ELSE ROUND((sales - prev_sales)::numeric / prev_sales * 100, 2) END AS growth_rate FROM sales_with_prev ORDER BY month;
LEGEND
Rows read / loaded
① FROM
FROM monthly_salesRead all five rows from monthly_sales. In the next step, LAG adds the sales value from one row earlier to each row.
1 / 3
monthsales
2024-01100,000
2024-02130,000
2024-03120,000
2024-04160,000
2024-05145,000
Read all 5 rows
LEARNING POINTS
LAG default value:The third argument of LAG(sales, 1, 0) prevents NULL on the first row. This is useful when an API response should avoid NULL.
Integer division:Many databases truncate integer division. Cast explicitly with ::numeric or an equivalent type before calculating.
ANTI-PATTERNS
LAG without ORDER BY:The row order becomes undefined. Always specify ORDER BY.
Division by a zero denominator:Guard with CASE WHEN prev_sales = 0 THEN NULL to prevent division-by-zero errors.
Practical note
Month-over-month and week-over-week comparisons are core dashboard metrics. LAG(sales, 7), which helps remove day-of-week noise, is another common pattern.
QUESTION 3

SUM() OVER (ROWS BETWEEN) — Calculate a Running Sales Total and Moving Average

SUM OVERROWS BETWEENRunning TotalMoving Average
Background

The ROWS BETWEEN clause of a window function precisely specifies which range of rows to aggregate for each current row.

SUM(sales) OVER (
  ORDER BY month
  ROWS BETWEEN UNBOUNDED PRECEDING    -- From the first row
           AND CURRENT ROW            -- Through the current row → running total
) AS running_total

AVG(sales) OVER (
  ORDER BY month
  ROWS BETWEEN 2 PRECEDING           -- From two rows earlier
           AND CURRENT ROW            -- Through the current row → three-month moving average
) AS moving_avg_3m
Batch-processing use case:This pattern is common in time-series batches, such as identifying the month when cumulative sales exceeded a target. A window function is essential because it aggregates without removing rows.
Problem

From the monthly_sales table below, calculate each month’s running sales total (running_total) and moving average for the latest three months (moving_avg_3m).

Source table
▸ monthly_sales
monthsales
2024-0180000
2024-02120000
2024-03100000
2024-04150000
2024-05130000
2024-06170000
Expected Output
monthsalesrunning_totalmoving_avg_3m
2024-01800008000080000.0
2024-02120000200000100000.0
2024-03100000300000100000.0
2024-04150000450000123333.3
2024-05130000580000126666.7
2024-06170000750000150000.0
Model Answer
SELECT
  month,
  sales,
  SUM(sales) OVER (                                   -- ① Running sales total: sum from the first row through the current row
    ORDER BY month
    ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  ) AS running_total,
  ROUND(                                              -- ② Three-month moving average: average from two rows earlier through the current row
    AVG(sales) OVER (
      ORDER BY month
      ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
    )::numeric, 1
  ) AS moving_avg_3m
FROM monthly_sales
ORDER BY month;

/*
  Evaluation order:
  1. FROM monthly_sales  → read rows
  2. SUM() OVER (...)    → calculate the running sales total
  3. AVG() OVER (...)    → calculate the three-month moving average
  4. ORDER BY month      → return rows in ascending month order
*/
Explanation (table transitions & key points)
SELECT month, sales, SUM(sales) OVER ( ORDER BY month ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS running_total, ROUND( AVG(sales) OVER ( ORDER BY month ROWS BETWEEN 2 PRECEDING AND CURRENT ROW )::numeric, 1 ) AS moving_avg_3m FROM monthly_sales ORDER BY month;
LEGEND
Rows read / loaded
① FROM
FROM monthly_salesRead all six rows from monthly_sales. In the next step, SUM OVER adds the running total from the first row through the current row.
1 / 3
monthsales
2024-0180,000
2024-02120,000
2024-03100,000
2024-04150,000
2024-05130,000
2024-06170,000
Read all 6 rows
LEARNING POINTS
ROWS vs. RANGE:ROWS BETWEEN defines a window by physical row count, while RANGE BETWEEN uses a range of values. Prefer ROWS when its clearer and typically faster semantics match the requirement.
Practical use of moving averages:They smooth short-term trends and provide a foundation for anomaly detection.
ANTI-PATTERNS
Running total without ORDER BY:SUM() OVER() returns the total across all rows. A running total requires ORDER BY and ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW.
Practical note
Moving averages smooth noise from weekdays, campaigns, and similar effects to reveal trends. Seven-day (7MA) and 28-day (28MA) windows are standard choices.
QUESTION 4

ROW_NUMBER() + CTE — Build an API for Top-N Products by Category

ROW_NUMBERPARTITION BYTop NRanking API
Background

APIs often need only the top N rows in each category. A standard pattern assigns each row a rank within its group using ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...), isolates that result in a CTE, and then filters with WHERE rn <= N.

WITH ranked AS (
  SELECT *,
    ROW_NUMBER() OVER (
      PARTITION BY category        -- Number rows independently within each category
      ORDER BY     sales DESC, product_id  -- Break ties by product_id ascending
    ) AS rn
  FROM products
)
SELECT *
FROM   ranked
WHERE  rn <= 2;   -- Keep only the top two rows in each category
ROW_NUMBER vs. RANK vs. DENSE_RANK:ROW_NUMBER always assigns a unique sequence even when values tie. RANK gives ties the same rank and skips the next number (1, 1, 3), while DENSE_RANK does not skip it (1, 1, 2). ROW_NUMBER is usually easiest for extracting exactly N rows.
Problem

From the product_sales table below, extract the top two products by sales within each category.

Source table
▸ product_sales
product_idcategoryproduct_namesales
P01FoodApple85000
P02FoodBanana62000
P03FoodMandarin62000
P04FoodGrape41000
P05RefreshmentsGreen tea95000
P06RefreshmentsCoffee78000
P07RefreshmentsJuice78000
P08RefreshmentsWater55000
Expected Output
categoryproduct_namesalesrn
FoodApple850001
FoodBanana620002
RefreshmentsGreen tea950001
RefreshmentsCoffee780002
Model Answer
-- WITH: split processing into a CTE
WITH ranked_products AS (
  -- ① Rank sales within each category
  SELECT
    category,
    product_name,
    sales,
    ROW_NUMBER() OVER (
      PARTITION BY category
      ORDER BY sales DESC, product_id
    ) AS rn
  FROM product_sales
)
-- ② Extract the top two rows in each category
SELECT
  category,
  product_name,
  sales,
  rn
FROM ranked_products
WHERE rn <= 2
ORDER BY category, rn;

/*
  Evaluation order:
  1. CTE ranked_products    → rank sales within each category
  2. Outer WHERE rn <= 2      → extract the top two rows per category
  3. ORDER BY category, rn  → sort and return the result
*/
Explanation (table transitions & key points)
WITH ranked_products AS ( SELECT category, product_name, sales, ROW_NUMBER() OVER ( PARTITION BY category ORDER BY sales DESC, product_id ) AS rn FROM product_sales ) SELECT category, product_name, sales, rn FROM ranked_products WHERE rn <= 2 ORDER BY category, rn;
LEGEND
Rows read / loaded
① FROM
FROM product_salesRead all eight rows from product_sales. In the next step, PARTITION BY forms one group for each category.
1 / 5
product_idcategoryproduct_namesales
P01FoodApple85,000
P02FoodBanana62,000
P03FoodMandarin62,000
P04FoodGrape41,000
P05RefreshmentsGreen tea95,000
P06RefreshmentsCoffee78,000
P07RefreshmentsJuice78,000
P08RefreshmentsWater55,000
Read all 8 rows
LEARNING POINTS
Why the CTE is necessary:Window functions are evaluated after WHERE. Complete the window calculation in a CTE, then filter it in the outer query.
PARTITION BY:Without it, the rank covers the entire result. It is required for ranks within groups such as categories.
ANTI-PATTERNS
WHERE at the same query level:Directly using a window-function result in WHERE at the same SELECT level causes an error.
Practical note
Top N per category and selecting only the latest history row with ORDER BY created_at DESC are essential data-mart patterns.
QUESTION 5

CASE WHEN + SUM — Pivot Counts by Status

CASE WHENPivotSUM FILTERCross Tabulation
Background

Pivot aggregation, which expands row-oriented data into columns, is common in dashboard APIs. Because many SQL dialects lack a dedicated PIVOT statement, it can be implemented with CASE WHEN + SUM.

SELECT
  category,
  SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) AS completed,
  SUM(CASE WHEN status = 'pending' THEN 1 ELSE 0 END) AS pending
FROM   tasks
GROUP BY category;
PostgreSQL also supports FILTER:COUNT(*) FILTER (WHERE status = 'done') produces the same result. CASE WHEN is more portable because some databases, including BigQuery, do not support this FILTER syntax.
Problem

From the tasks table below, return one row per assignee containing counts by status (done, in progress, and not started) plus the total count.

Source table
▸ tasks
task_idassigneestatus
T01Bakerdone
T02Bakerdone
T03Bakerin progress
T04Bakernot started
T05Clarkdone
T06Clarkin progress
T07Clarkin progress
T08Adamsnot started
T09Adamsnot started
T10Adamsdone
Expected Output
assigneedonein_progressnot_startedtotal
Adams1023
Baker2114
Clark1203
Model Answer
SELECT
  assignee,
  SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) AS done,          -- ① Count done tasks
  SUM(CASE WHEN status = 'in progress' THEN 1 ELSE 0 END) AS in_progress,  -- ② Count tasks in progress
  SUM(CASE WHEN status = 'not started' THEN 1 ELSE 0 END) AS not_started,  -- ③ Count tasks not started
  COUNT(*) AS total                                                -- ④ Count all tasks
FROM tasks
GROUP BY assignee
ORDER BY assignee;

/*
  Evaluation order:
  1. FROM tasks         → read rows
  2. GROUP BY assignee  → group by assignee
  3. SUM(CASE...)       → aggregate counts by status (pivot)
  4. ORDER BY assignee  → return rows in ascending order
*/
Explanation (table transitions & key points)
SELECT assignee, SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) AS done, SUM(CASE WHEN status = 'in progress' THEN 1 ELSE 0 END) AS in_progress, SUM(CASE WHEN status = 'not started' THEN 1 ELSE 0 END) AS not_started, COUNT(*) AS total FROM tasks GROUP BY assignee ORDER BY assignee;
LEGEND
Rows read / loaded
① FROM
FROM tasksRead all ten rows from tasks. In the next step, GROUP BY groups the data by assignee.
1 / 5
task_idassigneestatus
T01Bakerdone
T02Bakerdone
T03Bakerin progress
T04Bakernot started
T05Clarkdone
T06Clarkin progress
T07Clarkin progress
T08Adamsnot started
T09Adamsnot started
T10Adamsdone
Read all 10 rows
LEARNING POINTS
Creating flags with CASE WHEN:SUM(CASE WHEN ... THEN 1 ELSE 0 END) is an idiom for counting rows that satisfy a condition.
Compression through pivoting:Expand row-oriented data into columns to build an aggregated API response table in one query.
ANTI-PATTERNS
Omitting ELSE:A nonmatching condition returns NULL. Specify ELSE 0 to prevent unintended behavior.
Practical note
Pivoting in the database can substantially reduce data transfer and application memory use. It is effective for dashboard aggregation over large logs.