SQL Batch Processing — Applied CTEs, Window Functions

ADVBatch ProcessingCTEWindow FunctionsRunning Totals and PivotsAPI PracticePostgreSQL/BigQuery Compatible5 questions
1 / 5 · In progress 0 / 5 completed
QUESTION 1

CTE + GROUP BY — Aggregate a Daily Sales Summary for an API

WITHCTEGROUP BYDaily Batch
Background ▼

A CTE (Common Table Expression) is a temporary named query defined as WITH name AS (...). In batch processing, a standard pattern is to split the three stages—aggregation, filtering, and formatting—into CTEs.

WITH aggregate_name AS (
  -- ① Define the aggregation (GROUP BY) here
  SELECT col1, SUM(col2) AS total
  FROM   table_name
  GROUP BY col1         -- Aggregate rows for each col1 value
)
-- ② Filter and sort the aggregated result
SELECT *
FROM   aggregate_name
WHERE  total > 10000  -- A CTE lets WHERE reference the aggregated total
ORDER BY col1;          -- Sort using the default ASC order
Why use a CTE:You normally cannot write WHERE SUM(...) and must use HAVING. Moving the aggregation into a CTE lets you treat its result as an ordinary column. As batch SQL grows, naming and separating processing stages becomes increasingly valuable.
Problem ▼

From the orders table below, calculate the total sales, order count, and average order amount for each order_date.

Return only dates whose total sales are at least 50,000 yen, sorted by total sales in descending order.

Source table
▸ orders
order_idorder_dateamount
12024-04-0120000
22024-04-0135000
32024-04-0260000
42024-04-0215000
52024-04-0380000
62024-04-0340000
72024-04-0412000
82024-04-0418000
Expected Output ▼
order_datetotal_salesorder_countavg_order
2024-04-03120000260000
2024-04-0275000237500
2024-04-0155000227500
Model Answer ▼
-- WITH: split processing into a CTE
WITH daily_stats AS (
  -- ① Aggregate orders by date
  SELECT
    order_date,
    SUM(amount)  AS total_sales,
    COUNT(*)     AS order_count,
    AVG(amount)  AS avg_order
  FROM orders
  GROUP BY order_date
)
-- ② Filter and sort the CTE result
SELECT
  order_date,
  total_sales,
  order_count,
  avg_order
FROM daily_stats
WHERE total_sales >= 50000
ORDER BY total_sales DESC;

/*
  Evaluation order:
  1. CTE daily_stats            → aggregate by date (SUM/COUNT/AVG)
  2. Outer WHERE                  → filter by the total_sales threshold
  3. ORDER BY total_sales DESC  → return rows in descending order
*/
Explanation (table transitions & key points) ▼
WITH daily_stats AS ( SELECT order_date, SUM(amount) AS total_sales, COUNT(*) AS order_count, AVG(amount) AS avg_order FROM orders GROUP BY order_date ) SELECT order_date, total_sales, order_count, avg_order FROM daily_stats WHERE total_sales >= 50000 ORDER BY total_sales DESC;
LEGEND ▼
Rows read / loaded
① FROM ▼
FROM ordersRead all eight rows from orders. In the next step, GROUP BY collects rows by date.
1 / 5
order_idorder_dateamount
12024-04-0120,000
22024-04-0135,000
32024-04-0260,000
42024-04-0215,000
52024-04-0380,000
62024-04-0340,000
72024-04-0412,000
82024-04-0418,000
Read all 8 rows
LEARNING POINTS
Evaluation order:SQL is evaluated in the order FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY. Splitting processing into a CTE and an outer query makes this order easier to see.
COUNT(*) vs. COUNT(col):COUNT(*) counts every row, including rows containing NULL, while COUNT(col) excludes NULL values in that column. Use the former when counting row existence itself.
ANTI-PATTERNS
Aggregate function directly in WHERE:WHERE SUM(...) fails because of SQL evaluation order. Use HAVING or an outer WHERE around a CTE.
ORDER BY inside a CTE:Sorting inside a CTE does not guarantee output order. Apply ORDER BY in the outer SELECT.
Practical note
Organizing CTEs as a pipeline—1. raw-data processing, 2. master-data joins, 3. aggregation, and 4. formatting—lets you run each stage independently and greatly improves operational maintainability.
QUESTION 2

LAG() Window Function — Calculate Prior-Month Differences and Growth Rates

LAGWindow FunctionsMonth-over-MonthBatch Report
Background ▼

LAG() is a window function that brings a value from one row earlier (or n rows earlier) onto the current row. It is essential for month-over-month and day-over-day calculations.

LAG(value_column, 1, 0) OVER (
  PARTITION BY group_column   -- Reference the previous row independently within each group
  ORDER BY     sort_column       -- This order determines which row is previous
) AS previous_value

The arguments are LAG(column, offset, default). The offset defaults to 1 (one row earlier), and the default value is used when no previous row exists, as on the first row.

OVER and PARTITION BY:OVER() lets a window function calculate values without collapsing rows. If PARTITION BY is omitted, all rows form one group.
Problem ▼

From the monthly_sales table below, calculate each month’s previous-month sales, difference from the previous month, and month-over-month growth rate (%).

Source table
▸ monthly_sales
monthsales
2024-01100000
2024-02130000
2024-03120000
2024-04160000
2024-05145000
Expected Output ▼
monthsalesprev_salesdiffgrowth_rate
2024-011000000NULLNULL
2024-02130000100000+3000030.00
2024-03120000130000-10000-7.69
2024-04160000120000+4000033.33
2024-05145000160000-15000-9.38
Model Answer ▼
-- WITH: split processing into a CTE
WITH sales_with_prev AS (
  -- ① Use LAG to place previous-month sales on the current row
  SELECT
    month,
    sales,
    LAG(sales, 1, 0) OVER (
      ORDER BY month
    ) AS prev_sales
  FROM monthly_sales
)
-- ② Calculate the difference and growth rate
SELECT
  month,
  sales,
  prev_sales,
  CASE
    WHEN prev_sales = 0 THEN NULL
    ELSE sales - prev_sales
  END AS diff,
  CASE
    WHEN prev_sales = 0 THEN NULL
    ELSE ROUND((sales - prev_sales)::numeric / prev_sales * 100, 2)
  END AS growth_rate
FROM sales_with_prev
ORDER BY month;

/*
  Evaluation order:
  1. CTE sales_with_prev  → attach previous-month sales with LAG
  2. Outer CASE             → calculate the difference and growth rate
  3. ORDER BY month       → return rows in ascending month order
*/
Explanation (table transitions & key points) ▼
WITH sales_with_prev AS ( SELECT month, sales, LAG(sales, 1, 0) OVER ( ORDER BY month ) AS prev_sales FROM monthly_sales ) SELECT month, sales, prev_sales, CASE WHEN prev_sales = 0 THEN NULL ELSE sales - prev_sales END AS diff, CASE WHEN prev_sales = 0 THEN NULL ELSE ROUND((sales - prev_sales)::numeric / prev_sales * 100, 2) END AS growth_rate FROM sales_with_prev ORDER BY month;
LEGEND ▼
Rows read / loaded
① FROM ▼
FROM monthly_salesRead all five rows from monthly_sales. In the next step, LAG adds the sales value from one row earlier to each row.
1 / 3
monthsales
2024-01100,000
2024-02130,000
2024-03120,000
2024-04160,000
2024-05145,000
Read all 5 rows
LEARNING POINTS
LAG default value:The third argument of LAG(sales, 1, 0) prevents NULL on the first row. This is useful when an API response should avoid NULL.
Integer division:Many databases truncate integer division. Cast explicitly with ::numeric or an equivalent type before calculating.
ANTI-PATTERNS
LAG without ORDER BY:The row order becomes undefined. Always specify ORDER BY.
Division by a zero denominator:Guard with CASE WHEN prev_sales = 0 THEN NULL to prevent division-by-zero errors.
Practical note
Month-over-month and week-over-week comparisons are core dashboard metrics. LAG(sales, 7), which helps remove day-of-week noise, is another common pattern.
QUESTION 3

SUM() OVER (ROWS BETWEEN) — Calculate a Running Sales Total and Moving Average

SUM OVERROWS BETWEENRunning TotalMoving Average
Background ▼

The ROWS BETWEEN clause of a window function precisely specifies which range of rows to aggregate for each current row.

SUM(sales) OVER (
  ORDER BY month
  ROWS BETWEEN UNBOUNDED PRECEDING    -- From the first row
           AND CURRENT ROW            -- Through the current row → running total
) AS running_total

AVG(sales) OVER (
  ORDER BY month
  ROWS BETWEEN 2 PRECEDING           -- From two rows earlier
           AND CURRENT ROW            -- Through the current row → three-month moving average
) AS moving_avg_3m
Batch-processing use case:This pattern is common in time-series batches, such as identifying the month when cumulative sales exceeded a target. A window function is essential because it aggregates without removing rows.
Problem ▼

From the monthly_sales table below, calculate each month’s running sales total (running_total) and moving average for the latest three months (moving_avg_3m).

Source table
▸ monthly_sales
monthsales
2024-0180000
2024-02120000
2024-03100000
2024-04150000
2024-05130000
2024-06170000
Expected Output ▼
monthsalesrunning_totalmoving_avg_3m
2024-01800008000080000.0
2024-02120000200000100000.0
2024-03100000300000100000.0
2024-04150000450000123333.3
2024-05130000580000126666.7
2024-06170000750000150000.0
Model Answer ▼
SELECT
  month,
  sales,
  SUM(sales) OVER (                                   -- ① Running sales total: sum from the first row through the current row
    ORDER BY month
    ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  ) AS running_total,
  ROUND(                                              -- ② Three-month moving average: average from two rows earlier through the current row
    AVG(sales) OVER (
      ORDER BY month
      ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
    )::numeric, 1
  ) AS moving_avg_3m
FROM monthly_sales
ORDER BY month;

/*
  Evaluation order:
  1. FROM monthly_sales  → read rows
  2. SUM() OVER (...)    → calculate the running sales total
  3. AVG() OVER (...)    → calculate the three-month moving average
  4. ORDER BY month      → return rows in ascending month order
*/
Explanation (table transitions & key points) ▼
SELECT month, sales, SUM(sales) OVER ( ORDER BY month ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS running_total, ROUND( AVG(sales) OVER ( ORDER BY month ROWS BETWEEN 2 PRECEDING AND CURRENT ROW )::numeric, 1 ) AS moving_avg_3m FROM monthly_sales ORDER BY month;
LEGEND ▼
Rows read / loaded
① FROM ▼
FROM monthly_salesRead all six rows from monthly_sales. In the next step, SUM OVER adds the running total from the first row through the current row.
1 / 3
monthsales
2024-0180,000
2024-02120,000
2024-03100,000
2024-04150,000
2024-05130,000
2024-06170,000
Read all 6 rows
LEARNING POINTS
ROWS vs. RANGE:ROWS BETWEEN defines a window by physical row count, while RANGE BETWEEN uses a range of values. Prefer ROWS when its clearer and typically faster semantics match the requirement.
Practical use of moving averages:They smooth short-term trends and provide a foundation for anomaly detection.
ANTI-PATTERNS
Running total without ORDER BY:SUM() OVER() returns the total across all rows. A running total requires ORDER BY and ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW.
Practical note
Moving averages smooth noise from weekdays, campaigns, and similar effects to reveal trends. Seven-day (7MA) and 28-day (28MA) windows are standard choices.
QUESTION 4

ROW_NUMBER() + CTE — Build an API for Top-N Products by Category

ROW_NUMBERPARTITION BYTop NRanking API
Background ▼

APIs often need only the top N rows in each category. A standard pattern assigns each row a rank within its group using ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ...), isolates that result in a CTE, and then filters with WHERE rn <= N.

WITH ranked AS (
  SELECT *,
    ROW_NUMBER() OVER (
      PARTITION BY category        -- Number rows independently within each category
      ORDER BY     sales DESC, product_id  -- Break ties by product_id ascending
    ) AS rn
  FROM products
)
SELECT *
FROM   ranked
WHERE  rn <= 2;   -- Keep only the top two rows in each category
ROW_NUMBER vs. RANK vs. DENSE_RANK:ROW_NUMBER always assigns a unique sequence even when values tie. RANK gives ties the same rank and skips the next number (1, 1, 3), while DENSE_RANK does not skip it (1, 1, 2). ROW_NUMBER is usually easiest for extracting exactly N rows.
Problem ▼

From the product_sales table below, extract the top two products by sales within each category.

Source table
▸ product_sales
product_idcategoryproduct_namesales
P01FoodApple85000
P02FoodBanana62000
P03FoodMandarin62000
P04FoodGrape41000
P05RefreshmentsGreen tea95000
P06RefreshmentsCoffee78000
P07RefreshmentsJuice78000
P08RefreshmentsWater55000
Expected Output ▼
categoryproduct_namesalesrn
FoodApple850001
FoodBanana620002
RefreshmentsGreen tea950001
RefreshmentsCoffee780002
Model Answer ▼
-- WITH: split processing into a CTE
WITH ranked_products AS (
  -- ① Rank sales within each category
  SELECT
    category,
    product_name,
    sales,
    ROW_NUMBER() OVER (
      PARTITION BY category
      ORDER BY sales DESC, product_id
    ) AS rn
  FROM product_sales
)
-- ② Extract the top two rows in each category
SELECT
  category,
  product_name,
  sales,
  rn
FROM ranked_products
WHERE rn <= 2
ORDER BY category, rn;

/*
  Evaluation order:
  1. CTE ranked_products    → rank sales within each category
  2. Outer WHERE rn <= 2      → extract the top two rows per category
  3. ORDER BY category, rn  → sort and return the result
*/
Explanation (table transitions & key points) ▼
WITH ranked_products AS ( SELECT category, product_name, sales, ROW_NUMBER() OVER ( PARTITION BY category ORDER BY sales DESC, product_id ) AS rn FROM product_sales ) SELECT category, product_name, sales, rn FROM ranked_products WHERE rn <= 2 ORDER BY category, rn;
LEGEND ▼
Rows read / loaded
① FROM ▼
FROM product_salesRead all eight rows from product_sales. In the next step, PARTITION BY forms one group for each category.
1 / 5
product_idcategoryproduct_namesales
P01FoodApple85,000
P02FoodBanana62,000
P03FoodMandarin62,000
P04FoodGrape41,000
P05RefreshmentsGreen tea95,000
P06RefreshmentsCoffee78,000
P07RefreshmentsJuice78,000
P08RefreshmentsWater55,000
Read all 8 rows
LEARNING POINTS
Why the CTE is necessary:Window functions are evaluated after WHERE. Complete the window calculation in a CTE, then filter it in the outer query.
PARTITION BY:Without it, the rank covers the entire result. It is required for ranks within groups such as categories.
ANTI-PATTERNS
WHERE at the same query level:Directly using a window-function result in WHERE at the same SELECT level causes an error.
Practical note
Top N per category and selecting only the latest history row with ORDER BY created_at DESC are essential data-mart patterns.
QUESTION 5

CASE WHEN + SUM — Pivot Counts by Status

CASE WHENPivotSUM FILTERCross Tabulation
Background ▼

Pivot aggregation, which expands row-oriented data into columns, is common in dashboard APIs. Because many SQL dialects lack a dedicated PIVOT statement, it can be implemented with CASE WHEN + SUM.

SELECT
  category,
  SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) AS completed,
  SUM(CASE WHEN status = 'pending' THEN 1 ELSE 0 END) AS pending
FROM   tasks
GROUP BY category;
PostgreSQL also supports FILTER:COUNT(*) FILTER (WHERE status = 'done') produces the same result. CASE WHEN is more portable because some databases, including BigQuery, do not support this FILTER syntax.
Problem ▼

From the tasks table below, return one row per assignee containing counts by status (done, in progress, and not started) plus the total count.

Source table
▸ tasks
task_idassigneestatus
T01Bakerdone
T02Bakerdone
T03Bakerin progress
T04Bakernot started
T05Clarkdone
T06Clarkin progress
T07Clarkin progress
T08Adamsnot started
T09Adamsnot started
T10Adamsdone
Expected Output ▼
assigneedonein_progressnot_startedtotal
Adams1023
Baker2114
Clark1203
Model Answer ▼
SELECT
  assignee,
  SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) AS done,          -- ① Count done tasks
  SUM(CASE WHEN status = 'in progress' THEN 1 ELSE 0 END) AS in_progress,  -- ② Count tasks in progress
  SUM(CASE WHEN status = 'not started' THEN 1 ELSE 0 END) AS not_started,  -- ③ Count tasks not started
  COUNT(*) AS total                                                -- ④ Count all tasks
FROM tasks
GROUP BY assignee
ORDER BY assignee;

/*
  Evaluation order:
  1. FROM tasks         → read rows
  2. GROUP BY assignee  → group by assignee
  3. SUM(CASE...)       → aggregate counts by status (pivot)
  4. ORDER BY assignee  → return rows in ascending order
*/
Explanation (table transitions & key points) ▼
SELECT assignee, SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) AS done, SUM(CASE WHEN status = 'in progress' THEN 1 ELSE 0 END) AS in_progress, SUM(CASE WHEN status = 'not started' THEN 1 ELSE 0 END) AS not_started, COUNT(*) AS total FROM tasks GROUP BY assignee ORDER BY assignee;
LEGEND ▼
Rows read / loaded
① FROM ▼
FROM tasksRead all ten rows from tasks. In the next step, GROUP BY groups the data by assignee.
1 / 5
task_idassigneestatus
T01Bakerdone
T02Bakerdone
T03Bakerin progress
T04Bakernot started
T05Clarkdone
T06Clarkin progress
T07Clarkin progress
T08Adamsnot started
T09Adamsnot started
T10Adamsdone
Read all 10 rows
LEARNING POINTS
Creating flags with CASE WHEN:SUM(CASE WHEN ... THEN 1 ELSE 0 END) is an idiom for counting rows that satisfy a condition.
Compression through pivoting:Expand row-oriented data into columns to build an aggregated API response table in one query.
ANTI-PATTERNS
Omitting ELSE:A nonmatching condition returns NULL. Specify ELSE 0 to prevent unintended behavior.
Practical note
Pivoting in the database can substantially reduce data transfer and application memory use. It is effective for dashboard aggregation over large logs.