Apache Airflow Cron Scheduling: Complete Configuration Guide

Introduction

Apache Airflow uses cron expressions for scheduling DAGs (Directed Acyclic Graphs), but its scheduling semantics differ significantly from system cron. Understanding these differences is crucial to avoid common pitfalls that cause DAGs to run at unexpected times—or not at all.

This guide covers Airflow's scheduling system, how it differs from system cron, common scheduling patterns, and the traps that catch even experienced users.

Airflow Scheduling vs System Cron

Key Differences

The "After Period" Concept

The most confusing aspect of Airflow scheduling: Airflow schedules a DAG run AFTER the scheduled period ends, not AT the cron time.

Cron: 0 0 * * * (midnight)
System cron: Runs at 00:00:00
Airflow: Runs at 00:00:00 to process the PREVIOUS day's data

Example:

# This DAG's first run will be scheduled for 2024-01-02 00:00:00
# That run processes data for 2024-01-01
dag = DAG(
    'daily_etl',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 0 * * *',  # Midnight
)

schedule_interval vs Timetable

schedule_interval (Traditional Approach)

The schedule_interval parameter accepts:

• Cron expressions as strings: '0 0 * * *'
• Presets: '@daily', '@hourly', '@weekly'
• timedelta objects: timedelta(hours=6)

from airflow import DAG
from datetime import datetime, timedelta

dag = DAG(
    'example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 */6 * * *',  # Every 6 hours
    catchup=False,
)

Timetable (Modern Approach, Airflow 2.2+)

Timetables provide more control over scheduling logic, especially for complex business calendars:

from airflow.timetables.trigger import CronTriggerTimetable

dag = DAG(
    'example_dag',
    start_date=datetime(2024, 1, 1),
    # More explicit: runs at midnight in UTC
    schedule=Timetable(
        cron=CronTriggerTimetable("0 0 * * *", timezone="UTC"),
    ),
    catchup=False,
)

Custom Timetable Example

For business-day-only scheduling (skip weekends):

from airflow.timetables.base import Timetable
from datetime import datetime
import pendulum

class BusinessDayTimetable(Timetable):
    def next_dagrun_info(self, last_automated_data_interval, restriction):
        # Calculate next business day
        if last_automated_data_interval is None:
            next_start = datetime(2024, 1, 1, tzinfo=pendulum.UTC)
        else:
            next_start = last_automated_data_interval.end + timedelta(days=1)
            # Skip weekends
            while next_start.weekday() >= 5:  # 5=Saturday, 6=Sunday
                next_start += timedelta(days=1)
        
        return DagRunInfo(
            run_after=next_start,
            data_interval=(next_start, next_start + timedelta(days=1))
        )

dag = DAG(
    'business_day_dag',
    start_date=datetime(2024, 1, 1),
    schedule=BusinessDayTimetable(),
    catchup=False,
)

Common Scheduling Patterns

Daily at Midnight (UTC)

dag = DAG(
    'daily_midnight',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 0 * * *',
    catchup=False,
)

Every 6 Hours

dag = DAG(
    'every_6_hours',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 */6 * * *',
    catchup=False,
)

Weekly on Monday Morning

dag = DAG(
    'weekly_monday',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 8 * * 1',  # 8 AM every Monday
    catchup=False,
)

First Day of Month

dag = DAG(
    'monthly_report',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 0 1 * *',  # Midnight on the 1st
    catchup=False,
)

Every 15 Minutes

dag = DAG(
    'frequent_check',
    start_date=datetime(2024, 1, 1),
    schedule_interval='*/15 * * * *',
    catchup=False,
)

Business Hours Only (9 AM - 5 PM, Mon-Fri)

dag = DAG(
    'business_hours',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 9-17 * * 1-5',  # 9 AM to 5 PM, Mon-Fri
    catchup=False,
)

The start_date Trap

Misconception: start_date is When the DAG Starts Running

Many users think start_date=datetime(2024, 1, 1) means "start running on January 1st."

Reality: start_date defines the beginning of the first scheduled period. The DAG runs after that period ends.

# WRONG mental model:
# "This runs starting January 1st at midnight"
dag = DAG(
    'daily',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 0 * * *',
)

# CORRECT mental model:
# "The first scheduled period is January 1st (00:00-24:00)"
# "The DAG runs at January 2nd 00:00 to process January 1st's data"

The start_date + schedule_interval Interaction

from datetime import datetime

# Scenario: Today is 2024-01-10
# start_date is in the past
dag = DAG(
    'example',
    start_date=datetime(2024, 1, 1),  # 9 days ago
    schedule_interval='0 0 * * *',
    catchup=True,  # Default is True!
)

# With catchup=True, Airflow will schedule runs for:
# 2024-01-02 00:00 (processes Jan 1st data)
# 2024-01-03 00:00 (processes Jan 2nd data)
# ...
# 2024-01-10 00:00 (processes Jan 9th data)
# That's 9 backfill runs!

The fix: Set catchup=False unless you want backfill:

dag = DAG(
    'example',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 0 * * *',
    catchup=False,  # Only schedule the most recent period
)

Timezone Handling

Always Use UTC in start_date

# GOOD: Use UTC
from datetime import datetime
import pendulum

dag = DAG(
    'timezone_example',
    start_date=datetime(2024, 1, 1, tzinfo=pendulum.UTC),
    schedule_interval='0 0 * * *',
    catchup=False,
)

# BETTER: Use pendulum for clarity
dag = DAG(
    'timezone_example',
    start_date=pendulum.datetime(2024, 1, 1, tz="UTC"),
    schedule_interval='0 0 * * *',
    catchup=False,
)

Scheduling in a Specific Timezone

dag = DAG(
    'tokyo_daily',
    start_date=pendulum.datetime(2024, 1, 1, tz="Asia/Tokyo"),
    schedule_interval='0 9 * * *',  # 9 AM Tokyo time
    catchup=False,
)

# The cron expression '0 9 * * *' is interpreted in the DAG's timezone
# So this runs at 9 AM Asia/Tokyo, not 9 AM UTC

catchup: The Silent DAG Runner

What catchup Does

When catchup=True (the default), Airflow will schedule all missed runs between start_date and the current date.

# Today is 2024-01-10
dag = DAG(
    'catchup_example',
    start_date=datetime(2024, 1, 1),  # 9 days ago
    schedule_interval='0 0 * * *',
    catchup=True,  # This is the default!
)

# Airflow will create 9 DAG runs immediately:
# - 2024-01-02 00:00 (for Jan 1st data)
# - 2024-01-03 00:00 (for Jan 2nd data)
# ...
# - 2024-01-10 00:00 (for Jan 9th data)

When to Use catchup=True

• Initial backfill of historical data
• Re-processing after fixing a bug
• Data pipelines where historical data is needed

When to Use catchup=False

• Live production pipelines (only process current data)
• Resource-intensive DAGs (avoid sudden load spike)
• DAGs with external dependencies that only exist for recent data

Debugging Scheduling Issues

1. Check When the Next Run Is Scheduled

# Using CLI
airflow dags show <dag_id> --tree

# Or in the UI: Browse -> DAGs -> [Your DAG] -> "Next Run" column

2. Understand the Data Interval

from airflow import DAG
from datetime import datetime

dag = DAG(
    'debug_example',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 0 * * *',
)

# In your task, you can access:
# {{ data_interval_start }} - Start of the scheduled period
# {{ data_interval_end }} - End of the scheduled period (when DAG runs)

3. Test Scheduling Without Running

# Parse the schedule and show next runs
airflow dags test <dag_id> 2024-01-01

4. Common Issue: DAG Not Running

Check these in order:

1. Is the DAG paused? (airflow dags unpause <dag_id>)
2. Is start_date in the past? (Future start_date = won't run yet)
3. Is catchup=False and the scheduler already passed the start_date?
4. Check the scheduler logs: airflow scheduler output or /var/log/airflow/scheduler/

Best Practices

1. Always set catchup=False unless you need backfill
2. Use pendulum for timezone-aware datetimes
3. Don't use dynamic start_date (e.g., datetime.now()) - it causes unpredictable behavior
4. Test scheduling with airflow dags test before deploying
5. Use descriptive schedule_interval - cron strings are opaque; add a comment
6. Consider using Timetable for complex scheduling instead of cramming logic into cron expressions

Quick Reference: Cron to Airflow Mapping

Related Tools

• Cron Parser - Validate and understand cron expressions