airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tanveer Sattar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-380) Cannot cancel "catch-up" dag runs after unpausing a dag (or restarting scheduler after long)
Date Fri, 29 Jul 2016 12:57:20 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399291#comment-15399291
] 

Tanveer Sattar commented on AIRFLOW-380:
----------------------------------------

You can use "airflow backfill" command with --mark_success option. This marks all tasks of
a dag within date range, as succeeded, without actually running them. You can also provide
regex to filter specific task ids to be marked with -t option. This works even if max_active_runs
is set to 1. 

airflow backfill <dag-id> --mark_success -s <start-date> -e <end-date>

> Cannot cancel "catch-up" dag runs after unpausing a dag (or restarting scheduler after
long)
> --------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-380
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-380
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Vineet Goel
>
> Currently when we unpause a dag or bring the scheduler back up after long, the scheduler
starts scheduler dag runs for each execution in between (for the time that has elapsed). There
should be a way to cancel these "backfills" if they are not really required.
> While this is simpler to clear using the cli if the dagruns are created simultaneously,
it becomes a bit hard if the max_active_runs for the dag is 1. This way, we need to clear
each dag run individually which isn't ideal. One workaround is to keep start_date as datetime.now()
and deleting past dag runs from the database prior to unpausing the dag. However this isn't
ideal.
> Is there a way of doing that I don't know about?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message