airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruslan Dautkhanov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1178) @once may run more than one time
Date Thu, 08 Jun 2017 23:45:18 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ruslan Dautkhanov updated AIRFLOW-1178:
---------------------------------------
    Affects Version/s: 1.8.2

> @once may run more than one time
> --------------------------------
>
>                 Key: AIRFLOW-1178
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1178
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: Airflow 1.8, 1.8.1, 1.8.0, 1.8.2
>         Environment: Airflow master snapshot from May 05 2017
>            Reporter: Ruslan Dautkhanov
>            Priority: Critical
>         Attachments: onceDAG_got_scheduled_twice.png
>
>
> My DAG is running second (2nd) time although it is declared as @once.
> Here's DAG definition :
> {noformat}
> main_dag = DAG(
>     dag_id                         = 'Test-DAG-1',
>     default_args                   = default_args,                  # dafeult operators'
arguments - see above
>     user_defined_macros            = dag_macros,       # I do not get different between
>     ## params                         = dag_macros,       # user_defined_macros and params
>     #
>     start_date                     = datetime.now(),                # or e.g. datetime(2015,
6, 1)
>     # 'end_date'                   = datetime(2016, 1, 1),
>     catchup                        = True,                         # Perform scheduler
catchup (or only run latest)?
>                                                                         # - defaults
to True
>     schedule_interval              = '@once',                       # '@once'=None?
>                                                                      # doesn't create
multiple dag runs automatically
>     concurrency                    = 3,                             # task instances
allowed to run concurrently
>     max_active_runs                = 1,                             # only one DAG run
at a time
>     dagrun_timeout                 = timedelta(days=4),             # no way this dag
should ran for 4 days
>     orientation                    = 'TB',                          # default graph view
> )
> {noformat}
> As a workaround for AIRFLOW-1013 I changed catchup from False to True. Suggested on dev
list. It "worked around" AIRFLOW-1013 execution, but screwed @once logic - the DAG got scheduled
twice (!) which is a no-go for us. The DAG actually has to run not more than 1 time.
> IMO, catchup=True should be explicitly disallowed for @once schedule.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message