airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-812) Scheduler job terminates when there is no dag file despite run_numbers and run_duration set to unlimited.
Date Fri, 27 Jan 2017 10:44:24 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15842533#comment-15842533
] 

ASF subversion and git services commented on AIRFLOW-812:
---------------------------------------------------------

Commit f8d8ddac08dbec90b240a6b8670be9bd650dfb3b in incubator-airflow's branch refs/heads/master
from [~farlue]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f8d8dda ]

[AIRFLOW-812] Fix the scheduler termination bug.

When checking max-runs for dag_files the scheduler
would stop immediately if there were no files.

Closes #2027 from fenglu-g/master


> Scheduler job terminates when there is no dag file despite run_numbers and run_duration
set to unlimited.
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-812
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-812
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: utils
>            Reporter: Feng Lu
>            Assignee: Feng Lu
>
> When scheduler run_numbers and run_duration are set to -1 (unlimited), we expect the
scheduler to run continuously. However, when dag folder is empty, the scheduler terminates
immediately. This is due to a bug in utils/dag_processing when checking max_runs_reached.

> Details:
> airflow scheduler
> [2017-01-27 08:20:50,433] {__init__.py:57} INFO - Using executor SequentialExecutor
> [2017-01-27 08:20:50,508] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-01-27 08:20:50,529] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
>   ____________       _____________
>  ____    |__( )_________  __/__  /________      __
> ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
> ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
>  _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
> [2017-01-27 08:20:50,771] {jobs.py:514} ERROR - Cannot use more than 1 thread when using
sqlite. Setting max_threads to 1
> [2017-01-27 08:20:50,786] {jobs.py:1222} INFO - Starting the scheduler
> [2017-01-27 08:20:50,787] {jobs.py:1238} INFO - Processing files using up to 1 processes
at a time
> [2017-01-27 08:20:50,787] {jobs.py:1240} INFO - Running execute loop for -1 seconds
> [2017-01-27 08:20:50,787] {jobs.py:1242} INFO - Processing each file at most None times
> [2017-01-27 08:20:50,787] {jobs.py:1244} INFO - Process each file at most once every
0 seconds
> [2017-01-27 08:20:50,787] {jobs.py:1246} INFO - Checking for new files in /home/fenglu/airflow/dags
every 300 seconds
> [2017-01-27 08:20:50,787] {jobs.py:1249} INFO - Searching for files in /home/fenglu/airflow/dags
> [2017-01-27 08:20:50,787] {jobs.py:1252} INFO - There are 0 files in /home/fenglu/airflow/dags
> [2017-01-27 08:20:50,787] {jobs.py:1314} INFO - Resetting state for orphaned tasks
> [2017-01-27 08:20:50,789] {jobs.py:1326} INFO - Removing old import errors
> [2017-01-27 08:20:50,795] {jobs.py:1363} INFO - Heartbeating the process manager
> [2017-01-27 08:20:50,796] {jobs.py:1400} INFO - Heartbeating the executor
> [2017-01-27 08:20:51,797] {jobs.py:1432} INFO - Exiting loop as all files have been processed
None times
> [2017-01-27 08:20:51,797] {jobs.py:1448} INFO - Deactivating DAGs that haven't been touched
since 2017-01-27T08:20:50.795782
> [2017-01-27 08:20:51,801] {jobs.py:1271} INFO - Exited execute loop



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message