airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From san...@apache.org
Subject incubator-airflow git commit: [AIRFLOW-548] Load DAGs immediately & continually
Date Thu, 06 Oct 2016 00:26:28 GMT
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 4d567f439 -> fe5eaabb2


[AIRFLOW-548] Load DAGs immediately & continually

A recent commit has changed the scheduler behavior
that it now
always stops after a specified period of time. The
operation scripts
(systemd etc) are not updated for this behavior
and many users actually
prefer to run the scheduler continously.

Secondly the default behavior was changed to not
pickup new DAGs
immediately, this has lead to confusion with
users.

Closes #1823 from bolkedebruin/fix_duration


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/fe5eaabb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/fe5eaabb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/fe5eaabb

Branch: refs/heads/master
Commit: fe5eaabb27136af3a3bc34521c85885b415d8246
Parents: 4d567f4
Author: Bolke de Bruin <bolke@xs4all.nl>
Authored: Wed Oct 5 17:22:04 2016 -0700
Committer: Siddharth Anand <siddharthanand@yahoo.com>
Committed: Wed Oct 5 17:22:12 2016 -0700

----------------------------------------------------------------------
 airflow/configuration.py | 11 +++++++++--
 airflow/jobs.py          |  2 +-
 2 files changed, 10 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fe5eaabb/airflow/configuration.py
----------------------------------------------------------------------
diff --git a/airflow/configuration.py b/airflow/configuration.py
index c21ae73..c1d1335 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -319,10 +319,17 @@ job_heartbeat_sec = 5
 # how often the scheduler should run (in seconds).
 scheduler_heartbeat_sec = 5
 
-run_duration = 1800
+# after how much time should the scheduler terminate in seconds
+# -1 indicates to run continuously (see also num_runs)
+run_duration = -1
+
+# after how much time a new DAGs should be picked up from the filesystem
+min_file_process_interval = 0
+
 dag_dir_list_interval = 300
+
+# How often should stats be printed to the logs
 print_stats_interval = 30
-min_file_process_interval = 180
 
 child_process_log_directory = /tmp/airflow/scheduler/logs
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fe5eaabb/airflow/jobs.py
----------------------------------------------------------------------
diff --git a/airflow/jobs.py b/airflow/jobs.py
index d0e3c31..77bb1c4 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -1324,7 +1324,7 @@ class SchedulerJob(BaseJob):
 
         # For the execute duration, parse and schedule DAGs
         while (datetime.now() - execute_start_time).total_seconds() < \
-                self.run_duration:
+                self.run_duration or self.run_duration < 0:
             self.logger.debug("Starting Loop...")
             loop_start_time = time.time()
 


Mime
View raw message