airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Info needed regarding upgrading to 1.8.2
Date Fri, 18 Aug 2017 19:30:24 GMT
If you have a large number of dags, it can take some time before the scheduler processes them.
This is determined by the following settings in the [scheduler] section:

# after how much time a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
dag_dir_list_interval = 300

Bolke

> On 18 Aug 2017, at 21:26, Maxime Beauchemin <maximebeauchemin@gmail.com> wrote:
> 
> In 1.8.1 I'm pretty sure that in the context of the scheduler, the DAGs are
> fully re-evaluated in subprocesses which fixes the re-import
> sys.module-related caching issue of earlier versions that created
> staleness. 1.8.1 shouldn't have scheduler staleness issues.
> 
> Max
> 
> On Fri, Aug 18, 2017 at 6:09 AM, Boris Tyukin <boris@boristyukin.com> wrote:
> 
>> see how much time it takes to parse your dag, if you do any heavy
>> operations like database calls, file pulls etc. I remember someone had an
>> issue with web server because it took a while to parse one dag. By default
>> airflow will check dag folder very frequently (unless you changed the
>> timing in config). You can start both scheduler and web server manually and
>> just watch the logs
>> 
>> On Fri, Aug 18, 2017 at 9:03 AM, Sumit Maheshwari <sumeet.manit@gmail.com>
>> wrote:
>> 
>>> Thanks Boris,
>>> 
>>> Seems like there is some issue in my Scheduler, which sometimes picks up
>>> dag modification immediately and sometimes doesn't pickup at all. And
>> when
>>> Scheduler doesn't pickup new dag, UI also doesn't show it, i.e. no
>> refresh
>>> button would be available. I am trying to debug that issue with
>> Scheduler,
>>> but not sure if it's a real issue at all or something else.
>>> 
>>> 
>>> 
>>> On Fri, Aug 18, 2017 at 6:16 PM, Boris Tyukin <boris@boristyukin.com>
>>> wrote:
>>> 
>>>> we run 1.8.1 with no debug mode. New dags are picked up just fine
>> without
>>>> restarts. as well as modifications (adding new tasks or changing
>>>> definitions of new dags). When I want to see dag updated on UI, I hit
>>>> refresh button but this is just for UI to show it properly. No restarts
>>>> needed in our experience with 1.8.1
>>>> 
>>>> On Fri, Aug 18, 2017 at 6:40 AM, Sumit Maheshwari <
>>> sumeet.manit@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi All,
>>>>> 
>>>>> We are trying to upgrade to Airflow ver 1.8.2 from 1.7.0 and found a
>>>> couple
>>>>> of issues, major ones are:
>>>>> 
>>>>> 1. We used to run the webserver in debug mode (-d) so we don't need
>> to
>>>>> restart whenever we add/modify any dag. But with 1.8.2 debug mode
>>> doesn't
>>>>> have any effect and we need to restart it after any change in dags.
>>>>> 
>>>>> 2. Scheduler used to pick up any new dag landed in dags folder, but
>>> that
>>>> is
>>>>> not happening anymore and we required to restart the scheduler as
>> well.
>>>>> 
>>>>> Any help would be highly appreciated.
>>>>> 
>>>>> Thanks,
>>>>> Sumit
>>>>> 
>>>> 
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message