airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumit Maheshwari <sumeet.ma...@gmail.com>
Subject Re: Info needed regarding upgrading to 1.8.2
Date Sat, 19 Aug 2017 06:14:29 GMT
I think the I missed the *dag_dir_list_interval *setting and assumed that
scheduler picks up new dags instantly. I will retest things with a lower
value of that flag and see how it behaves.

Thanks for the help Boris, Max and Bolke.



Thanks,
Sumit Maheshwari
cell. 9632202950


On Sat, Aug 19, 2017 at 1:00 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> If you have a large number of dags, it can take some time before the
> scheduler processes them. This is determined by the following settings in
> the [scheduler] section:
>
> # after how much time a new DAGs should be picked up from the filesystem
> min_file_process_interval = 0
> dag_dir_list_interval = 300
>
> Bolke
>
> > On 18 Aug 2017, at 21:26, Maxime Beauchemin <maximebeauchemin@gmail.com>
> wrote:
> >
> > In 1.8.1 I'm pretty sure that in the context of the scheduler, the DAGs
> are
> > fully re-evaluated in subprocesses which fixes the re-import
> > sys.module-related caching issue of earlier versions that created
> > staleness. 1.8.1 shouldn't have scheduler staleness issues.
> >
> > Max
> >
> > On Fri, Aug 18, 2017 at 6:09 AM, Boris Tyukin <boris@boristyukin.com>
> wrote:
> >
> >> see how much time it takes to parse your dag, if you do any heavy
> >> operations like database calls, file pulls etc. I remember someone had
> an
> >> issue with web server because it took a while to parse one dag. By
> default
> >> airflow will check dag folder very frequently (unless you changed the
> >> timing in config). You can start both scheduler and web server manually
> and
> >> just watch the logs
> >>
> >> On Fri, Aug 18, 2017 at 9:03 AM, Sumit Maheshwari <
> sumeet.manit@gmail.com>
> >> wrote:
> >>
> >>> Thanks Boris,
> >>>
> >>> Seems like there is some issue in my Scheduler, which sometimes picks
> up
> >>> dag modification immediately and sometimes doesn't pickup at all. And
> >> when
> >>> Scheduler doesn't pickup new dag, UI also doesn't show it, i.e. no
> >> refresh
> >>> button would be available. I am trying to debug that issue with
> >> Scheduler,
> >>> but not sure if it's a real issue at all or something else.
> >>>
> >>>
> >>>
> >>> On Fri, Aug 18, 2017 at 6:16 PM, Boris Tyukin <boris@boristyukin.com>
> >>> wrote:
> >>>
> >>>> we run 1.8.1 with no debug mode. New dags are picked up just fine
> >> without
> >>>> restarts. as well as modifications (adding new tasks or changing
> >>>> definitions of new dags). When I want to see dag updated on UI, I hit
> >>>> refresh button but this is just for UI to show it properly. No
> restarts
> >>>> needed in our experience with 1.8.1
> >>>>
> >>>> On Fri, Aug 18, 2017 at 6:40 AM, Sumit Maheshwari <
> >>> sumeet.manit@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> We are trying to upgrade to Airflow ver 1.8.2 from 1.7.0 and found
a
> >>>> couple
> >>>>> of issues, major ones are:
> >>>>>
> >>>>> 1. We used to run the webserver in debug mode (-d) so we don't need
> >> to
> >>>>> restart whenever we add/modify any dag. But with 1.8.2 debug mode
> >>> doesn't
> >>>>> have any effect and we need to restart it after any change in dags.
> >>>>>
> >>>>> 2. Scheduler used to pick up any new dag landed in dags folder,
but
> >>> that
> >>>> is
> >>>>> not happening anymore and we required to restart the scheduler as
> >> well.
> >>>>>
> >>>>> Any help would be highly appreciated.
> >>>>>
> >>>>> Thanks,
> >>>>> Sumit
> >>>>>
> >>>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message