airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruiqin Yang <yrql...@gmail.com>
Subject Re: Disable Processing of DAG file
Date Tue, 29 May 2018 06:29:38 GMT
Hi folks,
This config line
<https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L414>
controls how often the scheduler scan the DAG folder and tries to discover/
forget DAGs.

For doing dag file processing part, scheduler does parse the DAG file
everytime before it schedules tasks through DagFileProcessor.

Cheers,
Kevin Y

On Mon, May 28, 2018 at 10:14 PM, Ananth Durai <vananth22@gmail.com> wrote:

> It is an interesting question. On a slightly related note, Correct me if
> I'm wrong, AFAIK we require restarting airflow scheduler in order pick any
> new DAG file changes by the scheduler. In that case, should the scheduler
> do the DAGFileProcessing every time before scheduling the tasks?
>
> Regards,
> Ananth.P,
>
>
>
>
>
>
> On 28 May 2018 at 21:46, ramandumcs@gmail.com <ramandumcs@gmail.com>
> wrote:
>
> > Hi All,
> >
> > We have a use case where there would be 100(s) of DAG files with schedule
> > set to "@once". Currently it seems that scheduler processes each and
> every
> > file and creates a Dag Object.
> > Is there a way or config to tell scheduler to stop processing certain
> > files.
> >
> > Thanks,
> > Raman Gupta
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message