Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <F22F6750-25C3-434C-80AE-5661CDC7AD95@gmail.com>
References: <CABN=AetFPtPwvF8b4+sdM=3gHC1tmzfcQDTaDUCkWwRiF+=Khg@mail.gmail.com>
 <CAMNEcwsep-VLrTLPqCyVOxbia1wecCZos6bQYiaJPj3bh8o0wg@mail.gmail.com> <F22F6750-25C3-434C-80AE-5661CDC7AD95@gmail.com>
From: Alek Storm <alek.storm@gmail.com>
Date: Tue, 5 Dec 2017 16:34:33 -0600
Message-ID: <CAMNEcwtqJbPUEqgUxngZDcPMoRpJJyOf9aVBRpCxZ+jO2K9yOA@mail.gmail.com>
Subject: Re: DAG Factory Issues
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary="001a1140417c696fde055f9f707a"
archived-at: Tue, 05 Dec 2017 22:34:47 -0000

--001a1140417c696fde055f9f707a
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Sorry I wasn=E2=80=99t clear; I just meant that it seems more useful for fi=
leloc to
reflect the file in the dags folder that the scheduler processed to yield
the DAG
<https://github.com/apache/incubator-airflow/blob/1359d87352bda220f5d88613f=
d81904378624c7b/airflow/jobs.py#L1710>
.

Actually, the full_filepath attribute is already exactly this, so I suppose
I=E2=80=99m advocating using full_filepath instead of fileloc in all cases.

Alek
=E2=80=8B

On Tue, Dec 5, 2017 at 1:12 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> I dont see how the scheduler would know. DAGs can come from any location,
> any module, and import is dependent on the python interpreter.
>
> If you know a way the suits every kind of structure please provide a pr.
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 5 dec. 2017 om 19:05 heeft Alek Storm <alek.storm@gmail.com> het
> volgende geschreven:
> >
> > We solved this (hackily) by setting the fileloc field on the generated
> DAG
> > object to inspect.getsourcefile(inspect.stack()[1][0]), as the DAG
> > constructor itself does. I agree a more general solution is needed;
> > presumably the scheduler knows which Python file in the dags folder it
> was
> > processing when it found the DAG object.
> >
> > Alek
> > =E2=80=8B
> >
> >> On Tue, Dec 5, 2017 at 12:00 PM, Michael Erdely <mjerdely@gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >> In order to support multiple environments with different DAG settings
> per
> >> environment, we created a DAG factory to create the DAG operator where
> each
> >> DAG has different params (eg schedule, catchup, etc).
> >>
> >> Unfortunately, we noticed that the webserver code view shows the facto=
ry
> >> code versus the actual Python code in the dagbag file. I also assume
> that
> >> the factory file modified date will be used versus the DAG file to
> >> determine if the scheduler should reload.
> >>
> >> I see this issue has already been reported here
> >> https://issues.apache.org/jira/browse/AIRFLOW-1108.
> >>
> >> Any ideas if this will be patched soon or have better suggestions on
> >> handling this?
> >>
> >> -Michael
> >>
>

--001a1140417c696fde055f9f707a--