airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Python DAG files must include an import from airflow for dag to be loaded to DagBag
Date Wed, 16 Nov 2016 19:15:54 GMT
The reason is this line of code:
https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L249

As we fill the DagBag, there's this "sanity check" that validates that the
strings "DAG" and "airflow" are contained somewhere in the file before even
interpreting it.

Recursing through a file system and interpreting all Python files can lead
to bad things or unintended results, and that line of code is a very basic
and imperfect way that attempts only interpreting files that have something
to do with Airflow.

Knowing this, you can just add `from airflow import DAG` to you file, it
can even be commented out and that should to the trick.

Also note that you need to stamp the instantiated DAG object in module
scope for it to be discovered. If you were to build N dags dynamically, you
can simply use python's `globals()` method as in `globals()[dag_id] =
DAG(dag_id)`

Max

On Wed, Nov 16, 2016 at 7:46 AM, Tamara Mendt <tm@hellofresh.com> wrote:

> Hello,
>
> I created a DAGFactory class which encapsulates my DAG creation logic and
> then I tried creating a DAG in a separate python file using the factory
> class. This resulted in me not importing any module from airflow in the
> file were the DAG is defined. I noticed that when doing this, the DAG is
> initialized properly but is not loaded into airflow's DagBag. I found this
> to be a bit restrictive and I wonder if there is any special reason for
> this.
>
> Cheers,
>
> Tamara
>
> --
>
> [image: logo]
>   <http://www.facebook.com/hellofreshde>   <http://twitter.com/
> HelloFreshde>
>    <http://instagram.com/hellofreshde/>   <http://blog.hellofresh.de/>
> <https://app.adjust.com/ayje08?campaign=Hellofresh&
> deep_link=hellofresh%3A%2F%2F&post_deep_link=https%3A%2F%
> 2Fwww.hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_
> source%3Demail_signature&fallback=https%3A%2F%2Fwww.
> hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_source%
> 3Demail_signature>
>
> *HelloFresh App –Download Now!*
> <https://app.adjust.com/ayje08?campaign=Hellofresh&
> deep_link=hellofresh%3A%2F%2F&post_deep_link=https%3A%2F%
> 2Fwww.hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_
> source%3Demail_signature&fallback=https%3A%2F%2Fwww.
> hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_source%
> 3Demail_signature>
> *We're active in:*
> US <https://www.hellofresh.com/?utm_medium=email&utm_source=
> email_signature>
>  |  DE
> <https://www.hellofresh.de/?utm_medium=email&utm_source=email_signature> |
> UK
> <https://www.hellofresh.co.uk/?utm_medium=email&utm_source=email_signature
> >
> |  NL
> <https://www.hellofresh.nl/?utm_medium=email&utm_source=email_signature> |
> AU
> <https://www.hellofresh.com.au/?utm_medium=email&utm_
> source=email_signature>
>  |  BE
> <https://www.hellofresh.be/?utm_medium=email&utm_source=email_signature> |
> AT <https://www.hellofresh.at/?utm_medium=email&utm_source=email_signature
> >
> |  CH
> <https://www.hellofresh.ch/?utm_medium=email&utm_source=email_signature> |
> CA <https://www.hellofresh.ca/?utm_medium=email&utm_source=email_signature
> >
>
> www.HelloFreshGroup.com
> <http://www.hellofreshgroup.com/?utm_medium=email&utm_
> source=email_signature>
>
> We are hiring around the world – Click here to join us
> <https://www.hellofresh.com/jobs/?utm_medium=email&utm_
> source=email_signature>
>
> --
>
> <https://www.hellofresh.com/jobs/?utm_medium=email&utm_
> source=email_signature>
> HelloFresh AG, Berlin (Sitz der Gesellschaft) | Vorstände: Dominik S.
> Richter (Vorsitzender), Thomas W. Griesel, Christian Gärtner | Vorsitzender
> des Aufsichtsrats: Jeffrey Lieberman | Eingetragen beim Amtsgericht
> Charlottenburg, HRB 171666 B | USt-Id Nr.: DE 302210417
>
> *CONFIDENTIALITY NOTICE:* This message (including any attachments) is
> confidential and may be privileged. It may be read, copied and used only by
> the intended recipient. If you have received it in error please contact the
> sender (by return e-mail) immediately and delete this message. Any
> unauthorized use or dissemination of this message in whole or in parts is
> strictly prohibited.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message