airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dennis O'Brien (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-664) airflow built from master unstable since change to parse DAG files in child process
Date Thu, 01 Dec 2016 21:44:58 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dennis O'Brien updated AIRFLOW-664:
-----------------------------------
    Attachment: airflow_broken_scheduler_screenshot.png

Screenshot of Airflow UI built from fdb7e949


> airflow built from master unstable since change to parse DAG files in child process
> -----------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-664
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-664
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Dennis O'Brien
>         Attachments: airflow_broken_scheduler_screenshot.png
>
>
> Prior to commit fdb7e949140b735b8554ae5b22ad752e86f6ebaf I was able to pip build, install,
and run airflow from the github commit.  E.g.,
> {code}
> pip install -e git://github.com/apache/incubator-airflow.git@835bcb6234956445d819ff39a5790f635c201c5d#egg=airflow[celery,crypto,hive,jdbc,ldap,password,postgres,s3,vertica]
> {code}
> Since this commit, airflow is not functional.
> * The on/off button to enable a DAG does not appear.
> * Next to each DAG an info icon appears with the hover text "This DAG seems to be existing
only locally. The master scheduler doesn't seem to be aware of its existence."
> * DAGs cannot be run from the command line.
> I am able via the command line to list the dags and list the tasks for a dag, but trigger_dag
does not actually cause any tasks to run (but no error messages either).
> {code}
> airflow@41e0e6920223:~$ airflow trigger_dag example_bash_operator
> [2016-12-01 19:24:21,928] {__init__.py:50} INFO - Using executor CeleryExecutor
> [2016-12-01 19:24:22,218] {models.py:162} INFO - Filling up the DagBag from /usr/local/airflow/dags
> [2016-12-01 19:24:22,341] {cli.py:180} INFO - Created <DagRun example_bash_operator
@ 2016-12-01 19:24:22: manual__2016-12-01T19:24:22.295709, externally triggered: True>
> {code}
> This commit was #1636 "[AIRFLOW-160] Parse DAG files through child processes".  I haven't
identified what part of this commit is to blame.  Or, what part of my configuration/setup
is incompatible with this change.
> My setup:
> * Docker and docker-compose with...
> * Ubuntu 16.04
> * Python 3.5.2 (via Miniconda distribution)
> * Separate container instances for scheduler, webserver, flower, and worker.
> * Docker container running mysql 5.7
> * Docker container running redis 3.2
> * Using CeleryExecutor with redis broker
> * Building and installing from master using `pip install -e git://...`
> I have a github project that uses Docker and docker-compose to build and test these components.
 [https://github.com/dennisobrien/airflow-docker-compose-example] .
> * pip install line in [Dockerfile|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/Dockerfile#L58]
> * [airflow.cfg template|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/config/airflow.cfg.tpl]
> * [docker-entrypoint.sh|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/bin/docker-entrypoint.sh]
defines how each service is run.
> * [docker-compose.yml|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/docker-compose.yml]
orchestrates the components.
> Let me know if there are additional tests I can do to narrow this down.  I suspect since
others have not complained about this that it is probably something related to my setup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message