airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kaxil Naik (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AIRFLOW-3118) DAGs not successful on new installation
Date Sat, 29 Sep 2018 14:22:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16633028#comment-16633028
] 

Kaxil Naik edited comment on AIRFLOW-3118 at 9/29/18 2:21 PM:
--------------------------------------------------------------

[~ashb] I dug into it. Looks like this bug has been since the inception of Airflow. I tested
it with 1.8.2, 1.9, 1.10 and the latest master. I think it will take more time than I expected.
Won't be working on this for a while, preparing for my 3-hour long presentation on Airflow
in my company :) Feel free to work on this.

The solution should be to change https://github.com/apache/incubator-airflow/blob/7a6f4b013335d42eb7005f49ff202da147b90af1/airflow/jobs.py#L1688-L1711
or the SimpleDAG implementation


{noformat}
            simple_dag_bag = SimpleDagBag(simple_dags)
            if len(simple_dags) > 0:


                # Handle cases where a DAG run state is set (perhaps manually) to
                # a non-running state. Handle task instances that belong to
                # DAG runs in those states


                # If a task instance is up for retry but the corresponding DAG run
                # isn't running, mark the task instance as FAILED so we don't try
                # to re-run it.
                self._change_state_for_tis_without_dagrun(simple_dag_bag,
                                                          [State.UP_FOR_RETRY],
                                                          State.FAILED)
                # If a task instance is scheduled or queued, but the corresponding
                # DAG run isn't running, set the state to NONE so we don't try to
                # re-run it.
                self._change_state_for_tis_without_dagrun(simple_dag_bag,
                                                          [State.QUEUED,
                                                           State.SCHEDULED],
                                                          State.NONE)


                self._execute_task_instances(simple_dag_bag,
                                             (State.SCHEDULED,))


{noformat}




was (Author: kaxilnaik):
[~ashb] I dug into it. Looks like this bug has been since the inception of Airflow. I tested
it with 1.8.2, 1.9, 1.10 and the latest master. I think it will take more time than I expected.
Won't be working on this for a while, preparing for my 3-hour long presentation on Airflow
in my company :) Feel free to work on this.

The solution should be to change https://github.com/apache/incubator-airflow/blob/7a6f4b013335d42eb7005f49ff202da147b90af1/airflow/jobs.py#L1688-L1711


{noformat}
            simple_dag_bag = SimpleDagBag(simple_dags)
            if len(simple_dags) > 0:


                # Handle cases where a DAG run state is set (perhaps manually) to
                # a non-running state. Handle task instances that belong to
                # DAG runs in those states


                # If a task instance is up for retry but the corresponding DAG run
                # isn't running, mark the task instance as FAILED so we don't try
                # to re-run it.
                self._change_state_for_tis_without_dagrun(simple_dag_bag,
                                                          [State.UP_FOR_RETRY],
                                                          State.FAILED)
                # If a task instance is scheduled or queued, but the corresponding
                # DAG run isn't running, set the state to NONE so we don't try to
                # re-run it.
                self._change_state_for_tis_without_dagrun(simple_dag_bag,
                                                          [State.QUEUED,
                                                           State.SCHEDULED],
                                                          State.NONE)


                self._execute_task_instances(simple_dag_bag,
                                             (State.SCHEDULED,))


{noformat}



> DAGs not successful on new installation
> ---------------------------------------
>
>                 Key: AIRFLOW-3118
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DAG
>    Affects Versions: 1.10.0
>         Environment: Ubuntu 18.04
> Python 3.6
>            Reporter: Brylie Christopher Oxley
>            Priority: Blocker
>         Attachments: Screenshot_20180926_161837.png, image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success'
state. They are getting stuck in 'running', or I manually label them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
>  # create new conda environment
>  ** conda create -n airflow
>  ** source activate airflow
>  # install airflow
>  ** pip install apache-airflow
>  # initialize Airflow db
>  ** airflow initdb
>  # disable default paused setting in airflow.cfg
>  ** dags_are_paused_at_creation = False
>  # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color}
>  ** {color:#6a8759}airflow scheduler{color}
>  ** {color:#6a8759}airflow webserver{color}
>  # {color:#6a8759}unpause example_bash_operator{color}
>  ** {color:#6a8759}airflow unpause example_bash_operator{color}
>  # {color:#6a8759}log in to Airflow UI{color}
>  # {color:#6a8759}turn on example_bash_operator{color}
>  # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message