airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bolke de Bruin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (AIRFLOW-934) airflow delayed the task to start
Date Tue, 14 Mar 2017 19:27:41 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Bolke de Bruin resolved AIRFLOW-934.
------------------------------------
    Resolution: Won't Fix

1.6.1. is an unsupported version. Report back if same issue exists on 1.8.0 (rc5)

> airflow delayed the task to start 
> ----------------------------------
>
>                 Key: AIRFLOW-934
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-934
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DagRun
>    Affects Versions: Airflow 1.6.2
>            Reporter: Yajun Dong
>            Priority: Blocker
>
> we have a complex DAG which includes many tasks. but recently we found some tasks start
delayed. for instance: 
> start_task(which will start at 00:00) --> create_cluster(will finished at 00:11) -->
wait_task(start at 00:16) , 
> note: 
> 1. wait_task only has one upstream that is create_cluster 
> 2. the server that the airflow hosts has enough memory, and celeryd_concurrency is 20.

> below is the log of wait_task: 
> [2017-03-02 00:16:39,602] {models.py:124} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:39,603] {models.py:197} INFO - Importing /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:39,613] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:16:40,333] {models.py:124} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:40,333] {models.py:197} INFO - Importing /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:40,345] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:16:40,373] {models.py:936} INFO - 
> --------------------------------------------------------------------------------
> New run starting @2017-03-02T00:16:40.369560
> --------------------------------------------------------------------------------
> [2017-03-02 00:16:40,402] {models.py:951} INFO - Queuing into pool None
> [2017-03-02 00:22:31,161] {models.py:124} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,162] {models.py:197} INFO - Importing /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,172] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:22:31,863] {models.py:124} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,863] {models.py:197} INFO - Importing /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,874] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:22:31,901] {models.py:936} INFO - 
> --------------------------------------------------------------------------------
> New run starting @2017-03-02T00:22:31.897547
> --------------------------------------------------------------------------------
> [2017-03-02 00:22:31,911] {models.py:974} INFO - Executing <Task(BashOperator): wait_mins>
on 2017-03-01 00:00:00
> [2017-03-02 00:22:31,922] {bash_operator.py:52} INFO - tmp dir root location: 
> /tmp



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message