airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-84) Queued tasks in Airflow not being prioritized and scheduled
Date Tue, 10 May 2016 03:19:12 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277538#comment-15277538
] 

Jeremiah Lowin commented on AIRFLOW-84:
---------------------------------------

Usually if the Celery command failed it means Airflow couldn't even start the task, or crashed
during task execution. As you suggest, I've seen that happen with out of memory errors causing
the celery workers to fail before running the task.

> Queued tasks in Airflow not being prioritized and scheduled
> -----------------------------------------------------------
>
>                 Key: AIRFLOW-84
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-84
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Kevin Lin
>
> We consistently hit an issue that Airflow didn't schedule any tasks/jobs for several
hours until we restart worker & scheduler.
> We're on 1.6.2 with celeryd enabled.
> From scheduler logs, we didn't observe any suspicious error messages. Other than during
the entire time period (several hours), it's always {quote} Prioritizing 0 queued jobs. {quote}
> While in the worker logs, we did find error messages like following. The message is sort
of generic, so not sure anything quite useful there.
> But wondering what might cause MainProcess keeps failing in celery_executor? e.g. OOM
on the client machine etc.? 
> {quote}
> 2016-05-04T07:44:10.10441 2016-05-04 07:44:10,104: ERROR/MainProcess Task airflow.executors.celery_executor.execute_commanda900d1f1-6abf-4d13-a403-7ec884abd565
raised unexpected: AirflowException('Celery command failed',)
> 2016-05-04T07:44:10.10442 Traceback (most recent call last):
> 2016-05-04T07:44:10.10442 File "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/celery/app/trace.py",
line 240, in trace_task
> 2016-05-04T07:44:10.10442 R = retval = fun(*args, **kwargs)
> 2016-05-04T07:44:10.10442 File "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/celery/app/trace.py",
line 438, in protected_call
> 2016-05-04T07:44:10.10443 return self.run(*args, **kwargs)
> 2016-05-04T07:44:10.10443 File "/usr/local/lib/service_venv/local/lib/python2.7/site-packages/airflow/executors/celery_executor.py",
line 43, in execute_command
> 2016-05-04T07:44:10.10443 raise AirflowException('Celery command failed')
> 2016-05-04T07:44:10.10443 AirflowException: Celery command failed
> {quote} 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message