airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xianping lin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1511) jobs failed ever in backfill due to concurrency limit
Date Tue, 15 Aug 2017 11:58:00 GMT
Xianping lin created AIRFLOW-1511:
-------------------------------------

             Summary: jobs failed ever in backfill due to concurrency limit
                 Key: AIRFLOW-1511
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1511
             Project: Apache Airflow
          Issue Type: Bug
          Components: backfill
    Affects Versions: Airflow 1.8
            Reporter: Xianping lin


backfill a DAG which supposed to run daily.  some jobs can't start forever.  this is in cluster
mode by celery executor.
current time is : Tue Aug 15 04:56:56 PDT 2017.
Those daily job still can't start. see below messages:

*** Log file isn't local.
*** Fetching here: http://PCDM3:8793/log/user_subs/user_subs_calc/2017-07-08T00:00:00

[2017-08-14 22:11:36,700] {cli.py:391} INFO - Loading pickle id 205
[2017-08-14 22:11:36,818] {base_task_runner.py:112} INFO - Running: ['bash', '-c', u'airflow
run user_subs user_subs_calc 2017-07-08T00:00:00 --pickle 205 --job_id 21317 --raw']
[2017-08-14 22:11:37,723] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:37,723]
{__init__.py:57} INFO - Using executor CeleryExecutor
[2017-08-14 22:11:37,878] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:37,877]
{driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-08-14 22:11:37,911] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:37,911]
{driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2017-08-14 22:11:38,176] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,176]
{cli.py:391} INFO - Loading pickle id 205
[2017-08-14 22:11:38,249] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,249]
{models.py:1126} INFO - Dependencies all met for <TaskInstance: user_subs.user_subs_calc
2017-07-08 00:00:00 [queued]>
[2017-08-14 22:11:38,253] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,253]
{models.py:1120} INFO - Dependencies not met for <TaskInstance: user_subs.user_subs_calc
2017-07-08 00:00:00 [queued]>, dependency 'Task Instance Slots Available' FAILED: The maximum
number of running tasks (user_subs) for this task's DAG '16' has been reached.
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,262]
{models.py:1300} WARNING - 
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: FIXME: Rescheduling due
to concurrency limits reached at task runtime. Attempt 1 of 4. State set to NONE.
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: 
[2017-08-14 22:11:38,263] {base_task_runner.py:95} INFO - Subtask: [2017-08-14 22:11:38,262]
{models.py:1304} INFO - Queuing into pool None
[2017-08-14 22:11:41,822] {jobs.py:2083} INFO - Task exited with return code 0



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message