airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Crawford (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1482) Error when try to backfill the example_trigger_controller_dag
Date Sun, 23 Sep 2018 11:10:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16625069#comment-16625069
] 

Michael Crawford commented on AIRFLOW-1482:
-------------------------------------------

Ran into this issue as well.  It appears to be a pickle issue (atleast in our case)  We
were able to work around it by turning donot_pickle to True.

 

We only had this issue when using celery.  

> Error when try to backfill the example_trigger_controller_dag
> -------------------------------------------------------------
>
>                 Key: AIRFLOW-1482
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1482
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: backfill
>    Affects Versions: 1.8.1, 1.8.2
>         Environment: Ubuntu: 16.04
> Python: 2.7
> CeleryExecutor
> Broker: Redis
>            Reporter: Timothee N
>            Priority: Blocker
>         Attachments: airflow_1.png, airflow_2.png, airflow_3.png
>
>
> Hello,
> Running a backfill command for the {noformat}example_trigger_controller_dag{noformat}
example dag, result in the failed task {noformat}test_trigger_dagrun{noformat}
> It seems to me that the problem comes from the TriggerDagRunOperator in the example_trigger_controller_dag
?
> Backfill command: {noformat}airflow backfill -s 2017-07-10 -e 2017-07-13 --pool backfill
example_trigger_controller_dag{noformat}
> Tested in 1.8.1 and 1.8.2rc1
> Here is the output log from the backfill command :
> {noformat}
> [2017-08-02 13:53:00,844] {__init__.py:57} INFO - Using executor CeleryExecutor
> [2017-08-02 13:53:00,888] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-08-02 13:53:00,902] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> /var/lib/airflow/local/lib/python2.7/site-packages/airflow/www/app.py:23: FlaskWTFDeprecationWarning:
"flask_wtf.CsrfProtect" has been renamed to "CSRFProtect" and will be removed in 1.0.
>   csrf = CsrfProtect()
> [2017-08-02 13:53:01,033] {models.py:168} INFO - Filling up the DagBag from /var/lib/airflow/dags
> [2017-08-02 13:53:01,332] {models.py:1128} INFO - Dependencies all met for <TaskInstance:
example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [scheduled]>
> [2017-08-02 13:53:01,337] {base_executor.py:50} INFO - Adding to queue: airflow run example_trigger_controller_dag
test_trigger_dagrun 2017-07-10T00:00:00 --pickle 1 --local --pool backfill
> [2017-08-02 13:53:06,267] {celery_executor.py:81} INFO - [celery] queuing (u'example_trigger_controller_dag',
u'test_trigger_dagrun', datetime.datetime(2017, 7, 10, 0, 0)) through celery, queue=default
> [2017-08-02 13:53:06,330] {models.py:4164} INFO - Updating state for <DagRun example_trigger_controller_dag
@ 2017-07-10 00:00:00: backfill_2017-07-10T00:00:00, externally triggered: False> considering
1 task(s)
> [2017-08-02 13:53:06,334] {jobs.py:2020} INFO - [backfill progress] | finished run 0
of 1 | tasks waiting: 0 | succeeded: 0 | kicked_off: 1 | failed: 0 | skipped: 0 | deadlocked:
0 | not ready: 0
> [2017-08-02 13:53:11,273] {jobs.py:1743} ERROR - Executor reports task instance <TaskInstance:
example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [queued]> finished
(failed) although the task says its queued. Was the task killed externally?
> [2017-08-02 13:53:11,273] {models.py:1433} ERROR - Executor reports task instance <TaskInstance:
example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [queued]> finished
(failed) although the task says its queued. Was the task killed externally?
> None
> [2017-08-02 13:53:11,273] {models.py:1457} INFO - Marking task as FAILED.
> [2017-08-02 13:53:11,279] {models.py:1478} ERROR - Executor reports task instance <TaskInstance:
example_trigger_controller_dag.test_trigger_dagrun 2017-07-10 00:00:00 [queued]> finished
(failed) although the task says its queued. Was the task killed externally?
> [2017-08-02 13:53:11,281] {jobs.py:1694} ERROR - Task instance <TaskInstance: example_trigger_controller_dag.test_trigger_dagrun
2017-07-10 00:00:00 [failed]> failed
> [2017-08-02 13:53:11,283] {models.py:4164} INFO - Updating state for <DagRun example_trigger_controller_dag
@ 2017-07-10 00:00:00: backfill_2017-07-10T00:00:00, externally triggered: False> considering
1 task(s)
> [2017-08-02 13:53:11,285] {models.py:4204} INFO - Marking run <DagRun example_trigger_controller_dag
@ 2017-07-10 00:00:00: backfill_2017-07-10T00:00:00, externally triggered: False> failed
> [2017-08-02 13:53:11,298] {jobs.py:2020} INFO - [backfill progress] | finished run 1
of 1 | tasks waiting: 0 | succeeded: 0 | kicked_off: 0 | failed: 1 | skipped: 0 | deadlocked:
0 | not ready: 0
> Traceback (most recent call last):
>   File "/var/lib/airflow/bin/airflow", line 28, in <module>
>     args.func(args)
>   File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/bin/cli.py", line
167, in backfill
>     pool=args.pool)
>   File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/models.py", line 3373,
in run
>     job.run()
>   File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/jobs.py", line 201,
in run
>     self._execute()
>   File "/var/lib/airflow/local/lib/python2.7/site-packages/airflow/jobs.py", line 2063,
in _execute
>     raise AirflowException(err)
> airflow.exceptions.AirflowException: ---------------------------------------------------
> Some task instances failed:
> set([(u'example_trigger_controller_dag', u'test_trigger_dagrun', datetime.datetime(2017,
7, 10, 0, 0))])
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message