airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Potiuk (Jira)" <j...@apache.org>
Subject [jira] [Reopened] (AIRFLOW-2511) Subdag failed by scheduler deadlock
Date Wed, 01 Jan 2020 10:39:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jarek Potiuk reopened AIRFLOW-2511:
-----------------------------------

> Subdag failed by scheduler deadlock
> -----------------------------------
>
>                 Key: AIRFLOW-2511
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2511
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: 1.9.0
>            Reporter: Yohei Shimomae
>            Assignee: lufeng
>            Priority: Major
>             Fix For: 1.10.3
>
>
> I am using subdag and sometimes main dag marked failed because of the following error.
In this case, tasks in the subdag stopped.
> {code:java}
> hourly_dag = DAG(
>   hourly_dag_name,
>   default_args=dag_default_args,
>   params=dag_custom_params,
>   schedule_interval=config_values.hourly_job_interval,
>   max_active_runs=2)
> hourly_subdag = SubDagOperator(
>   task_id='s3_to_hive',
>   subdag=LoadFromS3ToHive(
>   hourly_dag,
>   's3_to_hive'),
>   dag=hourly_dag)
> {code}
> I got this error in main dag. bug in scheduler?
> {code:java}
> [2018-05-22 21:52:19,683] {models.py:1595} ERROR - This Session's transaction has been
rolled back due to a previous exception during flush. To begin a new transaction with this
Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError)
(1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'UPDATE
task_instance SET state=%s WHERE task_instance.task_id = %s AND task_instance.dag_id = %s
AND task_instance.execution_date = %s'] [parameters: ('queued', 'transfer_from_tmp_table_into_cleaned_table',
'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive', datetime.datetime(2018, 5, 7,
5, 2))] (Background on this error at: http://sqlalche.me/e/e3q8)
> Traceback (most recent call last):
> sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due
to a previous exception during flush. To begin a new transaction with this Session, first
issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (1213,
'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'UPDATE task_instance
SET state=%s WHERE task_instance.task_id = %s AND task_instance.dag_id = %s AND task_instance.execution_date
= %s'] [parameters: ('queued', 'transfer_from_tmp_table_into_cleaned_table', 'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive',
datetime.datetime(2018, 5, 7, 5, 2))] (Background on this error at: http://sqlalche.me/e/e3q8)
> [2018-05-22 21:52:19,687] {models.py:1624} INFO - Marking task as FAILED.
> [2018-05-22 21:52:19,688] {base_task_runner.py:98} INFO - Subtask: [2018-05-22 21:52:19,688]
{slack_hook.py:143} INFO - Message is prepared: 
> [2018-05-22 21:52:19,688] {base_task_runner.py:98} INFO - Subtask: {"attachments": [{"color":
"danger", "text": "", "fields": [{"title": "DAG", "value": "<https://data-platform-airflow-tokyo.data.fastretailing.com:8080/admin/airflow/tree?dag_id=rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h|rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h>",
"short": true}, {"title": "Owner", "value": "airflow", "short": true}, {"title": "Task", "value":
"s3_to_hive", "short": false}, {"title": "Status", "value": "FAILED", "short": false}, {"title":
"Execution Time", "value": "2018-05-07T05:02:00", "short": true}, {"title": "Duration", "value":
"826.305929", "short": true}, {"value": "<https://data-platform-airflow-tokyo.data.fastretailing.com:8080/admin/airflow/log?dag_id=rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h&task_id=s3_to_hive&execution_date=2018-05-07T05:02:00|View
Task Log>", "short": false}]}]}
> [2018-05-22 21:52:19,688] {models.py:1638} ERROR - Failed at executing callback
> [2018-05-22 21:52:19,688] {models.py:1639} ERROR - This Session's transaction has been
rolled back due to a previous exception during flush. To begin a new transaction with this
Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError)
(1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: 'UPDATE
task_instance SET state=%s WHERE task_instance.task_id = %s AND task_instance.dag_id = %s
AND task_instance.execution_date = %s'] [parameters: ('queued', 'transfer_from_tmp_table_into_cleaned_table',
'rfid_warehouse_carton_wh_g_dl_dwh_csv_uqjp_1h.s3_to_hive', datetime.datetime(2018, 5, 7,
5, 2))] (Background on this error at: http://sqlalche.me/e/e3q8)
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message