airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] potiuk opened a new pull request #6988: [AIRFLOW-2516] Fix mysql deadlocks
Date Wed, 01 Jan 2020 09:50:22 GMT
potiuk opened a new pull request #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988
 
 
   Deadlocks were occuring in mysql when task_instance was modified
   by two queries at the same time. One query used state as selection
   criteria and updated it in the same query where second query just
   updated the state for the same table. The first query locked state
   index first and primary index afterwards, the second query locked
   primary index first and state afterwards - leading to deadlocks.
   
   This change splits the first query into two independent ones.
   First query makes select FOR UPDATE and selects all the task
   instances to act on (this will lock primary index only)
   and second updates all affected task instances.
   
   Note that performance impact for that is neglectable because this
   query is only run once every scheduler loop and the second part
   of it (looping through task instances) will only happen in case
   there are some manually modified DagRun states - so it is only
   run to correct some wrong states of DagRun. This should happen
   very infrequently.
   
   - [x] Description above provides context of the change
   - [x] Commit message contains [\[AIRFLOW-XXXX\]](https://issues.apache.org/jira/browse/AIRFLOW-XXXX)
or `[AIRFLOW-XXXX]` for document-only changes
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message