airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Anand (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-61) Corner case with deadlocked multiprocessing queues
Date Fri, 06 May 2016 16:21:13 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274299#comment-15274299
] 

Siddharth Anand commented on AIRFLOW-61:
----------------------------------------

The more bugs like this that we find (.e.g. scheduling / executing deadlocks, the more I'm
inclined to let the fix bake before cutting a release. In this case, the best place to bake
it would be in Airbnb (in prod) where there is adequate scale to expose the bug and fix. 

I'm curious why it was not in fact seen and reported during the current production bake cycle
at Airbnb. Is it because it shows up via a code path not exercised at Airbnb?

> Corner case with deadlocked multiprocessing queues
> --------------------------------------------------
>
>                 Key: AIRFLOW-61
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-61
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: Airflow 1.7.0
>            Reporter: Jeremiah Lowin
>            Assignee: Jeremiah Lowin
>             Fix For: Airflow 1.7.1
>
>
> I've never experienced this before, but it came up while testing this morning. Joining
a process that puts items in a queue can apparently lead to deadlocks. This happens in the
multiprocessing-enabled scheduler. 
> See for example: https://docs.python.org/3/library/multiprocessing.html#all-start-methods
("Joining processes that use queues") and related issues:
> http://stackoverflow.com/questions/31665328/python-3-multiprocessing-queue-deadlock-when-calling-join-before-the-queue-is-em
> http://stackoverflow.com/questions/31708646/process-join-and-queue-dont-work-with-large-numbers
> http://stackoverflow.com/questions/19071529/python-multiprocessing-125-list-never-finishes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message