airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-61) Corner case with deadlocked multiprocessing queues
Date Fri, 06 May 2016 16:11:13 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274285#comment-15274285
] 

Jeremiah Lowin commented on AIRFLOW-61:
---------------------------------------

It came up in testing for one of the items that is blocking 1.7.1 so I'd like to say yes...
but on the other hand no one has ever encountered it before and it appears to be non-deterministic
so it's not critical. In any case [~bolke] has approved the fix (which is relatively minor)
and I'm merging it to master now.

> Corner case with deadlocked multiprocessing queues
> --------------------------------------------------
>
>                 Key: AIRFLOW-61
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-61
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: Airflow 1.7.0
>            Reporter: Jeremiah Lowin
>             Fix For: Airflow 1.7.1
>
>
> I've never experienced this before, but it came up while testing this morning. Joining
a process that puts items in a queue can apparently lead to deadlocks. This happens in the
multiprocessing-enabled scheduler. 
> See for example: https://docs.python.org/3/library/multiprocessing.html#all-start-methods
("Joining processes that use queues") and related issues:
> http://stackoverflow.com/questions/31665328/python-3-multiprocessing-queue-deadlock-when-calling-join-before-the-queue-is-em
> http://stackoverflow.com/questions/31708646/process-join-and-queue-dont-work-with-large-numbers
> http://stackoverflow.com/questions/19071529/python-multiprocessing-125-list-never-finishes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message