airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-61) Corner case with deadlocked multiprocessing queues
Date Fri, 06 May 2016 14:53:12 GMT
Jeremiah Lowin created AIRFLOW-61:
-------------------------------------

             Summary: Corner case with deadlocked multiprocessing queues
                 Key: AIRFLOW-61
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-61
             Project: Apache Airflow
          Issue Type: Bug
          Components: scheduler
    Affects Versions: Airflow 1.7.0
            Reporter: Jeremiah Lowin
             Fix For: Airflow 1.7.1


I've never experienced this before, but it came up while testing this morning. Joining a process
that puts items in a queue can apparently lead to deadlocks. This happens in the multiprocessing-enabled
scheduler. 

See for example: https://docs.python.org/3/library/multiprocessing.html#all-start-methods
("Joining processes that use queues") and related issues:
http://stackoverflow.com/questions/31665328/python-3-multiprocessing-queue-deadlock-when-calling-join-before-the-queue-is-em
http://stackoverflow.com/questions/31708646/process-join-and-queue-dont-work-with-large-numbers
http://stackoverflow.com/questions/19071529/python-multiprocessing-125-list-never-finishes





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message