spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Davidson (JIRA)" <>
Subject [jira] [Created] (SPARK-1700) PythonRDD leaks socket descriptors during cancellation
Date Fri, 02 May 2014 22:12:18 GMT
Aaron Davidson created SPARK-1700:

             Summary: PythonRDD leaks socket descriptors during cancellation
                 Key: SPARK-1700
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 0.9.0, 1.0.0
            Reporter: Aaron Davidson
            Assignee: Aaron Davidson

Sockets from Spark to Python workers are not cleaned up over the duration of a job, causing
the total number of opened file descriptors to grow to around the number of partitions in
the job. Usually these go away if the job is successful, but in the case of cancellation (and
possibly exceptions, though I haven't investigated), the socket file descriptors remain indefinitely.

This message was sent by Atlassian JIRA

View raw message