hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Harris (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-12412) Concurrency in FileSystem$Cache is very broken
Date Mon, 14 Sep 2015 21:05:46 GMT
Michael Harris created HADOOP-12412:

             Summary: Concurrency in FileSystem$Cache is very broken
                 Key: HADOOP-12412
                 URL: https://issues.apache.org/jira/browse/HADOOP-12412
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 2.7.0
            Reporter: Michael Harris
            Assignee: Michael Harris
            Priority: Critical

The FileSystem cache uses a mild amount of concurrency to protect the cache itself, but does
nothing to prevent multiple of the same filesystem from being constructed and initialized
simultaneously.  At best, this leads to potentially expensive wasted work.  At worst, as is
the case for Spark, it can lead to deadlocks/livelocks, especially when the same configuration
object is passed into both calls.  This should be refactored to use a results cache approach
(reference Java Concurrency in Practice chapter 5 section 6 for an example of how to do this
correctly), which will be both higher-performance and safer.

This message was sent by Atlassian JIRA

View raw message