hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13726) Enforce that FileSystem initializes only a single instance of the requested FileSystem.
Date Mon, 13 Mar 2017 12:22:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907356#comment-15907356

Steve Loughran commented on HADOOP-13726:

I Think the main concern of Chris (and myself) is not the operations which fail, it's those
the block for a long time before failing.

If it was across all filesystems, then the mechanism would be a disaster: on a shared execution
environment I could submit queries tying to talk to endpoints which blocked during TCP setup,
and that would kill everything, rather than just my query.

If it's just per hash-key, then this can be defended against by: using a large enough initial
table to reduce collision risk (though please, not another config option for now), and making
sure that the hash key used in the table isn't going put everything in the same hash bucket:
want a broad spread of keys

> Enforce that FileSystem initializes only a single instance of the requested FileSystem.
> ---------------------------------------------------------------------------------------
>                 Key: HADOOP-13726
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13726
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Chris Nauroth
> The {{FileSystem}} cache is intended to guarantee reuse of instances by multiple call
sites or multiple threads.  The current implementation does provide this guarantee, but there
is a brief race condition window during which multiple threads could perform redundant initialization.
 If the file system implementation has expensive initialization logic, then this is wasteful.
 This issue proposes to eliminate that race condition and guarantee initialization of only
a single instance.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message