hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6356) Add a Cache for AbstractFileSystem in the new FileContext/AbstractFileSystem framework.
Date Tue, 05 Oct 2010 23:16:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918313#action_12918313

Sanjay Radia commented on HADOOP-6356:

Dhruba says > I think it is appropriate to do the compatible-version-check for each RPC
instead of doing it per creation-of-FileSystem object.

Having a FileSystem cache => RPCProxy is also cached => version check is done once per
file system object. Dhruba is right in that this is incorrect since the NN may reboot with
a different version.

However we will see a more serious performance problem with FileContext under certain usage
patterns. FileContext keeps a pointer to the default file system; hence all file operations
to the default filesystem will not require a version check (but it is susceptible to the NN
reboot with different verison problem). However operations to other file systems such as fc.open("hdfs://nn/foo/bar)
 will create a new AbstractFileSystem object that will then create a new dfsClient which will
create a new rpcProxy which will do a getVersion() - this is an extra round trip. Good news
is that the connection at the lowest layer is cached. 

We can fix this by having each rpc call also piggy back the version number so that the version
check is not needed when a proxy is created. This however will break wire compatibility. 

> Add a Cache for AbstractFileSystem in the new FileContext/AbstractFileSystem framework.
> ---------------------------------------------------------------------------------------
>                 Key: HADOOP-6356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6356
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
> The new filesystem framework, FileContext and AbstractFileSystem does not implement a
cache for AbstractFileSystem.
> This Jira proposes to add a cache to the new framework just like with the old FileSystem.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message