hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6356) Add a Cache for AbstractFileSystem in the new FileContext/AbstractFileSystem framework.
Date Tue, 03 Nov 2009 01:04:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772808#action_12772808

Sanjay Radia commented on HADOOP-6356:

FileContext keeps a pointer to the default-filesystem (ie the root or the slash-filesystem).
Any methods that pass a URI to a different file system will result in a new instance of the
file system (AbstractFileSystem) being 

So the question is should we add a cache? I filed this Jira to explore this question.

*Option1:* Do *not* add a cache, but do keep a pointer to the default filesystem (ie the slash).
 It is okay to creat a new java object for each URI file system being accessed. The RPC layer
reuses connections to the same HDFS so caching filesystems is not necessary to reuse a connection.
But we need to add a exit hook to close the open leases on JVM exit (note  the old FileSystem
has an exit hook on the cache which indirectly flushes the open leases on exit or on close.)

*Option2:* Add a AbstractFileSystem cache. This raises the following issue. Recently Hadoop-4655
added FileSystem#newInstace() so that Facebook's Scribe subsystem could bypass the cache.
Doing this is a little ugly in general because the notion of the cache is leaking through
the interface; further this is hard to do with FileContext/AbstractFileSystem because applications
do not create instances of AbstractFileSystem directly (FileContext does it automatically
as needed).

> Add a Cache for AbstractFileSystem in the new FileContext/AbstractFileSystem framework.
> ---------------------------------------------------------------------------------------
>                 Key: HADOOP-6356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6356
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
> The new filesystem framework, FileContext and AbstractFileSystem does not implement a
cache for AbstractFileSystem.
> This Jira proposes to add a cache to the new framework just like with the old FileSystem.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message