hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: FileSystem.get(Uri,Configuration,String) caching issue
Date Mon, 10 Sep 2012 15:14:15 GMT
What you're seeing is genuine.

You seem to be hitting the abuse scenario described by Daryn here:
https://issues.apache.org/jira/browse/HDFS-3545?focusedCommentId=13398502&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13398502

You can instead choose to skip passing a username for local FS
instances, as I think they don't make much sense when done locally -
as a workaround.

On Mon, Sep 10, 2012 at 4:59 PM, Himanshu Gupta <para_diz@yahoo.com> wrote:
> I am using FileSystem.get(URI uri, Configuration conf, String user) to
> create FileSystem implementation(LocalFileSystem in this case) instances.
> From what I know, FileSystem internally has a cache to retain the objects
> based on uri and user. So if I call FileSystem.get(..) method multiple times
> with same uri and user, then only one instance of LocalFileSystem needs to
> be created and cached. However, I observed(with hadoop-core-1.0.0) that each
> call creates a new instance of LocalFileSystem and puts it in the cache
> leading to memory issues.
>
> Please see the code below and let me know if I am doing something wrong.
>
> Thanks
>
>
> import java.net.URI;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
>
> public class FileSystemCacheIssue {
>
>     private static FileSystem getFileSystem(String user) throws Exception {
>         Configuration conf = new Configuration();
>         conf.set("fs.default.name", "file:///");
>         return FileSystem.get(new URI("file:///"),conf,user);
>     }
>
>     public static void main(String[] args) throws Exception {
>         for(int i = 0; i < 1000; i++) {
>             getFileSystem("himanshg");
>         }
>
>         FileSystem fs = getFileSystem("himanshg");
>         System.out.println(fs.getClass().getCanonicalName());
>
>         //put a breakpoint here and look at the heap dump for number of
> LocalFileSystem
>         //instances, Ideally I expect it to be 1, but there are 1001
>         System.out.println("Keep your debugger here and check.");
>     }
> }



-- 
Harsh J

Mime
View raw message