hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Himanshu Gupta <para_...@yahoo.com>
Subject Re: FileSystem.get(Uri,Configuration,String) caching issue
Date Tue, 11 Sep 2012 09:15:13 GMT
Thanks Harsh and Daryn !

Daryn, I tried the option you suggested and changed getFileSystem(String user) implementation
as following...

    private static FileSystem getFileSystem(String user) throws Exception {
        final Configuration conf = new Configuration();
        conf.set("fs.default.name", "file:///");
        UserGroupInformation ugi = UserGroupInformation.createRemoteUser(user);
        return ugi.doAs(new PrivilegedAction<FileSystem>() {

            public FileSystem run() {
                try {
                 return FileSystem.get(new URI("file:///"),conf);
                } catch(Exception ex) {
                    return null;

but with this implementation also multiple FileSystem instances are created. Did I miss something?

As per as my usecase is concerned, we are developing a oozie like application where clients
can give a workflow and their hadoop user and also store data on hdfs through us. We are using
FileSystem.get(..) api to get handle to FileSystem impl specific to the client's hadoop user.


----- Original Message -----
From: Daryn Sharp <daryn@yahoo-inc.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Cc: Himanshu Gupta <para_diz@yahoo.com>
Sent: Tuesday, 11 September 2012 12:49 AM
Subject: Re: FileSystem.get(Uri,Configuration,String) caching issue

Yes, the sample code demonstrates what happens when you use a new UGI for every FileSystem.get. 
If possible you should avoid the variant of fs.get that accepts the user as a string since
it may create another UGI from the string user on every call.  The cache will fill with instances
for every new UGI.  

If you need a filesystem for your kerberos or unix user, don't bother with passing the user
or manipulating the UGI since the default for fs.get uses the current user ugi.  If you are
trying to be another user, use UGI.createRemoteUser and use it to execute your code within
a doAs block.  All the fs gets will return the same cached object as long as you don't pass
in the user string.

Normally only daemons that accept requests for arbitrary users have to deal with the ugi. 
If this is simple app code then I'd suggest leaving the ugi alone.  If you have a complex
use case, please describe it to us so better advice can be offered.  I hope this helps!


On Sep 10, 2012, at 10:14 AM, Harsh J wrote:

> What you're seeing is genuine.
> You seem to be hitting the abuse scenario described by Daryn here:
> https://issues.apache.org/jira/browse/HDFS-3545?focusedCommentId=13398502&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13398502
> You can instead choose to skip passing a username for local FS
> instances, as I think they don't make much sense when done locally -
> as a workaround.
> On Mon, Sep 10, 2012 at 4:59 PM, Himanshu Gupta <para_diz@yahoo.com> wrote:
>> I am using FileSystem.get(URI uri, Configuration conf, String user) to
>> create FileSystem implementation(LocalFileSystem in this case) instances.
>> From what I know, FileSystem internally has a cache to retain the objects
>> based on uri and user. So if I call FileSystem.get(..) method multiple times
>> with same uri and user, then only one instance of LocalFileSystem needs to
>> be created and cached. However, I observed(with hadoop-core-1.0.0) that each
>> call creates a new instance of LocalFileSystem and puts it in the cache
>> leading to memory issues.
>> Please see the code below and let me know if I am doing something wrong.
>> Thanks
>> import java.net.URI;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FileSystem;
>> public class FileSystemCacheIssue {
>>    private static FileSystem getFileSystem(String user) throws Exception {
>>        Configuration conf = new Configuration();
>>        conf.set("fs.default.name", "file:///");
>>        return FileSystem.get(new URI("file:///"),conf,user);
>>    }
>>    public static void main(String[] args) throws Exception {
>>        for(int i = 0; i < 1000; i++) {
>>            getFileSystem("himanshg");
>>        }
>>        FileSystem fs = getFileSystem("himanshg");
>>        System.out.println(fs.getClass().getCanonicalName());
>>        //put a breakpoint here and look at the heap dump for number of
>> LocalFileSystem
>>        //instances, Ideally I expect it to be 1, but there are 1001
>>        System.out.println("Keep your debugger here and check.");
>>    }
>> }
> -- 
> Harsh J
View raw message