hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Finkelshteyn <iefin...@gmail.com>
Subject Re: Checking Which Filesystem Being Used?
Date Wed, 08 Feb 2012 15:56:48 GMT
It's kind of silly and unnecessary, but I have a task where I needed to 
take an implementation of a recommender in stand-alone Mahout and get it 
to optionally pull files from HDFS. Since nothing is really built to 
support this because if you're using HDFS it's assumed that you'd be 
using Mahout over Hadoop, and because the person who built the 
recommender used local File objects all over the place, which you can't 
really convert to from Hadoop Path objects (and rightfully so), I had to 
hack at things quite a bit.

Anyway, the upshot is that at a couple of points throughout the program, 
I needed to figure out what file system was being used, so I could 
appropriately handle whether I could use File objects directly on a 
given URI or not.

Eli

On 2/7/12 10:47 PM, Harsh J wrote:
> The right way to get a specific filesystem would be to load it with
> the URI itself -- FileSystem.get(URI, conf) for example, and that
> wouldn't go wrong.
>
> Just curious on what you're trying to solve here, that you'd need to
> check the instance?
>
> On Wed, Feb 8, 2012 at 4:19 AM, Eli Finkelshteyn<iefinkel@gmail.com>  wrote:
>> Thanks! I wound up using fs.getUri(), so I can do stuff like
>> fs.getUri().toString().matches("^hdfs://.*").
>>
>>
>> On 2/7/12 5:42 PM, Edward Capriolo wrote:
>>> On Tue, Feb 7, 2012 at 5:24 PM, Eli Finkelshteyn<iefinkel@gmail.com>
>>>   wrote:
>>>
>>>> Hi Folks,
>>>> This might be a stupid question, but I'm new to Java and Hadoop, so...
>>>>
>>>> Anyway, if I want to check what FileSystem is currently being used at
>>>> some
>>>> point (i.e. evaluating FileSystem.get(conf)), what would be the most
>>>> elegant way of doing that? Should I just do something like:
>>>>     if (FileSystem.get(conf) == "HDFS") {...}
>>>> Or is there a better way?
>>>>
>>>> Eli
>>>>
>>> conf.get("fs.default.name") would return a URI such as hdfs://bla:8000 or
>>> file:///this. Although an application could have two Configurations which
>>> could be used to connect to two separate FileSystems inside the same java
>>> application.
>>>
>>> Edward
>>>
>
>


Mime
View raw message