hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maha <m...@umail.ucsb.edu>
Subject Re: Open HDFS in mappers
Date Fri, 11 Mar 2011 05:52:31 GMT
Thanks for the reply Harsh as usual :)

Yet, the problem is in the value of the mapper being = hdfs://localhost:9000/tmp/in/file1
, I thought I wasn't using the same HDFS but in fact I was !  using the same idea you presented.

The problem however, is that the map value = hdfs://localhost:9000/tmp/in/file1 is different
than being a String of hdfs://localhost:9000/tmp/in/file1 . The former was taking 44 bytes
where the later took 89 bytes. 

I tried Path p = new Path (value.toString()); 

hdfs.Exists(p); // The answer was no. 

So, I'm doing silly string manipulation to overcome this problem, which I don't appreciate
:( If you have any other idea, let me know.

Thank you,


On Mar 10, 2011, at 6:44 PM, Harsh J wrote:

> Once you have a JobConf/Configuration conf object in your Mapper (via
> setup/configure methods), you can do the following to get the default
> file-system impl:
>    FileSystem fs = FileSystem.get(conf); // Gets the fs.default.name
> file-system impl.
> Then use fs to open/create/etc. any file you need on the set fs.default.name.
> -- 
> Harsh J
> www.harshj.com

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message