hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amareshwari Sriramadasu <amar...@yahoo-inc.com>
Subject Re: Does anyone have a working example for using MapFiles on the DistributedCache?
Date Mon, 29 Dec 2008 03:59:56 GMT
Sean Shanny wrote:
> To all,
>
> Version:  hadoop-0.17.2.1-core.jar
>
> I have created a MapFile.
>
> What I don't seem to be able to do is correctly place the MapFile in 
> the DistributedCache and the make use of it in a map method.
>
> I need the following info please:
>
> 1.    How and where to place the MapFile directory so that it is 
> visible to the hadoop job.
You have to place your files in DFS. If it is directory you can place an 
archive of it.
> 2.    How to add the files to the DistributedCache.
You can use DistributedCache.addCacheFile or 
DistributedCache.addCacheArchive.
See more documentation @ 
http://hadoop.apache.org/core/docs/r0.17.2/api/org/apache/hadoop/filecache/DistributedCache.html
and
http://hadoop.apache.org/core/docs/r0.17.2/mapred_tutorial.html#DistributedCache
> 3.    How to create a MapFile.Reader from files in the DistributedCache.
>
I didn't understand what you want to do here. Do you want see the files 
in directory MapFile? or do you want them in classpath etc?
You can use DistributedCache.addFileToClassPath or 
DistributedCache.addArchiveToClassPath

Hope this helps.

Thanks
Amareshwari
> I can get this to work with a local file on a single node system 
> outside of the DistributedCache but for the life of me cannot get it 
> to work within a DistributedCache.
>
> We are trying to load up key value mappings for a Data Warehouse ETL 
> process.  The mapper will take an input record, lookup the keys based 
> on values and emit the resulting key only record.
>
> Happy to answer any questions to help me make this work.
>
> Thanks.
>
> --sean
>
>
>
>


Mime
View raw message