hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Botelho, Andrew" <Andrew.Bote...@emc.com>
Subject RE: New Distributed Cache
Date Thu, 11 Jul 2013 15:19:02 GMT
So in my driver code, I try to store the file in the cache with this line of code:

job.addCacheFile(new URI("file location"));

Then in my Mapper code, I do this to try and access the cached file:

URI[] localPaths = context.getCacheFiles();
File f = new File(localPaths[0]);

However, I get a NullPointerException when I do that in the Mapper code.

Any suggesstions?


From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Wednesday, July 10, 2013 9:43 PM
To: user@hadoop.apache.org
Subject: Re: New Distributed Cache

Also, once you have the array of URIs after calling getCacheFiles  you can iterate over them
using File class or Path (http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/Path.html#Path(java.net.URI))


On Wed, Jul 10, 2013 at 5:08 PM, Omkar Joshi <ojoshi@hortonworks.com<mailto:ojoshi@hortonworks.com>>
did you try JobContext.getCacheFiles() ?

Omkar Joshi
Hortonworks Inc.<http://www.hortonworks.com>

On Wed, Jul 10, 2013 at 10:15 AM, Botelho, Andrew <Andrew.Botelho@emc.com<mailto:Andrew.Botelho@emc.com>>

I am trying to store a file in the Distributed Cache during my Hadoop job.
In the driver class, I tell the job to store the file in the cache with this code:

Job job = Job.getInstance();
job.addCacheFile(new URI("file name"));

That all compiles fine.  In the Mapper code, I try accessing the cached file with this method:

Path[] localPaths = context.getLocalCacheFiles();

However, I am getting warnings that this method is deprecated.
Does anyone know the newest way to access cached files in the Mapper code? (I am using Hadoop

Thanks in advance,


View raw message