hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Q <bill.q....@gmail.com>
Subject Re: New Distributed Cache
Date Thu, 09 Jan 2014 16:10:12 GMT
Hi Omkar,
Did you find out why the getLocalCacheFiles is deprecated? If it is indeed
deprecated, what would be the alternative to use?

On Thursday, July 11, 2013, Omkar Joshi wrote:

> Yeah Andrew.. there seems to be some problem with context.getCacheFiles()
> api which is returning null..
>
>  Path[] cachedFilePaths =
>
>           context.getLocalCacheFiles(); // I am checking why it is
> deprecated...
>
>       for (Path cachedFilePath : cachedFilePaths) {
>
>         File cachedFile = new File(cachedFilePath.toUri().getRawPath());
>
>         System.out.println("cached fie path >> "
>
>             + cachedFile.getAbsolutePath());
>
>       }
>
> I hope this helps for the time being.. JobContext was suppose to replace
> DistributedCache api (it will be deprecated) however there is some problem
> with that or I am missing something... Will reply if I find the solution to
> it.
>
> context.getCacheFiles will give you the uri used for localizing files...
> (original uri used for adding it to cache)... However you can use
> DistributedCache.getCacheFiles() api till context api is fixed.
>
> context.getLocalCacheFiles .. will give you the actual file path on node
> manager... (after file is localized).
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Thu, Jul 11, 2013 at 8:19 AM, Botelho, Andrew <Andrew.Botelho@emc.com>wrote:
>
> So in my driver code, I try to store the file in the cache with this line
> of code:
>
>
>
> job.addCacheFile(new URI("file location"));
>
>
>
> Then in my Mapper code, I do this to try and access the cached file:
>
>
>
> URI[] localPaths = context.getCacheFiles();
>
> File f = new File(localPaths[0]);
>
>
>
> However, I get a NullPointerException when I do that in the Mapper code.
>
>
>
> Any suggesstions?
>
>
>
> Andrew
>
>
>
> *From:* Shahab Yunus [mailto:shahab.yunus@gmail.com]
> *Sent:* Wednesday, July 10, 2013 9:43 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: New Distributed Cache
>
>
>
> Also, once you have the array of URIs after calling getCacheFiles  you
> can iterate over them using File class or Path (
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/Path.html#Path(java.net.URI)
> )
>
>
>
> Regards,
>
> Shahab
>
>
>
> On Wed, Jul 10, 2013 at 5:08 PM, Omkar Joshi <ojoshi@hortonworks.com>
> wrote:
>
> did you try JobContext.getCacheFiles() ?
>
>
>
>
> Thanks,
>
> Omkar Joshi
>
> <http://www.hortonworks.com>
>
>

-- 
Many thanks.


Bill

Mime
View raw message