hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: FileNotFoundException in distributed mode
Date Mon, 22 Dec 2014 13:04:04 GMT
You should not use DistrubutedCache. It is deprecated.

See this:
http://stackoverflow.com/questions/21239722/hadoop-distributedcache-is-deprecated-what-is-the-preferred-api

Regards,
Shahab

On Mon, Dec 22, 2014 at 6:22 AM, Marko Dinic <marko.dinic@nissatech.com>
wrote:

> Thanks a lot, it works!
>
> Do you have any ideas what was the problem? Should this work in fully
> distributed mode, or I need to make some modifications? When to use
> DistributedCache.addCacheFile and when DIstributedCache.addLocalCacheFile?
>
> Thanks again!
>
> On Mon 22 Dec 2014 11:03:36 AM CET, unmesha sreeveni wrote:
>
>> Driver
>>
>> Configuration conf = new Configuration();
>> FileSystem fs = FileSystem.get(conf);
>> Path cachefile = new Path("path/to/file");
>> FileStatus[] list = fs.globStatus(cachefile);
>> for (FileStatus status : list) {
>>   DistributedCache.addCacheFile(status.getPath().toUri(), conf);
>> }
>> In setup
>> public void setup(Context context) throws IOException{
>>   Configuration conf = context.getConfiguration();
>>   FileSystem fs = FileSystem.get(conf);
>>   URI[] cacheFiles = DistributedCache.getCacheFiles(conf);
>>   Path getPath = new Path(cacheFiles[0].getPath());
>>   BufferedReader bf = new BufferedReader(new InputStreamReader(fs.open(
>> getPath)));
>>   String setupData = null;
>>   while ((setupData = bf.readLine()) != null) {
>>     System.out.println("Setup Line in reducer "+setupData);
>>   }
>> }
>>
>> Hope this link helps:
>> http://unmeshasreeveni.blogspot.in/2014/10/how-to-
>> load-file-in-distributedcache-in.html
>>
>> On Mon, Dec 22, 2014 at 2:58 PM, Marko Dinic
>> <marko.dinic@nissatech.com <mailto:marko.dinic@nissatech.com>> wrote:
>>
>>     Hello Hadoopers,
>>
>>     I'm getting this exception in Hadoop while trying to read file
>>     that was added to distributed cache, and the strange thing is that
>>     the file exists on the given location
>>
>>         java.io.FileNotFoundException: File does not exist:
>>     /tmp/hadoop-pera/mapred/local/__taskTracker/distcache/-__
>> 1517670662102870873_-__1918892372_1898431787/__localhost/work/output/__
>> temporalcentroids/centroids-__iteration0-noOfClusters2/part-__r-00000
>>
>>     I'm adding the file in before starting my job using
>>
>>         DistributedCache.addCacheFile(__URI.create(args[2]),
>>     job.getConfiguration());
>>
>>     And I'm trying to read from the file from setup metod in my mapper
>>     using
>>
>>         DistributedCache.__getLocalCacheFiles(conf);
>>
>>     As I said, I can confirm that the file is on the local system, but
>>     the exception is thrown.
>>
>>     I'm running the job in pseudo-distributed mode, on one computer.
>>
>>     Any ideas?
>>
>>     Thanks
>>
>>
>>
>>
>> --
>> */Thanks & Regards/ *
>> /*
>> */
>> *Unmesha Sreeveni U.B/
>> /*
>> *Hadoop, Bigdata Developer*
>> *Centre for Cyber Security | Amrita Vishwa Vidyapeetham*
>> http://www.unmeshasreeveni.blogspot.in/
>>
>> /
>> /
>>
>

Mime
View raw message