Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 42C0C9B39 for ; Mon, 22 Dec 2014 13:06:52 +0000 (UTC) Received: (qmail 66815 invoked by uid 500); 22 Dec 2014 13:06:46 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 66679 invoked by uid 500); 22 Dec 2014 13:06:46 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 66667 invoked by uid 99); 22 Dec 2014 13:06:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Dec 2014 13:06:45 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of shahab.yunus@gmail.com designates 209.85.217.180 as permitted sender) Received: from [209.85.217.180] (HELO mail-lb0-f180.google.com) (209.85.217.180) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Dec 2014 13:06:20 +0000 Received: by mail-lb0-f180.google.com with SMTP id l4so3870279lbv.11 for ; Mon, 22 Dec 2014 05:04:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=wxd3eIdR4Risp1Y4YAw/UXgYXzOvGLFCLrNs/+6CwX0=; b=OogvFpFBZ70otbEw6wL0EoL/y5zaj4fYXeukAH2YgRrwAiVLen12SNGavrYDITo7ew MefwOHltgfh4nLJQdEw9mGCJP0qnzAkQqorRrW1qTyecA6l5vRk2ookt4+iF5E+eE0iv e8Eouu3DvgR3g4cg3eL8yfyyooM6h4xxUXLJ4bFPbsaxAVpzyykvToESPKLCiFjZGlvm ZJHfQTA98L8NXKREhe8hO9bERZs1eBRI0XqUq/D8VJaHhAfQ/Ng9IQG8TaGGOcTVHp0M uy33HGkfJf1QFVqU0YS96XAiUvSAvkyMiP/9HkEVVLfQbT+UMrMHT/LkAu5Puo77obX/ qNGQ== MIME-Version: 1.0 X-Received: by 10.152.2.165 with SMTP id 5mr21920690lav.40.1419253444615; Mon, 22 Dec 2014 05:04:04 -0800 (PST) Received: by 10.25.39.76 with HTTP; Mon, 22 Dec 2014 05:04:04 -0800 (PST) In-Reply-To: <5497FF05.3040400@nissatech.com> References: <5497E45B.2040407@nissatech.com> <5497FF05.3040400@nissatech.com> Date: Mon, 22 Dec 2014 08:04:04 -0500 Message-ID: Subject: Re: FileNotFoundException in distributed mode From: Shahab Yunus To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=089e01229ba269a5a4050acdb1fa X-Virus-Checked: Checked by ClamAV on apache.org --089e01229ba269a5a4050acdb1fa Content-Type: text/plain; charset=UTF-8 You should not use DistrubutedCache. It is deprecated. See this: http://stackoverflow.com/questions/21239722/hadoop-distributedcache-is-deprecated-what-is-the-preferred-api Regards, Shahab On Mon, Dec 22, 2014 at 6:22 AM, Marko Dinic wrote: > Thanks a lot, it works! > > Do you have any ideas what was the problem? Should this work in fully > distributed mode, or I need to make some modifications? When to use > DistributedCache.addCacheFile and when DIstributedCache.addLocalCacheFile? > > Thanks again! > > On Mon 22 Dec 2014 11:03:36 AM CET, unmesha sreeveni wrote: > >> Driver >> >> Configuration conf = new Configuration(); >> FileSystem fs = FileSystem.get(conf); >> Path cachefile = new Path("path/to/file"); >> FileStatus[] list = fs.globStatus(cachefile); >> for (FileStatus status : list) { >> DistributedCache.addCacheFile(status.getPath().toUri(), conf); >> } >> In setup >> public void setup(Context context) throws IOException{ >> Configuration conf = context.getConfiguration(); >> FileSystem fs = FileSystem.get(conf); >> URI[] cacheFiles = DistributedCache.getCacheFiles(conf); >> Path getPath = new Path(cacheFiles[0].getPath()); >> BufferedReader bf = new BufferedReader(new InputStreamReader(fs.open( >> getPath))); >> String setupData = null; >> while ((setupData = bf.readLine()) != null) { >> System.out.println("Setup Line in reducer "+setupData); >> } >> } >> >> Hope this link helps: >> http://unmeshasreeveni.blogspot.in/2014/10/how-to- >> load-file-in-distributedcache-in.html >> >> On Mon, Dec 22, 2014 at 2:58 PM, Marko Dinic >> > wrote: >> >> Hello Hadoopers, >> >> I'm getting this exception in Hadoop while trying to read file >> that was added to distributed cache, and the strange thing is that >> the file exists on the given location >> >> java.io.FileNotFoundException: File does not exist: >> /tmp/hadoop-pera/mapred/local/__taskTracker/distcache/-__ >> 1517670662102870873_-__1918892372_1898431787/__localhost/work/output/__ >> temporalcentroids/centroids-__iteration0-noOfClusters2/part-__r-00000 >> >> I'm adding the file in before starting my job using >> >> DistributedCache.addCacheFile(__URI.create(args[2]), >> job.getConfiguration()); >> >> And I'm trying to read from the file from setup metod in my mapper >> using >> >> DistributedCache.__getLocalCacheFiles(conf); >> >> As I said, I can confirm that the file is on the local system, but >> the exception is thrown. >> >> I'm running the job in pseudo-distributed mode, on one computer. >> >> Any ideas? >> >> Thanks >> >> >> >> >> -- >> */Thanks & Regards/ * >> /* >> */ >> *Unmesha Sreeveni U.B/ >> /* >> *Hadoop, Bigdata Developer* >> *Centre for Cyber Security | Amrita Vishwa Vidyapeetham* >> http://www.unmeshasreeveni.blogspot.in/ >> >> / >> / >> > --089e01229ba269a5a4050acdb1fa Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You should not use DistrubutedCache. It is deprecated.
See this:

Regards,
=
Shahab

On Mon, Dec 22, 2014 at 6:22 AM, Marko Dinic <marko.dinic@ni= ssatech.com> wrote:
Thanks = a lot, it works!

Do you have any ideas what was the problem? Should this work in fully distr= ibuted mode, or I need to make some modifications? When to use DistributedC= ache.addCacheFile and when DIstributedCache.addLocalCacheFile?

Thanks again!

On Mon 22 Dec 2014 11:03:36 AM CET, unmesha sreeveni wrote:
Driver

Configuration conf =3D new Configuration();
FileSystem fs =3D FileSystem.get(conf);
Path cachefile =3D new Path("path/to/file");
FileStatus[] list =3D fs.globStatus(cachefile);
for (FileStatus status : list) {
=C2=A0 DistributedCache.addCacheFile(status.getPath().toUri(), conf)= ;
}
In setup
public void setup(Context context) throws IOException{
=C2=A0 Configuration conf =3D context.getConfiguration();
=C2=A0 FileSystem fs =3D FileSystem.get(conf);
=C2=A0 URI[] cacheFiles =3D DistributedCache.getCacheFiles(conf); =C2=A0 Path getPath =3D new Path(cacheFiles[0].getPath());
=C2=A0 BufferedReader bf =3D new BufferedReader(new InputStreamReader(fs.op= en(getPath)));
=C2=A0 String setupData =3D null;
=C2=A0 while ((setupData =3D bf.readLine()) !=3D null) {
=C2=A0 =C2=A0 System.out.println("Setup Line in reducer "+setupDa= ta);
=C2=A0 }
}

Hope this link helps:
http://unmeshasreeveni.bl= ogspot.in/2014/10/how-to-load-file-in-distributedcache-in.htm= l

On Mon, Dec 22, 2014 at 2:58 PM, Marko Dinic
 <marko.di= nic@nissatech.com <mailto:marko.dinic@nissatech.com>> wrote:
=C2=A0 =C2=A0 Hello Hadoopers,

=C2=A0 =C2=A0 I'm getting this exception in Hadoop while trying to read= file
=C2=A0 =C2=A0 that was added to distributed cache, and the strange thing is= that
=C2=A0 =C2=A0 the file exists on the given location

=C2=A0 =C2=A0 =C2=A0 =C2=A0 java.io.FileNotFoundException: File does not ex= ist:
 =C2=A0 =C2=A0 /tmp/hadoop-pera/mapred/local/__taskTracker/distcache/= -__1517670662102870873_-__1918892372_1898431787/__loca= lhost/work/output/__temporalcentroids/centroids-__iteration0-= noOfClusters2/part-__r-00000

=C2=A0 =C2=A0 I'm adding the file in before starting my job using

 =C2=A0 =C2=A0 =C2=A0 =C2=A0 DistributedCache.addCacheFile(__URI.crea= te(args[2]),
=C2=A0 =C2=A0 job.getConfiguration());

=C2=A0 =C2=A0 And I'm trying to read from the file from setup metod in = my mapper
=C2=A0 =C2=A0 using

 =C2=A0 =C2=A0 =C2=A0 =C2=A0 DistributedCache.__getLocalCacheFiles(co= nf);

=C2=A0 =C2=A0 As I said, I can confirm that the file is on the local system= , but
=C2=A0 =C2=A0 the exception is thrown.

=C2=A0 =C2=A0 I'm running the job in pseudo-distributed mode, on one co= mputer.

=C2=A0 =C2=A0 Any ideas?

=C2=A0 =C2=A0 Thanks




--
 */Thanks & Regards/ *
/*
*/
*Unmesha Sreeveni U.B/
/*
*Hadoop, Bigdata Developer*
*Centre for Cyber Security | Amrita Vishwa Vidyapeetham*
http:= //www.unmeshasreeveni.blogspot.in/

/
/

--089e01229ba269a5a4050acdb1fa--