Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 81727DBAD for ; Tue, 27 Nov 2012 05:23:10 +0000 (UTC) Received: (qmail 66874 invoked by uid 500); 27 Nov 2012 05:23:05 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 66638 invoked by uid 500); 27 Nov 2012 05:23:04 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 66621 invoked by uid 99); 27 Nov 2012 05:23:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Nov 2012 05:23:04 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of balijamahesh.mca@gmail.com designates 209.85.216.41 as permitted sender) Received: from [209.85.216.41] (HELO mail-qa0-f41.google.com) (209.85.216.41) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Nov 2012 05:22:47 +0000 Received: by mail-qa0-f41.google.com with SMTP id c26so4039020qad.14 for ; Mon, 26 Nov 2012 21:22:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=VPU6VYowEkinSEpCDq3Tols0Km9f6rgNlobOxyTIsNA=; b=kcC2g9PCl/sLTgPqG17BhKwOD2MJtvcz0M0demnEhkNcX+L3aMhP7oE3lYBOMOo2et IYKU9xm2IZi/Fknc+l9tdPxMpF2amKrSxawt4Gk2dKm1meB/kwTxrx6NSsLg1E61HSBp sk4efmX1rltIvrwlCb4zJ0hu2dY5gIxZgU+2gWoQcJ26NOD2j1VbXzk/m2jhWocmaLf8 VLAqEGGYDNBDV1HU647trCEyufHYT6OMi/hlA3jZGvcTTxN4krn9cWAVY4vojWl1i78B prEnrk5laVImnJRneVGIOVoOZKstqphJwU15VFgf/Iqxz+GV8A6NjmpFk065+OEJ/bzv nxMw== MIME-Version: 1.0 Received: by 10.229.106.202 with SMTP id y10mr3403485qco.134.1353993746007; Mon, 26 Nov 2012 21:22:26 -0800 (PST) Received: by 10.49.60.69 with HTTP; Mon, 26 Nov 2012 21:22:25 -0800 (PST) In-Reply-To: References: Date: Tue, 27 Nov 2012 10:52:25 +0530 Message-ID: Subject: Re: MapReduce APIs From: Mahesh Balija To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0023544712d042466104cf733c86 X-Virus-Checked: Checked by ClamAV on apache.org --0023544712d042466104cf733c86 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi AK, I don't really understand what is stopping you to use the job.getConfiguration() method to pass the configuration instance to DistributedCache.addCacheFile(URI, job.getConfiguration()). Only thing you need to do is pass the URI and configuration object (getting it from org.apache.hadoop.mapreduce.Job instance). Best, Mahesh.B. Calsoft Labs. On Mon, Nov 26, 2012 at 8:18 PM, Kartashov, Andy wr= ote: > Harsh, > > Thanks for the " DistributedCache.addCacheFile(URI, > job.getConfiguration());" suggestion. > What class is your instance job belongs to? It is not Job class, for sure= . > So must be JobContext? > > When I write my driver using new API I write: > > Job job =3D new Job(); > job.setJarByClass(.... > job.setJobName(... > job.setSetMapOutputKey... | .. value > ......Redeuce..... > > > So, how can I use your piece of code here, i.e? > DistributedCache.addCacheFile(URI, job.getConfiguration()); > > How can I wire JobConf to Job instances? > > Thanks, > AK > > -----Original Message----- > From: Harsh J [mailto:harsh@cloudera.com] > Sent: Saturday, November 24, 2012 2:22 AM > To: > Subject: Re: MapReduce APIs > > You could use the org.apache.hadoop.filecache.DistributedCache API as: > > DistributedCache.addCacheFile(URI, job.getConfiguration()); > > On Sat, Nov 24, 2012 at 3:06 AM, Kartashov, Andy > wrote: > > Guys, > > > > > > > > I know that there is old and new API for MapReduce. The old API is > > found under org.apache.hadoop.mapred and the new is under > > org.apache.hadoop.mapreduce > > > > > > > > I successfully used both (the old and the new API) writing my > > MapReduce drivers. > > > > > > > > The problem came up when I tried to use distributed cache. My new API > > Job object could not locate > > > > public void addCacheFile(URI uri) method and I was scratching my head > why. > > > > > > > > What I did not reaslise is that despite new and oold API there is also > > Hadoop 0.20 vs Hadoop 2.0.0 APIs that use exact same packages. > > > > The old Hadoop.0.20.00 new Mapreduce API class Job simply doesn't > > have that method "addCacheFile(URI uri)". > > > > > > > > I am running Hadoop 2.0.0. so could not understand why the method was > > not inside the class. I ended up rewriting MR job under old API mapred > > package and ran soccessfully. > > > > > > > > Can anyone shed some light on this? > > > > > > > > Thanks > > > > AK > > > > > > > > > > > > NOTICE: This e-mail message and any attachments are confidential, > > subject to copyright and may be privileged. Any unauthorized use, > > copying or disclosure is prohibited. If you are not the intended > > recipient, please delete and contact the sender immediately. Please > > consider the environment before printing this e-mail. AVIS : le > > pr=E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont > > confidentiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre cou= verts > > par le secret professionnel. Toute utilisation, copie ou divulgation > > non autoris=E9e est interdite. Si vous n'=EAtes pas le destinataire pr= =E9vu de > ce courriel, supprimez-le et contactez imm=E9diatement l'exp=E9diteur. > > Veuillez penser =E0 l'environnement avant d'imprimer le pr=E9sent courr= iel > > > > -- > Harsh J > NOTICE: This e-mail message and any attachments are confidential, subject > to copyright and may be privileged. Any unauthorized use, copying or > disclosure is prohibited. If you are not the intended recipient, please > delete and contact the sender immediately. Please consider the environmen= t > before printing this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8c= e > jointe qui l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'a= uteur > et peuvent =EAtre couverts par le secret professionnel. Toute utilisation= , > copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas = le > destinataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diat= ement > l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le p= r=E9sent > courriel > --0023544712d042466104cf733c86 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi AK,

=A0=A0=A0=A0=A0=A0=A0=A0=A0 I don't really understand wha= t is stopping you to use the job.getConfiguration() method to pass the conf= iguration instance to DistributedCache.addCacheFile(URI, job.getConfigurati= on()).
=A0=A0=A0=A0=A0=A0=A0=A0=A0 Only thing you need to do is pass th= e URI and configuration object (getting it from org.apach= e.hadoop.mapreduce.Job instance).

Best,
Mahesh.B.
Calsoft Labs.

O= n Mon, Nov 26, 2012 at 8:18 PM, Kartashov, Andy <Andy.Kartashov@mpac.= ca> wrote:
Harsh,

Thanks for the " DistributedCache.addCacheFile(URI, job.getConfigurati= on());" suggestion.
What class is your instance job belongs to? It is not Job class, for sure. = So must be JobContext?

When I write my driver using new API I write:

Job job =3D new Job();
job.setJarByClass(....
job.setJobName(...
job.setSetMapOutputKey... | .. value
......Redeuce.....


So, how can I use your =A0piece of code here, i.e?
DistributedCache.addCacheFile(URI, job.getConfiguration());

How can I wire JobConf to Job instances?

Thanks,
AK

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.= com]
Sent: Saturday, November 24, 2012 2:22 AM
To: <user@hadoop.apache.org>
Subject: Re: MapReduce APIs

You could use the org.apache.hadoop.filecache.DistributedCache API as:

DistributedCache.addCacheFile(URI, job.getConfiguration());

On Sat, Nov 24, 2012 at 3:06 AM, Kartashov, Andy <Andy.Kartashov@mpac.ca> wrote:
> Guys,
>
>
>
> I know that there is old and new API for MapReduce. The old API is
> found under org.apache.hadoop.mapred and the new is under
> org.apache.hadoop.mapreduce
>
>
>
> I successfully used =A0both (the old and the new API) writing my
> MapReduce drivers.
>
>
>
> The problem came up when I tried to use distributed cache. My new API<= br> > Job object could not locate
>
> public void addCacheFile(URI uri) method and I was scratching my head = why.
>
>
>
> What I did not reaslise is that despite new and oold API there is also=
> Hadoop 0.20 vs Hadoop 2.0.0 APIs that use exact same packages.
>
> =A0The old Hadoop.0.20.00 new Mapreduce API class Job simply doesn'= ;t
> have that method "addCacheFile(URI uri)".
>
>
>
> I am running Hadoop 2.0.0. so could not understand why the method was<= br> > not inside the class. I ended up rewriting MR job under old API mapred=
> package and ran soccessfully.
>
>
>
> Can anyone shed some light on this?
>
>
>
> Thanks
>
> AK
>
>
>
>
>
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use,
> copying or disclosure is prohibited. If you are not the intended
> recipient, please delete and contact the sender immediately. Please > consider the environment before printing this e-mail. AVIS : le
> pr=E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont > confidentiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtr= e couverts
> par le secret professionnel. Toute utilisation, copie ou divulgation > non autoris=E9e est interdite. Si vous n'=EAtes pas le destinatair= e pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement l'e= xp=E9diteur.
> Veuillez penser =E0 l'environnement avant d'imprimer le pr=E9s= ent courriel



--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before pri= nting this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8ce jointe qui= l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'auteu= r et peuvent =EAtre couverts par le secret professionnel. Toute utilisation= , copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes = pas le destinataire pr=E9vu de ce courriel, supprimez-le et contactez imm= =E9diatement l'exp=E9diteur. Veuillez penser =E0 l'environnement av= ant d'imprimer le pr=E9sent courriel

--0023544712d042466104cf733c86--