Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C659AEBBE for ; Tue, 27 Nov 2012 14:51:27 +0000 (UTC) Received: (qmail 42731 invoked by uid 500); 27 Nov 2012 14:51:22 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 42392 invoked by uid 500); 27 Nov 2012 14:51:22 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 42365 invoked by uid 99); 27 Nov 2012 14:51:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Nov 2012 14:51:21 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.216.48] (HELO mail-qa0-f48.google.com) (209.85.216.48) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Nov 2012 14:51:15 +0000 Received: by mail-qa0-f48.google.com with SMTP id s11so4686965qaa.14 for ; Tue, 27 Nov 2012 06:50:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:content-type:x-gm-message-state; bh=ydMYY1sVfEnjxi8EG/gLyQVY8QveNIgwoAITbD1PEs8=; b=TOkeOZ0+jrG6hbGYceS4AiCq09JnIJmWhO7xGSv4VfsUCJKSDTD8fd2ZEnc1EjcC6Q 6736KntOIhhj78+HLAXYrf/epk+oXTrCxQ6asCO153WNJmnMjRisAN7C5jT4JPABTQ/I HGwTkqM4WvaeFu9LrAHntBKWx8mU7E6WlmNSnAFY970Rmh4Tfr6ApWdYXgkwfIu/Gr7d rt0GiIeWDyYUv4ybvhRArmVmdv+6Vofp/g5aiRB9vC70ErBSmwqLbEXjYDUb62+WcAqq Rv2BlzVVG3MjNtC6rmwka8qqJ7IdLXE+gMT7/l/bY04TI3/SfUhgi5GGwAHNOJ/iKU/O Z7Fw== MIME-Version: 1.0 Received: by 10.224.42.80 with SMTP id r16mr16437102qae.90.1354027852912; Tue, 27 Nov 2012 06:50:52 -0800 (PST) Received: by 10.49.29.132 with HTTP; Tue, 27 Nov 2012 06:50:52 -0800 (PST) X-Originating-IP: [143.252.1.30] In-Reply-To: References: Date: Tue, 27 Nov 2012 14:50:52 +0000 Message-ID: Subject: Re: MapReduce APIs From: Dave Beech To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf3074d6243057df04cf7b2dc7 X-Gm-Message-State: ALoCoQkczvJsAOcm1A07hsyD/jlgqDCtjBcgwf9SAqqw+UoD2BsAdyzsjVp86ZZMrqUlBK4fjpaP X-Virus-Checked: Checked by ClamAV on apache.org --20cf3074d6243057df04cf7b2dc7 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable AK - look again at that javadoc. Job does a have getConfiguration() method. You may have missed it the first time because it's inherited from a parent class, JobContext. On 27 November 2012 14:23, Kartashov, Andy wrote: > Thank man for the response. Much appreciated. > > > > Why? Because Job object doesn=92t have the below method getConfiguration(= ). > See for yourself under mapreduce.Job: > > http://hadoop.apache.org/docs/r0.20.2/api/index.html or > > http://hadoop.apache.org/docs/current/api/index.html > > > > So, back to my original question. If *job* isn=92t instance of *Job*objec= t, then what object does it belong to in the following statement > (DistributedCache.addCacheFile(URI, *job*.getConfiguration()); > > ** > > * * > > > > *From:* Mahesh Balija [mailto:balijamahesh.mca@gmail.com] > *Sent:* Tuesday, November 27, 2012 12:22 AM > > *To:* user@hadoop.apache.org > *Subject:* Re: MapReduce APIs > > > > Hi AK, > > I don't really understand what is stopping you to use the > job.getConfiguration() method to pass the configuration instance to > DistributedCache.addCacheFile(URI, job.getConfiguration()). > Only thing you need to do is pass the URI and configuration > object (getting it from org.apache.hadoop.mapreduce.Job instance). > > Best, > Mahesh.B. > Calsoft Labs. > > On Mon, Nov 26, 2012 at 8:18 PM, Kartashov, Andy > wrote: > > Harsh, > > Thanks for the " DistributedCache.addCacheFile(URI, > job.getConfiguration());" suggestion. > What class is your instance job belongs to? It is not Job class, for sure= . > So must be JobContext? > > When I write my driver using new API I write: > > *Job* *job* =3D new * Job*(); > job.setJarByClass(.... > job.setJobName(... > job.setSetMapOutputKey... | .. value > ......Redeuce..... > > > So, how can I use your piece of code here, i.e? > DistributedCache.addCacheFile(URI, job.getConfiguration()); > > How can I wire JobConf to Job instances? > > Thanks, > AK > > > -----Original Message----- > From: Harsh J [mailto:harsh@cloudera.com] > Sent: Saturday, November 24, 2012 2:22 AM > To: > Subject: Re: MapReduce APIs > > You could use the org.apache.hadoop.filecache.DistributedCache API as: > > DistributedCache.addCacheFile(URI, job.getConfiguration()); > > On Sat, Nov 24, 2012 at 3:06 AM, Kartashov, Andy > wrote: > > Guys, > > > > > > > > I know that there is old and new API for MapReduce. The old API is > > found under org.apache.hadoop.mapred and the new is under > > org.apache.hadoop.mapreduce > > > > > > > > I successfully used both (the old and the new API) writing my > > MapReduce drivers. > > > > > > > > The problem came up when I tried to use distributed cache. My new API > > Job object could not locate > > > > public void addCacheFile(URI uri) method and I was scratching my head > why. > > > > > > > > What I did not reaslise is that despite new and oold API there is also > > Hadoop 0.20 vs Hadoop 2.0.0 APIs that use exact same packages. > > > > The old Hadoop.0.20.00 new Mapreduce API class Job simply doesn't > > have that method "addCacheFile(URI uri)". > > > > > > > > I am running Hadoop 2.0.0. so could not understand why the method was > > not inside the class. I ended up rewriting MR job under old API mapred > > package and ran soccessfully. > > > > > > > > Can anyone shed some light on this? > > > > > > > > Thanks > > > > AK > > > > > > > > > > > > NOTICE: This e-mail message and any attachments are confidential, > > subject to copyright and may be privileged. Any unauthorized use, > > copying or disclosure is prohibited. If you are not the intended > > recipient, please delete and contact the sender immediately. Please > > consider the environment before printing this e-mail. AVIS : le > > pr=E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont > > confidentiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre cou= verts > > par le secret professionnel. Toute utilisation, copie ou divulgation > > non autoris=E9e est interdite. Si vous n'=EAtes pas le destinataire pr= =E9vu de > ce courriel, supprimez-le et contactez imm=E9diatement l'exp=E9diteur. > > Veuillez penser =E0 l'environnement avant d'imprimer le pr=E9sent courr= iel > > > > -- > Harsh J > NOTICE: This e-mail message and any attachments are confidential, subject > to copyright and may be privileged. Any unauthorized use, copying or > disclosure is prohibited. If you are not the intended recipient, please > delete and contact the sender immediately. Please consider the environmen= t > before printing this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8c= e > jointe qui l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'a= uteur > et peuvent =EAtre couverts par le secret professionnel. Toute utilisation= , > copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas = le > destinataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diat= ement > l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le p= r=E9sent > courriel > > > NOTICE: This e-mail message and any attachments are confidential, subjec= t > to copyright and may be privileged. Any unauthorized use, copying or > disclosure is prohibited. If you are not the intended recipient, please > delete and contact the sender immediately. Please consider the environmen= t > before printing this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8c= e > jointe qui l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'a= uteur > et peuvent =EAtre couverts par le secret professionnel. Toute utilisation= , > copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas = le > destinataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diat= ement > l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le p= r=E9sent > courriel > --20cf3074d6243057df04cf7b2dc7 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable AK - look again at that javadoc. Job does a have getConfiguration() method.= You may have missed it the first time because it's inherited from a pa= rent class, JobContext.=A0


On 27 November 2012 14:23, Kartashov, Andy <Andy.Kartashov@mpac.ca> wrote:

Thank man for the respons= e. Much appreciated.

=A0

Why? Because Job object d= oesn=92t have the below method getConfiguration(). See for yourself under m= apreduce.Job:

http://hadoop.apa= che.org/docs/r0.20.2/api/index.html =A0or

http://hadoop.apache.org/docs/current/api/index.html

=A0

So, back to my original q= uestion. If job isn=92t instance of Job object, then what object does it belong to in the following statement (DistributedCache.addCacheFile(URI, job.getConfiguration());

=A0

=A0

From: Mahesh Balija [mailto:balijamahesh.mca@gmail.com]
Sent: Tuesday, November 27, 2012 12:22 AM


To: user= @hadoop.apache.org
Subject: Re: MapReduce APIs

=A0

Hi AK,

=A0=A0=A0=A0=A0=A0=A0=A0=A0 I don't really understand what is stopping = you to use the job.getConfiguration() method to pass the configuration inst= ance to DistributedCache.addCacheFile(URI, job.getConfiguration()).
=A0=A0=A0=A0=A0=A0=A0=A0=A0 Only thing you need to do is pass the URI and c= onfiguration object (getting it from org.a= pache.hadoop.mapreduce.Job instance).

Best,
Mahesh.B.
Calsoft Labs.

On Mon, Nov 26, 2012 at 8:18 PM, Kartashov, Andy <= ;Andy.Kartashov= @mpac.ca> wrote:

Harsh,

Thanks for the " DistributedCache.addCacheFile(URI, job.getConfigurati= on());" suggestion.
What class is your instance job belongs to? It is not Job class, for sure. = So must be JobContext?

When I write my driver using new API I write:

Job job =3D new Job();
job.setJarByClass(....
job.setJobName(...
job.setSetMapOutputKey... | .. value
......Redeuce.....


So, how can I use your =A0piece of code here, i.e?
DistributedCache.addCacheFile(URI, job.getConfiguration());

How can I wire JobConf to Job instances?

Thanks,
AK


-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Saturday, November 24, 2012 2:22 AM
To: <user@ha= doop.apache.org>
Subject: Re: MapReduce APIs

You could use the org.apache.hadoop.filecache.DistributedCache API as:

DistributedCache.addCacheFile(URI, job.getConfiguration());

On Sat, Nov 24, 2012 at 3:06 AM, Kartashov, Andy <Andy.Kartashov@mpac.ca> wrote:=
> Guys,
>
>
>
> I know that there is old and new API for MapReduce. The old API is
> found under org.apache.hadoop.mapred and the new is under
> org.apache.hadoop.mapreduce
>
>
>
> I successfully used =A0both (the old and the new API) writing my
> MapReduce drivers.
>
>
>
> The problem came up when I tried to use distributed cache. My new API<= br> > Job object could not locate
>
> public void addCacheFile(URI uri) method and I was scratching my head = why.
>
>
>
> What I did not reaslise is that despite new and oold API there is also=
> Hadoop 0.20 vs Hadoop 2.0.0 APIs that use exact same packages.
>
> =A0The old Hadoop.0.20.00 new Mapreduce API class Job simply doesn'= ;t
> have that method "addCacheFile(URI uri)".
>
>
>
> I am running Hadoop 2.0.0. so could not understand why the method was<= br> > not inside the class. I ended up rewriting MR job under old API mapred=
> package and ran soccessfully.
>
>
>
> Can anyone shed some light on this?
>
>
>
> Thanks
>
> AK
>
>
>
>
>
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use,
> copying or disclosure is prohibited. If you are not the intended
> recipient, please delete and contact the sender immediately. Please > consider the environment before printing this e-mail. AVIS : le
> pr=E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont > confidentiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtr= e couverts
> par le secret professionnel. Toute utilisation, copie ou divulgation > non autoris=E9e est interdite. Si vous n'=EAtes pas le destinatair= e pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement l'e= xp=E9diteur.
> Veuillez penser =E0 l'environnement avant d'imprimer le pr=E9s= ent courriel



--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr= =E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont confiden= tiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre couverts pa= r le secret professionnel. Toute utilisation, copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le = destinataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatem= ent l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'= ;imprimer le pr=E9sent courriel

=A0

NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr= =E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont confiden= tiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre couverts pa= r le secret professionnel. Toute utilisation, copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le = destinataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatem= ent l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'= ;imprimer le pr=E9sent courriel

--20cf3074d6243057df04cf7b2dc7--