Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AFC0DEC71 for ; Tue, 27 Nov 2012 15:07:51 +0000 (UTC) Received: (qmail 17076 invoked by uid 500); 27 Nov 2012 15:07:46 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 16980 invoked by uid 500); 27 Nov 2012 15:07:46 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 16967 invoked by uid 99); 27 Nov 2012 15:07:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Nov 2012 15:07:45 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [206.47.135.205] (HELO Spam1.prd.mpac.ca) (206.47.135.205) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Nov 2012 15:07:40 +0000 Received: from Spam1.prd.mpac.ca (unknown [127.0.0.1]) by IMSVA80 (Postfix) with ESMTP id 21AAD1D805A for ; Tue, 27 Nov 2012 10:07:17 -0500 (EST) Received: from SMAIL1.prd.mpac.ca (unknown [172.29.2.53]) by Spam1.prd.mpac.ca (Postfix) with ESMTP id C47C11D8054 for ; Tue, 27 Nov 2012 10:07:16 -0500 (EST) Received: from SMAIL1.prd.mpac.ca ([fe80::d548:4221:967c:4cfb]) by SMAIL1.prd.mpac.ca ([fe80::18cb:8648:b77f:2b55%11]) with mapi id 14.02.0318.004; Tue, 27 Nov 2012 10:07:16 -0500 From: "Kartashov, Andy" To: "user@hadoop.apache.org" Subject: RE: MapReduce APIs Thread-Topic: MapReduce APIs Thread-Index: Ac3JwpV0EUJkeNKVRzm3f1XG+FeuDgAe7IMAAGliR0AAKVHdgAAIPpCwAAubxwAACfLtcA== Date: Tue, 27 Nov 2012 15:07:15 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.29.60.102] Content-Type: multipart/alternative; boundary="_000_BD42F346AE90F544A731516A805D1B8AD87D1CSMAIL1prdmpacca_" MIME-Version: 1.0 X-TM-AS-Product-Ver: IMSVA-8.0.0.1304-6.5.0.1024-19396.007 X-TM-AS-Result: No--26.050-5.0-31-10 X-imss-scan-details: No--26.050-5.0-31-10 X-TM-AS-Result-Xfilter: Match text exemption rules:No X-TMASE-MatchedRID: 1ZHks2aQIki+SzPGLtd+jzTR2TFg0xG3BcCEAZkHsGe7+NPPxj+R6rcK r9noZuZJu958H+JYQswXIJWO/t2WjhDAulUi0M/IJrUxoq6hvw+O74ZfTyAQsO3Gz2tVgEFHGbe QYoZPf1Ws3OU3ZUeKuwymPwe8/cy9czj9grvSUEsR0Wxq9RAoByFNSRSGhTH1kzE2kM4b6HpDyL yEFs/t+vbdNxXHFX5WuhYk0rmkjp9B1m5nja+ukd35+5/2RxqmuLwbhNl9B5UaK6XhXABo99nfJ rUSEbFDw+7eVo5qUunqR0ztm0W62P3P0tuR+I4bzNY33yIEF4YL8TGleseLPFpbYq2f4jz+XYet l6Jggm+19T3v1Bw+JUQ0ZbDoCgw3U/SF2hrhoS0fBnGazfXNjqvd3WuZF0KAgwAxxxH8MzoRZcC YVXElTvhrPJGbPiPvZ7TMxxr9286A/dH4BaZIK2g4D2QV/2zL6r3HCixfuKdpr7yB77LWO4QK7n shlepip25VhBmPgZIBZOsOGHKpoUjlo2JU8qCxmHJeAYKlcpN3Bf9JIqsoeA8YwboCQc88oEMHf z15cBgCrD5a9fVAVE+OPKOjB+I8HFfqSAd6/1MiPTMUjkOgki1RIwMdx4yupOwTxn+Ouv4wzc7c h8CkTGwySqv4CcG5v/9UIaiB4/Pq6ddglKIimC+PrAd8gbHJosmMxONWlCcX/3xdyEASp90pRIF bsjfbb7mJ6HbhlhHHkD3MU9pI3QQES63KuaqOEzQnFLEeMUleqSmMX0XSKYPHFS+mNCEbEqWlBg Bv4m6HrHpzGQ5widemBQOZT5vtiIIt0sbLbU42RRIMOrvjaQ== X-Virus-Checked: Checked by ClamAV on apache.org --_000_BD42F346AE90F544A731516A805D1B8AD87D1CSMAIL1prdmpacca_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Super! I did miss the fact that Job class indeed inherits from JobContext. = This clarifies my issue. Thanks Dave and Harsh. From: Dave Beech [mailto:dave@paraliatech.com] Sent: Tuesday, November 27, 2012 9:51 AM To: user@hadoop.apache.org Subject: Re: MapReduce APIs AK - look again at that javadoc. Job does a have getConfiguration() method.= You may have missed it the first time because it's inherited from a parent= class, JobContext. On 27 November 2012 14:23, Kartashov, Andy > wrote: Thank man for the response. Much appreciated. Why? Because Job object doesn't have the below method getConfiguration(). S= ee for yourself under mapreduce.Job: http://hadoop.apache.org/docs/r0.20.2/api/index.html or http://hadoop.apache.org/docs/current/api/index.html So, back to my original question. If job isn't instance of Job object, then= what object does it belong to in the following statement (DistributedCache= .addCacheFile(URI, job.getConfiguration()); From: Mahesh Balija [mailto:balijamahesh.mca@gmail.com] Sent: Tuesday, November 27, 2012 12:22 AM To: user@hadoop.apache.org Subject: Re: MapReduce APIs Hi AK, I don't really understand what is stopping you to use the job.get= Configuration() method to pass the configuration instance to DistributedCac= he.addCacheFile(URI, job.getConfiguration()). Only thing you need to do is pass the URI and configuration objec= t (getting it from org.apache.hadoop.mapreduce.Job instance). Best, Mahesh.B. Calsoft Labs. On Mon, Nov 26, 2012 at 8:18 PM, Kartashov, Andy > wrote: Harsh, Thanks for the " DistributedCache.addCacheFile(URI, job.getConfiguration())= ;" suggestion. What class is your instance job belongs to? It is not Job class, for sure. = So must be JobContext? When I write my driver using new API I write: Job job =3D new Job(); job.setJarByClass(.... job.setJobName(... job.setSetMapOutputKey... | .. value ......Redeuce..... So, how can I use your piece of code here, i.e? DistributedCache.addCacheFile(URI, job.getConfiguration()); How can I wire JobConf to Job instances? Thanks, AK -----Original Message----- From: Harsh J [mailto:harsh@cloudera.com] Sent: Saturday, November 24, 2012 2:22 AM To: > Subject: Re: MapReduce APIs You could use the org.apache.hadoop.filecache.DistributedCache API as: DistributedCache.addCacheFile(URI, job.getConfiguration()); On Sat, Nov 24, 2012 at 3:06 AM, Kartashov, Andy > wrote: > Guys, > > > > I know that there is old and new API for MapReduce. The old API is > found under org.apache.hadoop.mapred and the new is under > org.apache.hadoop.mapreduce > > > > I successfully used both (the old and the new API) writing my > MapReduce drivers. > > > > The problem came up when I tried to use distributed cache. My new API > Job object could not locate > > public void addCacheFile(URI uri) method and I was scratching my head why= . > > > > What I did not reaslise is that despite new and oold API there is also > Hadoop 0.20 vs Hadoop 2.0.0 APIs that use exact same packages. > > The old Hadoop.0.20.00 new Mapreduce API class Job simply doesn't > have that method "addCacheFile(URI uri)". > > > > I am running Hadoop 2.0.0. so could not understand why the method was > not inside the class. I ended up rewriting MR job under old API mapred > package and ran soccessfully. > > > > Can anyone shed some light on this? > > > > Thanks > > AK > > > > > > NOTICE: This e-mail message and any attachments are confidential, > subject to copyright and may be privileged. Any unauthorized use, > copying or disclosure is prohibited. If you are not the intended > recipient, please delete and contact the sender immediately. Please > consider the environment before printing this e-mail. AVIS : le > pr=E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont > confidentiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre couve= rts > par le secret professionnel. Toute utilisation, copie ou divulgation > non autoris=E9e est interdite. Si vous n'=EAtes pas le destinataire pr=E9= vu de ce courriel, supprimez-le et contactez imm=E9diatement l'exp=E9diteur= . > Veuillez penser =E0 l'environnement avant d'imprimer le pr=E9sent courrie= l -- Harsh J NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before pri= nting this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8ce jointe qui= l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'auteur et peu= vent =EAtre couverts par le secret professionnel. Toute utilisation, copie = ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le desti= nataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement l= 'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le pr= =E9sent courriel NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before pri= nting this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8ce jointe qui= l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'auteur et peu= vent =EAtre couverts par le secret professionnel. Toute utilisation, copie = ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le desti= nataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement l= 'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le pr= =E9sent courriel NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before pri= nting this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8ce jointe qui= l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'auteur et peu= vent =EAtre couverts par le secret professionnel. Toute utilisation, copie = ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le desti= nataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement l= 'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le pr= =E9sent courriel --_000_BD42F346AE90F544A731516A805D1B8AD87D1CSMAIL1prdmpacca_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Super! I did miss the f= act that Job class indeed inherits from JobContext. This clarifies my issue= . Thanks Dave and Harsh.

 

From: Dave Beech [mailto:dave@paraliatech.com]
Sent: Tuesday, November 27, 2012 9:51 AM
To: user@hadoop.apache.org
Subject: Re: MapReduce APIs

 

AK - look again at that javadoc. Job does a have get= Configuration() method. You may have missed it the first time because it's = inherited from a parent class, JobContext. 

 

On 27 November 2012 14:23, Kartashov, Andy <Andy.Kartashov@mpac.= ca> wrote:

Thank man fo= r the response. Much appreciated.

 

Why? Because= Job object doesn’t have the below method getConfiguration(). See for= yourself under mapreduce.Job:

http= ://hadoop.apache.org/docs/r0.20.2/api/index.html  or

http://hadoop.apache.org/docs/current/api/i= ndex.html

 

So, back to my original question. If job isn’t instance of Job object, then what object does it belong to in the following stateme= nt (DistributedCache.addCacheFile(URI, job= .getConfiguration());

 

 

From: Mahesh Balija [mailto:balijamahesh.mca@gmail.com]
Sent: Tuesday, November 27, 2012 12:22 AM


To: user= @hadoop.apache.org
Subject: Re: MapReduce APIs

 

Hi AK,

          I don't really under= stand what is stopping you to use the job.getConfiguration() method to pass= the configuration instance to DistributedCache.addCacheFile(URI, job.getCo= nfiguration()).
          Only thing you need = to do is pass the URI and configuration object (getting it from org.apache.hadoop.mapreduce.Job instance).
Best,
Mahesh.B.
Calsoft Labs.

On Mon, Nov 26, 2012 at 8:18 PM, Kartasho= v, Andy <And= y.Kartashov@mpac.ca> wrote:

Harsh,

Thanks for the " DistributedCache.addCacheFile(URI, job.getConfigurati= on());" suggestion.
What class is your instance job belongs to? It is not Job class, for sure. = So must be JobContext?

When I write my driver using new API I write:

Job job =3D new Job();
job.setJarByClass(....
job.setJobName(...
job.setSetMapOutputKey... | .. value
......Redeuce.....


So, how can I use your  piece of code here, i.e?
DistributedCache.addCacheFile(URI, job.getConfiguration());

How can I wire JobConf to Job instances?

Thanks,
AK


-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Saturday, November 24, 2012 2:22 AM
To: <user@ha= doop.apache.org>
Subject: Re: MapReduce APIs

You could use the org.apache.hadoop.filecache.DistributedCache API as:

DistributedCache.addCacheFile(URI, job.getConfiguration());

On Sat, Nov 24, 2012 at 3:06 AM, Kartashov, Andy <Andy.Kartashov@mpac.ca> wrote:=
> Guys,
>
>
>
> I know that there is old and new API for MapReduce. The old API is
> found under org.apache.hadoop.mapred and the new is under
> org.apache.hadoop.mapreduce
>
>
>
> I successfully used  both (the old and the new API) writing my > MapReduce drivers.
>
>
>
> The problem came up when I tried to use distributed cache. My new API<= br> > Job object could not locate
>
> public void addCacheFile(URI uri) method and I was scratching my head = why.
>
>
>
> What I did not reaslise is that despite new and oold API there is also=
> Hadoop 0.20 vs Hadoop 2.0.0 APIs that use exact same packages.
>
>  The old Hadoop.0.20.00 new Mapreduce API class Job simply doesn'= t
> have that method "addCacheFile(URI uri)".
>
>
>
> I am running Hadoop 2.0.0. so could not understand why the method was<= br> > not inside the class. I ended up rewriting MR job under old API mapred=
> package and ran soccessfully.
>
>
>
> Can anyone shed some light on this?
>
>
>
> Thanks
>
> AK
>
>
>
>
>
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use,
> copying or disclosure is prohibited. If you are not the intended
> recipient, please delete and contact the sender immediately. Please > consider the environment before printing this e-mail. AVIS : le
> pr=E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont
> confidentiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre co= uverts
> par le secret professionnel. Toute utilisation, copie ou divulgation > non autoris=E9e est interdite. Si vous n'=EAtes pas le destinataire pr= =E9vu de ce courriel, supprimez-le et contactez imm=E9diatement l'exp=E9dit= eur.
> Veuillez penser =E0 l'environnement avant d'imprimer le pr=E9sent cour= riel



--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr= =E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont confidentiel= s, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre couverts par le sec= ret professionnel. Toute utilisation, copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le dest= inataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement = l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le pr= =E9sent courriel

 

NOTICE: This e-mail message and any attachments are = confidential, subject to copyright and may be privileged. Any unauthorized = use, copying or disclosure is prohibited. If you are not the intended recip= ient, please delete and contact the sender immediately. Please consider the environment before printing this e= -mail. AVIS : le pr=E9sent courriel et toute pi=E8ce jointe qui l'accompagn= e sont confidentiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre = couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris=E9e est interdite. Si vous n= '=EAtes pas le destinataire pr=E9vu de ce courriel, supprimez-le et contact= ez imm=E9diatement l'exp=E9diteur. Veuillez penser =E0 l'environnement avan= t d'imprimer le pr=E9sent courriel

 

NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr= =E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont confidentiel= s, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre couverts par le sec= ret professionnel. Toute utilisation, copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le dest= inataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatement = l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le pr= =E9sent courriel --_000_BD42F346AE90F544A731516A805D1B8AD87D1CSMAIL1prdmpacca_--