hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartashov, Andy" <Andy.Kartas...@mpac.ca>
Subject RE: MapReduce APIs
Date Tue, 27 Nov 2012 15:07:15 GMT
Super! I did miss the fact that Job class indeed inherits from JobContext. This clarifies my
issue. Thanks Dave and Harsh.

From: Dave Beech [mailto:dave@paraliatech.com]
Sent: Tuesday, November 27, 2012 9:51 AM
To: user@hadoop.apache.org
Subject: Re: MapReduce APIs

AK - look again at that javadoc. Job does a have getConfiguration() method. You may have missed
it the first time because it's inherited from a parent class, JobContext.

On 27 November 2012 14:23, Kartashov, Andy <Andy.Kartashov@mpac.ca<mailto:Andy.Kartashov@mpac.ca>>
wrote:
Thank man for the response. Much appreciated.

Why? Because Job object doesn't have the below method getConfiguration(). See for yourself
under mapreduce.Job:
http://hadoop.apache.org/docs/r0.20.2/api/index.html  or
http://hadoop.apache.org/docs/current/api/index.html

So, back to my original question. If job isn't instance of Job object, then what object does
it belong to in the following statement (DistributedCache.addCacheFile(URI, job.getConfiguration());


From: Mahesh Balija [mailto:balijamahesh.mca@gmail.com<mailto:balijamahesh.mca@gmail.com>]
Sent: Tuesday, November 27, 2012 12:22 AM

To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: MapReduce APIs

Hi AK,

          I don't really understand what is stopping you to use the job.getConfiguration()
method to pass the configuration instance to DistributedCache.addCacheFile(URI, job.getConfiguration()).
          Only thing you need to do is pass the URI and configuration object (getting it from
org.apache.hadoop.mapreduce.Job instance).

Best,
Mahesh.B.
Calsoft Labs.
On Mon, Nov 26, 2012 at 8:18 PM, Kartashov, Andy <Andy.Kartashov@mpac.ca<mailto:Andy.Kartashov@mpac.ca>>
wrote:
Harsh,

Thanks for the " DistributedCache.addCacheFile(URI, job.getConfiguration());" suggestion.
What class is your instance job belongs to? It is not Job class, for sure. So must be JobContext?

When I write my driver using new API I write:

Job job = new Job();
job.setJarByClass(....
job.setJobName(...
job.setSetMapOutputKey... | .. value
......Redeuce.....


So, how can I use your  piece of code here, i.e?
DistributedCache.addCacheFile(URI, job.getConfiguration());

How can I wire JobConf to Job instances?

Thanks,
AK

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com<mailto:harsh@cloudera.com>]
Sent: Saturday, November 24, 2012 2:22 AM
To: <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: MapReduce APIs

You could use the org.apache.hadoop.filecache.DistributedCache API as:

DistributedCache.addCacheFile(URI, job.getConfiguration());

On Sat, Nov 24, 2012 at 3:06 AM, Kartashov, Andy <Andy.Kartashov@mpac.ca<mailto:Andy.Kartashov@mpac.ca>>
wrote:
> Guys,
>
>
>
> I know that there is old and new API for MapReduce. The old API is
> found under org.apache.hadoop.mapred and the new is under
> org.apache.hadoop.mapreduce
>
>
>
> I successfully used  both (the old and the new API) writing my
> MapReduce drivers.
>
>
>
> The problem came up when I tried to use distributed cache. My new API
> Job object could not locate
>
> public void addCacheFile(URI uri) method and I was scratching my head why.
>
>
>
> What I did not reaslise is that despite new and oold API there is also
> Hadoop 0.20 vs Hadoop 2.0.0 APIs that use exact same packages.
>
>  The old Hadoop.0.20.00 new Mapreduce API class Job simply doesn't
> have that method "addCacheFile(URI uri)".
>
>
>
> I am running Hadoop 2.0.0. so could not understand why the method was
> not inside the class. I ended up rewriting MR job under old API mapred
> package and ran soccessfully.
>
>
>
> Can anyone shed some light on this?
>
>
>
> Thanks
>
> AK
>
>
>
>
>
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use,
> copying or disclosure is prohibited. If you are not the intended
> recipient, please delete and contact the sender immediately. Please
> consider the environment before printing this e-mail. AVIS : le
> présent courriel et toute pièce jointe qui l'accompagne sont
> confidentiels, protégés par le droit d'auteur et peuvent être couverts
> par le secret professionnel. Toute utilisation, copie ou divulgation
> non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel,
supprimez-le et contactez immédiatement l'expéditeur.
> Veuillez penser à l'environnement avant d'imprimer le présent courriel



--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe
qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite.
Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe
qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite.
Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe
qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite.
Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

Mime
View raw message