hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Beech <d...@paraliatech.com>
Subject Re: MapReduce APIs
Date Tue, 27 Nov 2012 14:50:52 GMT
AK - look again at that javadoc. Job does a have getConfiguration() method.
You may have missed it the first time because it's inherited from a parent
class, JobContext.


On 27 November 2012 14:23, Kartashov, Andy <Andy.Kartashov@mpac.ca> wrote:

>  Thank man for the response. Much appreciated.
>
>
>
> Why? Because Job object doesn’t have the below method getConfiguration().
> See for yourself under mapreduce.Job:
>
> http://hadoop.apache.org/docs/r0.20.2/api/index.html  or
>
> http://hadoop.apache.org/docs/current/api/index.html
>
>
>
> So, back to my original question. If *job* isn’t instance of *Job*object, then what
object does it belong to in the following statement
> (DistributedCache.addCacheFile(URI, *job*.getConfiguration());
>
> **
>
> * *
>
>
>
> *From:* Mahesh Balija [mailto:balijamahesh.mca@gmail.com]
> *Sent:* Tuesday, November 27, 2012 12:22 AM
>
> *To:* user@hadoop.apache.org
> *Subject:* Re: MapReduce APIs
>
>
>
> Hi AK,
>
>           I don't really understand what is stopping you to use the
> job.getConfiguration() method to pass the configuration instance to
> DistributedCache.addCacheFile(URI, job.getConfiguration()).
>           Only thing you need to do is pass the URI and configuration
> object (getting it from org.apache.hadoop.mapreduce.Job instance).
>
> Best,
> Mahesh.B.
> Calsoft Labs.
>
> On Mon, Nov 26, 2012 at 8:18 PM, Kartashov, Andy <Andy.Kartashov@mpac.ca>
> wrote:
>
> Harsh,
>
> Thanks for the " DistributedCache.addCacheFile(URI,
> job.getConfiguration());" suggestion.
> What class is your instance job belongs to? It is not Job class, for sure.
> So must be JobContext?
>
> When I write my driver using new API I write:
>
> *Job* *job* = new * Job*();
> job.setJarByClass(....
> job.setJobName(...
> job.setSetMapOutputKey... | .. value
> ......Redeuce.....
>
>
> So, how can I use your  piece of code here, i.e?
> DistributedCache.addCacheFile(URI, job.getConfiguration());
>
> How can I wire JobConf to Job instances?
>
> Thanks,
> AK
>
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Saturday, November 24, 2012 2:22 AM
> To: <user@hadoop.apache.org>
> Subject: Re: MapReduce APIs
>
> You could use the org.apache.hadoop.filecache.DistributedCache API as:
>
> DistributedCache.addCacheFile(URI, job.getConfiguration());
>
> On Sat, Nov 24, 2012 at 3:06 AM, Kartashov, Andy <Andy.Kartashov@mpac.ca>
> wrote:
> > Guys,
> >
> >
> >
> > I know that there is old and new API for MapReduce. The old API is
> > found under org.apache.hadoop.mapred and the new is under
> > org.apache.hadoop.mapreduce
> >
> >
> >
> > I successfully used  both (the old and the new API) writing my
> > MapReduce drivers.
> >
> >
> >
> > The problem came up when I tried to use distributed cache. My new API
> > Job object could not locate
> >
> > public void addCacheFile(URI uri) method and I was scratching my head
> why.
> >
> >
> >
> > What I did not reaslise is that despite new and oold API there is also
> > Hadoop 0.20 vs Hadoop 2.0.0 APIs that use exact same packages.
> >
> >  The old Hadoop.0.20.00 new Mapreduce API class Job simply doesn't
> > have that method "addCacheFile(URI uri)".
> >
> >
> >
> > I am running Hadoop 2.0.0. so could not understand why the method was
> > not inside the class. I ended up rewriting MR job under old API mapred
> > package and ran soccessfully.
> >
> >
> >
> > Can anyone shed some light on this?
> >
> >
> >
> > Thanks
> >
> > AK
> >
> >
> >
> >
> >
> > NOTICE: This e-mail message and any attachments are confidential,
> > subject to copyright and may be privileged. Any unauthorized use,
> > copying or disclosure is prohibited. If you are not the intended
> > recipient, please delete and contact the sender immediately. Please
> > consider the environment before printing this e-mail. AVIS : le
> > présent courriel et toute pièce jointe qui l'accompagne sont
> > confidentiels, protégés par le droit d'auteur et peuvent être couverts
> > par le secret professionnel. Toute utilisation, copie ou divulgation
> > non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de
> ce courriel, supprimez-le et contactez immédiatement l'expéditeur.
> > Veuillez penser à l'environnement avant d'imprimer le présent courriel
>
>
>
> --
> Harsh J
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
>
>  NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>

Mime
View raw message