hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kunaal <kunaalbha...@gmail.com>
Subject Re: Hadoop and Hibernate
Date Fri, 02 Mar 2012 17:46:02 GMT
Are you looking to use DistributedCache for better performance?

On Fri, Mar 2, 2012 at 9:42 AM, Geoffry Roberts
<geoffry.roberts@gmail.com>wrote:

> This is a tardy response.  I'm spread pretty thinly right now.
>
> DistributedCache<
> http://hadoop.apache.org/common/docs/r1.0.0/mapred_tutorial.html#DistributedCache
> >is
> apparently deprecated.  Is there a replacement?  I didn't see anything
> about this in the documentation, but then I am still using 0.21.0. I have
> to for performance reasons.  1.0.1 is too slow and the client won't have
> it.
>
> Also, the DistributedCache<
> http://hadoop.apache.org/common/docs/r1.0.0/mapred_tutorial.html#DistributedCache
> >approach
> seems only to work from within a hadoop job.  i.e. From within a
> Mapper or a Reducer, but not from within a Driver.  I have libraries that I
> must access both from both places.  I take it that I am stuck keeping two
> copies of these libraries in synch--Correct?  It's either that, or copy
> them into hdfs, replacing them all at the beginning of each job run.
>
> Looking for best practices.
>
> Thanks
>
> On 28 February 2012 10:17, Owen O'Malley <omalley@apache.org> wrote:
>
> > On Tue, Feb 28, 2012 at 5:15 PM, Geoffry Roberts
> > <geoffry.roberts@gmail.com> wrote:
> >
> > > If I create an executable jar file that contains all dependencies
> > required
> > > by the MR job do all said dependencies get distributed to all nodes?
> >
> > You can make a single jar and that will be distributed to all of the
> > machines that run the task, but it is better in most cases to use the
> > distributed cache.
> >
> > See
> >
> http://hadoop.apache.org/common/docs/r1.0.0/mapred_tutorial.html#DistributedCache
> >
> > > If I specify but one reducer, which node in the cluster will the
> reducer
> > > run on?
> >
> > The scheduling is done by the JobTracker and it isn't possible to
> > control the location of the reducers.
> >
> > -- Owen
> >
>
>
>
> --
> Geoffry Roberts
>



-- 
"What we are is the universe's gift to us.
What we become is our gift to the universe."

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message