hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason hadoop <jason.had...@gmail.com>
Subject Re: MapReduce jobs with expensive initialization
Date Sun, 08 Mar 2009 05:04:26 GMT
You can have your item in a separate jar and pin the reference so that it
becomes perm-gen, which will pin it. Then you can search the class loader
hierarchy for the reference.
A quick scan through the Child.java main loop shows no magic with class

I wrote some code to check this against 0.19.0. Very clearly the JVM is
doing nothing special with class loaders.
The classes are loaded exactly once.
The job counters contain details of how many times the map method was called
and the number of times the singleton was taken, including the jvm pid's

I wrote a mapper and a singleton, the singleton has 1 method that returns
the number of times that the getSingleton
The code fragment is from the examples in my book

On Fri, Mar 6, 2009 at 12:55 PM, Scott Carey <scott@richrelevance.com>wrote:

> One further thought on this, the mapper jvm may be loading the jar and
> overwriting / throwing away all previous class descriptions from the
> previous map job, which will remove the statics and reinitialize.  In this
> case, the singleton won't work if it is in the job jar.  What will work is
> putting the singleton in a global classpath (shared library not in the job
> jar).
> On 3/6/09 12:46 PM, "Scott Carey" <scott@richrelevance.com> wrote:
> The difference is that if the whole mapper class itself is being reloaded
> somehow (instantiated by reflection and then de-referenced and gc'd?) the
> static won't work the way you expect.  Not knowing how that works, and
> assuming that statics there don't work, a singleton in another class may
> still work.  The singleton class is certainly not being instantiated by
> reflection so (I believe) only a classloader closing will get rid of it.
> At least, its worth a try, since unlike the mapper class, you control how
> it is instantiated.  So the two cases are not the same.
> On 3/6/09 12:13 AM, "Rasit OZDAS" <rasitozdas@gmail.com> wrote:
> Owen, I tried this, it doesn't work.
> I doubt if static singleton method will work either,
> since it's much or less the same.
> Rasit
> 2009/3/2 Owen O'Malley <omalley@apache.org>
> >
> > On Mar 2, 2009, at 3:03 AM, Tom White wrote:
> >
> >  I believe the static singleton approach outlined by Scott will work
> >> since the map classes are in a single classloader (but I haven't
> >> actually tried this).
> >>
> >
> > Even easier, you should just be able to do it with static initialization
> in
> > the Mapper class. (I haven't tried it either... )
> >
> > -- Owen
> >
> --
> M. Raşit ÖZDAŞ

View raw message