hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject RE: Custom Class Loader for Hadoop M/R jobs?
Date Fri, 16 Apr 2010 11:31:15 GMT

David,

The only reason to roll your own is if you do this in a specific map reduce job and the functionality
doesn't already exist.
And no, I'm not suggesting that.

As to OSGi for a solution, sure, that would work. I think Chris was looking at something like
Tomcat because they have also had to deal with this issue, so a solution exists. I wasn't
aware of the OSGi implementations so that too is an option.

The reason I started to look at this in terms of the Configuration class is that it looks
like some of the pieces are already in place and I was wondering if the class already has
enough of a solution in place to allow for custom class loads?

Thanks to everyone!

-Mike

> Date: Fri, 16 Apr 2010 10:05:05 +0100
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> From: david.savage@paremus.com
> To: general@hadoop.apache.org
> 
> Hi there,
> 
> Many thanks to Caspar for the kind words. Yep we have had Hadoop
> running in an OSGi environment in the past - though it was for a proof
> of concept so I've not tracked the current Hadoop trunk to see if that
> patch still applies. I'm more than happy to try to offer advice if
> this is something you wish to pursue, though I'd also highly recommend
> the Felix mailing lists (user@felix.apache.org or
> dev@felix.apache.org) as excellent sources of OSGi wisdom.
> 
> Regarding licensing again IANAL but here's the list of projects at
> apache are using OSGi (that I am aware of) so I'm pretty certain it's
> compatible...
> 
> http://felix.apache.org
> http://aries.apache.org
> http://sling.apache.org
> http://tuscany.apache.org
> http://servicemix.apache.org
> http://activemq.apache.org
> http://cxf.apache.org
> 
> Also I know there's been work to make other library projects such as
> Apache Commons OSGi compatible.
> 
> Regarding custom classloaders I'd tend to advise checking if OSGi
> solves your problems first before trying to role your own - the
> benefits of using a common classloading model is that your components
> can then interact with other frameworks - also if you run into complex
> problems their are a wealth of experts out there who can help you -
> where as with a custom approach you are on your own.
> 
> Regards,
> 
> Dave
> 
> > On Thu, 15 Apr 2010 10:47, Caspar MacRae wrote:
> >
> > ServiceMix and ActiveMQ are also using OSGi in their core AFAIK.
> >
> > The way this (these) problems have been solved is more of an afterthought,
> > and regardless is quite simplistic (thinking of classloader isolation in web
> > containers, app servers, and the monolithic WARs produced as a result),
> > however OSGi's approach to modularity provides a truly elegant solution.
> >
> > wrt to the ticket https://issues.apache.org/jira/browse/MAPREDUCE-243, Dave
> > Savage is a nice chap who works for Paremus (an enterprise OSGi provider) I
> > spoke with him at the recent OSGi devcon in London and he did quite a bit of
> > work to get this working for private clients - I'm sure with his expertise
> > it would not be a huge effort to get this rolling again.
> >
> > > On 15 April 2010 18:23, Segel, Mike <mse...@navteq.com> wrote:
> > >
> > > Ok, so are you using OSGi's code, or just writing your own code to
> > > implement a published API/specification?
> > >
> > > I realize this is getting off topic, so you may just want to e-mail me
> > > directly.
> > >
> > > Thx
> > >
> > > -Mike
> > >
> > > -----Original Message-----
> > > From: Marcel May [mailto:marcel....@consol.de]
> > > Sent: Thursday, April 15, 2010 11:35 AM
> > > To: general@hadoop.apache.org
> > > Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> > >
> > >
> > > On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:
> > >
> > > > Caspar,
> > > >
> > > > While IANAL, I did a first blush read of OSGi's Specification License,
> > > Version 1.0. on the website link:
> > > http://www.osgi.org/Specifications/Licensing as well as the license on the
> > > download page: http://www.osgi.org/Download/Release4V42?info=nothanks.
> > > >
> > > > In a nutshell, OSGi's Specification License, Version 1.0. is incompatible
> > > with Apache's.
> > > >
> > > > That alone can explain why Apache projects won't touch OSGi.
> > > >
> > >
> > > http://felix.apache.org/
> > >
> > > > From a more practical issue, this problem is solved in other Apache
> > > projects, so why not borrow from another Apache project like Tomcat?
> > > >
> > > > Does that make sense? It appears OSGi is a non-starter.
> > > >
> > > > HTH
> > > >
> > > > -Mike
> > > ...
> > >
> > > Cheers,
> > > Marcel
> > >
> > >
> > > The information contained in this communication may be CONFIDENTIAL and is
> > > intended only for the use of the recipient(s) named above.  If you are not
> > > the intended recipient, you are hereby notified that any dissemination,
> > > distribution, or copying of this communication, or any of its contents, is
> > > strictly prohibited.  If you have received this communication in error,
> > > please notify the sender and delete/destroy the original message and any
> > > copy of it from your computer or paper files.
> > >
 		 	   		  
_________________________________________________________________
The New Busy is not the old busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_3
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message