geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jencks <>
Subject Re: Whence the geronimo kernel?
Date Mon, 06 Apr 2009 18:00:26 GMT
Hi Gianny,

On Mar 17, 2009, at 4:24 AM, Gianny Damour wrote:

> On 13/03/2009, at 6:44 PM, David Jencks wrote:
>> On Mar 12, 2009, at 10:02 AM, David Jencks wrote:
>>> On Mar 12, 2009, at 3:25 AM, Gianny Damour wrote:
>>>> On 12/03/2009, at 4:29 AM, David Jencks wrote:
>>>>> I think I probably have the most experience with classloading  
>>>>> problems in geronimo and the only real problem that arises is  
>>>>> loading a jar in two different classloaders.   This can be  
>>>>> solved by a classloader-per-jar model which offers no  
>>>>> theoretical problems to set up in geronimo but practically would  
>>>>> take a lot of work (and maven projects to build a plugin per  
>>>>> jar).  So I think we'll have to see what kind of problems we get  
>>>>> with trying to actually use OSGI.
>>>> Hi,
>>>> Thinking more about this, I believe we can expedite the  
>>>> implementation of a classloader-per-jar model. Under the hood of  
>>>> a MultiParentClassLoader we can replace the current  
>>>> implementation of find class and resources contracts by an  
>>>> implementation which delegates to a bunch of URLClassLoaders (one  
>>>> per jar). These bunch of URLClassLoaders are global classloaders,  
>>>> i.e. shared across all the configs/MultiParentClassLoaders. The  
>>>> core challenge is to create them in a hierarchy respecting the  
>>>> maven dependency declarations. So, we could install the pom of  
>>>> the dependencies in the repo and lazily parse them when  
>>>> MultiParentClassLoader are created to build this global and share  
>>>> tree of URLClassLoaders.
>>> IMO the danger here is that the maven pom may not exist or may be  
>>> wrong.  OSGI has the same problem in that the vast majority of  
>>> released jars don't have osgi manifests.  I think I saw a rumor  
>>> that spring spent a lot of effort osgi-ifying a lot of commonly  
>>> used jars to try to solve this problem.
>>> I also don't know if there are situations in which a small number  
>>> of closely related jars need to be loaded in a single classloader,  
>>> perhaps because one of the jars is "optional" but if present the  
>>> "main" jar needs access to its classes.  I think there was an osgi  
>>> feature that looked sort of like this.
>>>> I just started to work on it and I will post back my findings (i  
>>>> should be able to complete this over the week-end). Even if we  
>>>> switch to an OSGi kernel, part of this work may hopefully still  
>>>> be useful.
>>> Unless you are pretty sure we won't have the kind of problems with  
>>> bad community metadata suggested above it might be a good idea to  
>>> do this in a sandbox branch?
>> Thinking about this more I really expect the code will be easy  
>> compared to straightening out the dependencies :-(
>> As a concrete example, the JACC spec needs the servlet and ejb  
>> specs but in our maven poms we've excluded them as dependencies so  
>> as to make it easier to upgrade the jars independently.  While  
>> (especially with maven 3) we can probably put in the dependencies  
>> with version ranges, they aren't there now.  Getting the server to  
>> work again may be time consuming.  I really hope I'm wrong :-D
> Hi David,
> I just created a patch
> which provides a preview of the code changes necessary to move to a  
> one classloader per jar model. It is not yet good enough to create a  
> branch as only 22 configurations out of the 27 of the jetty-jee5  
> assemblies can successfully start (failing config is certainly a  
> maven dependency problem).
> The design is rather simple:
> 1. when a configuraton classloader is created, configuration  
> dependencies and classPath entries are used to generate a bunch of  
> GlobalClassLoaders - one per dependency or one for all the classPath  
> entries.
> 2. this set of GlobalClassLoaders are put in a hierarchy according  
> to maven dependency declarations (note that dependency version  
> declarations are not honored even if present). Maven dependency  
> declarations can be augmented by dropping an XML file with maven  
> like <dependencies> declarations.
> 3. the classloaders of the parent configuration of the configuration  
> being created are added to this set of GlobalClassLoaders.
> 4. this set of GlobalClassLoaders is used under the hood of the  
> configuration's MultiParentClassLoader to implement find class and  
> resources contracts.
> Four maven dependency declarations had to be supplemented to start  
> 22 configurations. Assuming the same ratio, about 14 dependency  
> declarations may have to be supplemented to start the jetty-jee5  
> assembly. This is not too involving.
> Could you please have a cursory review of this preview patch and let  
> me know your thoughts?

I finally got some more time to look at this -- thanks for applying it  
to my sandbox framework branch.

There are two things that worry me about this.

1. IIUC whenever you include a jar in a configuration all the  
configuration's parents get added as parents to the jar's global  
classloader, in this code in MultiParentClassLoader2:

     protected void addParents(Set<GlobalClassLoader>  
globalClassLoaders) {
         for (GlobalClassLoader globalClassLoader :  
globalClassLoaders) {
             for (ClassLoader parent : parents) {

I don't think this is acceptable.  I think that once we have a  
GlobalClassloader set up for a jar it needs to be immutable.  With  
your code which jar a class is loaded from could depend on which  
configurations are started when you try to load the class.

I'm sure I haven't thought this through completely but I really wonder  
if adding parents to the global classloaders is necessary.  If we  
ignore geronimo plugins based on javaee apps for a moment, geronimo  
plugins won't have any classes in them, and jars will have maven  
dependencies on the appropriate other jars anyway.  So I think that in  
the normal case adding these parents won't add any missing classes.   
Constructing a classloader for a javaee app that can be used as a  
parent of some other classloader is a geronimo specific feature anyway  
not supported by maven so I think it would be OK to just have people  
include such dependencies in the <baseURL>-additional.xml (or the pom,  

If we drop this add-parents behavior we might need to add classloading  
rules to the global classloaders and either specify it directly or  
push it down from a plugin config.  Again I haven't thought this  

2. IIUC the <baseURL>-additional.xml only lets you add to the maven  
pom.  I think we need to be able to delete stuff too.

As a probably easy-to-fix bug I think that we currently decide if  
something is a jar or a plugin in MavenDependencyResolver by whether  
there is exactly one URL for the artifact.  However a geronimo plugin  
with classes inside -- say an ejb jar -- could have exactly one url  
but not be a jar.

These are just my first impressions, I'll keep looking.

david jencks

> Thanks,
> Gianny
>> thanks
>> david jencks
>>> thanks
>>> david jencks
>>>> Thanks,
>>>> Gianny
>>>>> One thing I'd really like actual user data on is how people  
>>>>> actually specify osgi classloading info in real life.  I'm very  
>>>>> aware that in theory you are supposed to specify the package  
>>>>> imports and exports for your bundle but I've been told that in  
>>>>> real life everyone with a serious osgi project actually  
>>>>> specifies the jar dependencies they want using require-bundle.
>>>>> thanks
>>>>> david jencks
>>>>>> Thanks,
>>>>>> Gianny
>>>>>> On 11/03/2009, at 7:11 PM, Guillaume Nodet wrote:
>>>>>>> On Wed, Mar 11, 2009 at 08:57, Gianny Damour <

>>>>>>> > wrote:
>>>>>>> Hi,
>>>>>>> FWIW, I believe that improving the configuration style to  
>>>>>>> simplify the means of creating a bunch of objects in the  
>>>>>>> kernel has more benefits than swapping the classloading infra.
>>>>>>> On paper OSGi may appear as superior from a classloading  
>>>>>>> isolation perspective; however, I believe the current CLing 

>>>>>>> design is nearly up to par with the OSGi one and that the main
>>>>>>> challenge is to properly tune export/import dependency  
>>>>>>> declarations.
>>>>>>> I have to disagree with that.  The CLing mechanism is very  
>>>>>>> different in Geronimo (from what I recall) and OSGi.  Geronimo
>>>>>>> uses a multi-parent classloader style with some nice features
>>>>>>> to be able to hide / never override + parent or self-first  
>>>>>>> delegation.
>>>>>>> OSGi CLind is very different: the first one is that you don't
>>>>>>> really have parent classloaders: the classloader for a given
>>>>>>> OSGi bundle is calculated wrt to the constraints expressed in
>>>>>>> the OSGi manifest using imported packages or required bundles.
>>>>>>> Let's take an example:
>>>>>>> bundle A needs api packages from bundles B and C
>>>>>>> implementation classes from bundle B and C needs something  
>>>>>>> from bundle D but with different versions
>>>>>>> OSGi will be able to handle that because of non tree-like  
>>>>>>> CLind mechanism: if bundle A is wired to bundle B, it does not
>>>>>>> have to see all the requirements from bundle B, and same for
>>>>>>> C.  Therefore, bundle A can be wired to both B and C without
>>>>>>> problems because it will not see bundle D at all (so there's
>>>>>>> no conflicts between the two versions of bundle D).
>>>>>>> OSGi has a much more powerful CLing mechanism where you can 

>>>>>>> express lots of different constraints.  The drawback is that
>>>>>>> establishing the classloader can take a bit of time, so going
>>>>>>> to OSGi most certainly leads to a big slowdown at startup  
>>>>>>> while creating the classloaders.
>>>>>>> Also, OSGi does not really play nicely with the usual JEE way
>>>>>>> to discover implementations through the MANIFEST/services  
>>>>>>> entries.  That's kinda what we've tried to solve in servicemix
>>>>>>> specs, though I'm not sure if that really applies everywhere
>>>>>>> because I would imagine the classloaders for EARs are not  
>>>>>>> really OSGi classloaders ...
>>>>>>> I certainly don't want to say OSGi is not the way to go, just
>>>>>>> want to make the point that there are benefits but also  
>>>>>>> drawbacks.
>>>>>>> The JAXB approach to turn xml plans to a bunch of objects is
>>>>>>> certainly interesting. I believe it is still a technology  
>>>>>>> limiting decision whereby a lot of custom code will have to be
>>>>>>> implemented to support various style (factory methods or beans
>>>>>>> et cetera) of configurations. I have been bouncing around this
>>>>>>> idea a while back and here it is again. Why do we want to  
>>>>>>> define a XML language to create a bunch of objects when  
>>>>>>> scripting can do that for us?
>>>>>>> I believe that xbean-spring is still unnecessary noisy when 

>>>>>>> compared to something like the Spring Bean Builder (

>>>>>>> ).
>>>>>>> If there is an interest in a scripting approach, then I can 

>>>>>>> investigate further.
>>>>>>> Thoughts?
>>>>>>> Thanks,
>>>>>>> Gianny
>>>>>>> On 11/03/2009, at 6:54 AM, David Jencks wrote:
>>>>>>> So as mentioned below I'm starting to look into the osgi  
>>>>>>> classloading bit, sort of "from the bottom".
>>>>>>> Another approach to many of these issues is perhaps "from the
>>>>>>> top", from the point of view of going from a presumably xml 

>>>>>>> plan to a bunch of objects.
>>>>>>> I've long thought that it must be possible to leverage jaxb to
>>>>>>> do most of the heavy lifting here.  In particular sxc is some
>>>>>>> code we can presumably actually extend to do stuff like  
>>>>>>> constructor dependency injection.  So another avenue that  
>>>>>>> could perhaps be approached in parallel would be to  
>>>>>>> investigate sxc, jaxb, xbean-spring, xbean-reflect, the  
>>>>>>> blueprint service schema, and jsr299 requirements and see what
>>>>>>> we can come up with.
>>>>>>> For instance, it might be possible to have a large part of the
>>>>>>> blueprint service functionality in jaxb-enabled objects that
>>>>>>> jaxb instantiates from the xml.  The "init" method could deal
>>>>>>> with feeding the metadata into the blueprint service core.  

>>>>>>> Maybe we can get sxc to use xbean-reflect to create the objects.
>>>>>>> So far this is more or less wild speculation in my head...  

>>>>>>> but I think it would be a lot of fun to investigate.
>>>>>>> thanks
>>>>>>> david jencks
>>>>>>> On Mar 4, 2009, at 4:56 PM, David Jencks wrote:
>>>>>>> Geronimo has been around for a while and despite the many good
>>>>>>> features gbeans and the geronimo kernel are not catching on 

>>>>>>> big time.  I think we want to consider taking action now to 

>>>>>>> avoid ending up being dragged down by supporting a dead  
>>>>>>> container.  Here are a few thoughts.
>>>>>>> Actual problems with geronimo:
>>>>>>> - gbeans are too restrictive.  It's too hard to instantiate 

>>>>>>> other peoples components as gbeans.  GBeans don't support  
>>>>>>> common patterns like factory methods, factory beans, etc etc,
>>>>>>> and require the component to be instantiated directly by the
>>>>>>> gbean framework.
>>>>>>> - it's too hard to get the classloaders to work.  The most  
>>>>>>> common problem is a class cast exception due to loading the 

>>>>>>> same jar in two plugins.  NoClassDefFound errors from an  
>>>>>>> optional jar in a child classloader are also really annoying.
>>>>>>> Really good things about geronimo I haven't seen elsewhere (at
>>>>>>> least in one place):
>>>>>>> - gbean dependencies work across plugins.  Dependencies are a
>>>>>>> unified system, not per-plugin.
>>>>>>> - gbean dependencies are resolved in the ancestors of a  
>>>>>>> plugin, not server wide.  This means that you can't make a  
>>>>>>> partially specified dependency ambiguous by deploying  
>>>>>>> additional plugins.  I consider this an extremely important 

>>>>>>> feature for predictability.
>>>>>>> - plugin dependencies allow assembly of a server from the  
>>>>>>> explicit dependencies which are normally the same as the maven
>>>>>>> dependencies.
>>>>>>> Other projects and specs that have stuff we should look into:
>>>>>>> maven.  Maven has a lot better infrastructure for dealing with
>>>>>>> dependency resolution from partial transitive dependency  
>>>>>>> specification than we do.  We should look into using more of
>>>>>>> their infrastructure.
>>>>>>> osgi. osgi has a lot of similarities to geronimo. The osgi  
>>>>>>> classloading model is getting a lot of people excited.  The 

>>>>>>> import-bundle idea is pretty much the same as our classloader
>>>>>>> model where every jar is a plugin.  I don't know if people are
>>>>>>> really using the allegedly recommended method of specifying 

>>>>>>> imports and exports and letting the osgi runtime figure out 

>>>>>>> where they come from; this seems worth investigating to me. 

>>>>>>> Also, we get periodic inquiries about when we are going to  
>>>>>>> support osgi and the was ce folks get even more.
>>>>>>> osgi blueprint service (rfc 124) This appears to be a simple
>>>>>>> wiring framework for a single plugin.  IIUC it uses the osgi
>>>>>>> service registry for component dependencies between bundles.
>>>>>>> xbean-spring.  I'd be reluctant to try to implement a  
>>>>>>> blueprint service that didn't provide the xbean-spring  
>>>>>>> capabilities really well
>>>>>>> ee6 dependency injection.  EE6 is going to have a pretty  
>>>>>>> sophisticated dependency injection service which we'll need to
>>>>>>> support anyway.  We should try to figure out how much of the
>>>>>>> core we can assemble using it.
>>>>>>> Other great stuff we have:
>>>>>>> xbean-reflect, xbean-finder, xbean-spring
>>>>>>> These ideas have been floating around in my head for a long 

>>>>>>> time and I've chatted with various people about them  
>>>>>>> occasionally.   While more discussion is certainly needed on
>>>>>>> everything here I need to do some implementation to understand
>>>>>>> much more.  So, what I'm planning to do:
>>>>>>> Dave's crazy work plan...
>>>>>>> - Try to use the osgi classloader.  I think this involves  
>>>>>>> putting the classloader creation in Configuration into a  
>>>>>>> service.  Configurations will turn into osgi bundles.  I'll 

>>>>>>> put the Kernel in the osgi ServiceRegistry so the  
>>>>>>> Configuration bundle activator should be able to use it to  
>>>>>>> resolve cross-plugin dependencies.
>>>>>>> - try to figure out how maven dependency resolution fits into
>>>>>>> osgi.
>>>>>>> - see if eclipse p2 is relevant for provisioning geronimo  
>>>>>>> repositories
>>>>>>> at this point I think geronimo would be running on osgi, still
>>>>>>> using gbeans.
>>>>>>> - look into relaxing the gbean framework so it is more plugin-

>>>>>>> at-a-time rather than gbean-at-a-time
>>>>>>> - see how that differs from the blueprint service, ee DI, and
>>>>>>> xbean-spring.  Try to support all of these at once.
>>>>>>> Thoughts? Counter proposals?  Anyone interested?
>>>>>>> many thanks
>>>>>>> david jencks
>>>>>>> -- 
>>>>>>> Cheers,
>>>>>>> Guillaume Nodet
>>>>>>> ------------------------
>>>>>>> Blog:
>>>>>>> ------------------------
>>>>>>> Open Source SOA

View raw message