river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Firmstone <j...@zeus.net.au>
Subject Re: Service API, class visibility, isolation and garbage collection - ClassLoaders
Date Tue, 22 Jun 2010 08:39:47 GMT
Thanks Gregg, much appreciated.

Gregg Wonderly wrote:
> Getting back around to some of these, hopefully no confusion...
Yea, can be difficult to express things using only text communication, 
especially with something so complex.  Replies inline below.

>
> Peter Firmstone wrote:
>> Hi Gregg,
>>
>> Sorry I missed this earlier, have a look at my latest messages, they 
>> might provide some more background to my thoughts.
>>
>> Mike Warres has a good example of when the existing mechanism causes 
>> codebase annotation loss and type conflicts on page 23 of his paper 
>> "Class Loading Issues in Java RMI and Jini Network Technology".
>>
>> https://issues.apache.org/jira/secure/attachment/12413650/Java+Classloader+issues+relating+to+Jini+smli_tr-2006-149.pdf

>>
>> Because the Service API is loaded from the local file system and 
>> isn't dynamically downloaded, it may not have an annotation, I'm 
>> thinking about creating a new URL scheme, that provides information 
>> about all Jar files, whether local or downloaded dynamically, such as 
>> jar name and version annotation.  This should help people who wish to 
>> provision their codebases.
>
> For me, the important thing to understand about class 
> resolution/loading, is that there are two distinct issues.  First, do 
> you have the correct version of the class somewhere in the class 
> loading system in use, and Second, if the right version or no version 
> is available, do you have a way to go look for a definition that would 
> be the right version to use?
>
> ClassLoaders have historically been the domain of issue #1.  Issue #2 
> has been dealt with in a surprisingly large number of ways.  
> Serialization annotations that MarshalledObject et.al. provide one way 
> to deal with #2.  But we also have later development of things like 
> OSGi, Maven and other packaging systems which include ways to declare 
> or facilitate resolution of missing dependencies.
>
> RMIClassLoaderSPI is the pluggable way to deal with the String 
> annotations on classes.  It can facilitate a wide number of 
> possibilities.  The predominate issues is that it doesn't include an 
> "implementation" indication in concert with the String value, so you 
> don't know how to "delegate" to other mechanisms to make use of that 
> String value in a "varied" way.
>
> It seems to me that if we do something with a new RMIClassLoaderSPI 
> implementation.  My changes add no real value other than facilitating 
> the addition of another implementation that is offered remotely or 
> through some other path.  What we really need is the ability to look 
> at the annotation and use it intelligently to ask a factory mechanism 
> to use it to create a class loader.
>
>> Note I haven't had any thoughts about removing 
>> PreferredClassProvider, but I'm thinking about another ClassLoader 
>> structure, although I'm prone to changing my mind as I struggle to 
>> understand it all.  I haven't got any implementations, yet, still 
>> working it all out.
>
> PreferredClassLoading allows you to fix code that is generally used 
> across an application "suite", by overriding the use of something in 
> the classpath with something you provide in your -dl.jar (or other 
> codebase content) by making it preferred.
Ok, this is something I'm trying to fix using a different approach with 
Classloaders.  If the Application code is not visible to the proxy, the 
proxy can have it's own independent classes, with no class sharing 
between the proxy and application other than platform and API classes.   
I'd like a way to define classes for loading into an application or 
service classloader too, similar to a classpath.  Smart Proxy's can 
utilise their own libraries without conflicting with the application's 
library versions.

This is sort of what I'm thinking, evolved since last post, based on 
comments:

                                      CLASS LOADER VISIBILITY

 _______________________________________________________________________
|                                                                       |
|           Platform & API ClassLoader (incl Dumb Proxy Objects)        |
|----------------|------------------------------------------------------|
|       _________|________________________________                      |
|      |         |        |                       |                     |
|  ServiceUI     |   Smart Proxy        Extended Service API            |
|  ClassLoader   |  (incl ServiceUI)   & Dumb Proxy ClassLoader         |
| (UI that uses  |   ClassLoader's                |                     |
|  Dumb Proxy's) |             ___________________|___________          |
|                |_________   |             |                 |         |
|                |         |  |         ServiceUI        Smart Proxy    |
|                |    Parameter Imp     ClassLoader's    ClassLoader's  |
|                |    ClassLoader's   (for Dumb Proxy's)                |
|                |    (Server side)                                     |
|----------------|------------------------------------------------------|
|                |                                                      |
|          Common Classloader (As Per Dennis' comments)                 |
|       _________|________________________                              |
|      |                                  |                             |
| Service Imp                       Application                         |
| ClassLoader's                     ClassLoader                         |
|                                (incl Parameter Imp classes)           |
|_______________________________________________________________________|

Everything between the two --- lines is downloaded dynamically, cached 
or provisioned.
Dumb Proxy's have no download archives or classes other than those in 
the API,
so can be safely loaded into the Platform API ClassLoader or Extended 
Service API
ClassLoader.   These classloaders will be available via a weak reference 
index.

All classes in child ClassLoaders can see all parent ClassLoaders above 
in the tree, but classes in parent ClassLoaders cannot see the classes 
below them in child ClassLoaders.  For example, classes in the Platform 
& API CL, Common CL are visible to the Application CL classes.

This visibility, between ClassLoader's mapped out above, prevents 
implementations from interfering with each other, they can only 
cooperate between each other using common API classes.  No more worries 
about versioning conflicts etc.  Versions can be specified by 
implementations to allow codebase caching and provisioning to ensure 
that Serialization and implementation compatibility remains between the 
Server and it's Proxy.

This may cause some libraries to become duplicated in memory, however we 
must ensure a particular version is only downloaded once.

Something that has my attention about security is, currently we've based 
proxy security on the ClassLoader.  However when the AccessController 
checks permission, it allows only the permissions common to all 
ProtectionDomains on the Stack (see AccessControlContext).  Permissions 
really should be dynamically granted to ProtectionDomains, not the 
ClassLoader.

A ProtectionDomain, might represent a Principal, or it might represent a 
downloaded codebase, any number of which may exist in a single 
ClassLoader (so long as classes are not duplicated, this should be 
avoided since duplicate classes cannot be loaded into a ClassLoader).

Hence if a number of Services use the same codebases for their smart 
proxy's then that codebases ProtectionDomain will need the required 
Permissions and so will the ProtectionDomain for a Principal in order 
for those Permissions to become effective.  So we can preserve security 
and have multiple Services share the same proxy code.  But to do so we 
must forget about ClassLoader based Permission grants.

We can base Codebase trust on:

   1. Certificates[]  "Who wrote it?"
   2. CodeSource "Who wrote it and the name of the Codebase?"


We can base Principal trust on:

   1. Principals[] "Who are you?"


When a ProtectionDomain is created with the constructor:

public ProtectionDomain(CodeSource codesource,
                        PermissionCollection permissions,
                        ClassLoader classloader,
                        Principal[] principals)

The permissions granted to the ProtectionDomain will be dynamic and the 
current Policy will always be asked prior to the PermissionCollection 
passed into the constructor (Performance Hint: make sure permissions is 
null, use the Policy only)

The Principal[] can be java.security.acl.Group[] objects (Group extends 
Principal), then your free to add and remove Principals to that Group 
and hence the Group's ProtectionDomain, that associated with the 
CodeSource of the Proxy.  A group might represent each type of Service.

ProtectionDomains representing each will be on the stack and only the 
Permissions common to all ProtectionDomains on the stack will be granted.

Some observations:

   1. Codebases should advertise their required Permissions if trust is
      required.
   2. Codebases should be signed if trust is required.
   3. Utility Library Codebases can be released and signed for others to
      "trust" to be utilised in proxy's.  You might not trust unknown
      proxy code, but you might trust the utility library that provides
      trusted functionality for it.

So client Proxy trust is currently:

   1. Download the Proxy codebase.
   2. Unmarshall the Proxy.
   3. Ask the Proxy for the Proxy verifier.
   4. Ask the Server if it trusts the proxy.
   5. Grant trust to the proxy ClassLoader.

We can optionally do this too instead (feel free to correct any poorly 
conceived assumptions):

   1. Download the Proxy codebase (unless cached or downloaded prior)
   2. Record the Permissions required by the CodeSource advertised in
      the jar file.
   3. If Permissions are required, Create a Group and Create a
      ProtectionDomain for that CodeSource, passing the new Group
      (Principal) to the constructor (on first use of the codebase).  A
      Group might be specific a particular Service Interface and may
      already exist.
   4. Unmarshall the Proxy
   5. Check Constraints.
   6. Authenticate the Service if required.
   7. Authenticate the Client if required.
   8. Ask the Proxy for the Proxy Verifier.
   9. Does the Server trust the Proxy?
  10. Add the Service's Principal (or it's Group) to the Group, in the
      ProtectionDomain.
  11. Grant trust
         1. Using  grantCodeSource(CodeSource cs, Principal[] groups,
            Permission[] permissions)
         2. OR grantProtectionDomain(Class cl, Permission[] permissions)
         3. OR grant(Class cl, Principal[] principals, Permission[]
            permissions) - ClassLoader grant.
         4. OR grant(Certificate[] certs, Principal[] principals,
            Permission[] permissions)

Notes:

    * The permissions granted are limited to those declared in the jar file.
    * If the Trust grant is performed using 11.3. then permission grants
      are the same as the current ClassLoader based behaviour, all other
      libraries in with the proxy will be loaded into that ClassLoader
      and also receive those permissions.
    * Permissions granted using 11.4 are granted based on the codebase
      signers (developers) and Principals (groups) and may be granted at
      any time prior.
    * Permissions granted using 11.2 are granted directly to the proxy
      ProtectionDomain and only apply to the proxy's Code source, not
      bundled libraries or other jar archives distributed with the proxy.
    * Permissions granted using 11.1 are granted directly to the
      CodeSource and Principals (groups) and may be applied prior to
      downloading the code source.
    * Other proxy objects from different services, with identical proxy
      CodeSource may share the ProtectionDomain and class files, in this
      case the trust sequence is limited to steps 4 to 10 as the
      CodeSource itself is already trusted, the only thing that remains
      is to verify the proxy and perform authentication if required.

These Dynamic Grants have been implemented in 
DynamicConcurrentPolicyProvider.  Permissions granted to Groups differ 
from current Policy implementations that test Principals for equality 
only, this new Policy implementation also implies true when Principals 
are members of Groups that have been granted Permissions.

It might help to provide a Principal and Group framework implementation 
to make Authentication easier.

So this does actually point to a use for Patrick's suggested Codebase 
Entry?  Yes, for example if many services existed on the internet that 
utilise the same codebase versions, then a client could specify only a 
particular version, to ensure it only downloads one codebase, once for 
all matching instances of a Service.  It might first get the Codebase 
Entry's for a particular Service type, then lookup by each codebase 
version and deal with them in compatible batches.  Waiting for the 
Codebase URL annotation would be too late, filtering would be performed 
at the client rather than by the ServiceRegistrar.

We do need to implement something like RMIClassLoaderSPI, where 
ClassLoaders are created based on the names of jar archives *proxy.jar, 
*ext-api.jar, *ui.jar and *param.jar.  Where versioning is taken into 
consideration and a new URL format is created that allows for the needed 
flexibility.  Then of course class resolution needs to be figured out 
with the new URL annotation.

This may take some time to digest.  Please see the code in:

https://svn.apache.org/repos/asf/incubator/river/jtsk/trunk/src/org/apache/river/

Cheers,

Peter.
> I find this mechanism extremely useful in my service UI client that 
> has the same classes as all my UIs use for Swing and AWT stuff.  If I 
> need additional functionality or to fix something that only the UI of 
> a service needs, then I can make that class preferred and not have to 
> update all of my service UI client instances.  Mobile code solutions 
> are very convenient.
>
> Gregg Wonderly
>
>> Gregg Wonderly wrote:
>>> I have a Jini based application that is a content based router 
>>> system for satellite networks.  There are multiple servers running 
>>> with comms cards, and the ServiceUI needs to see all of them at 
>>> once, because I use transactions to commit data changes to all 
>>> servers.  In this case, one clients ServiceUI must lookup all of the 
>>> services, ignoring itself, and then get the service proxies to work 
>>> with.  All of the services proxies need to share interfaces and data 
>>> classes that are commonly exchanged.
>>>
>>> I didn't need a different kind of classloader tree, I just needed to 
>>> make sure that the parent class loader of the RMIClassLoaderSPI was 
>>> the ClassLoader of the client (The context ClassLoader which is a 
>>> PreferredClassLoader), which was already happening.  All the other 
>>> service instance unmarshalling would in fact make use of the 
>>> PreferredClassLoader, so that versioning could be done by preferring 
>>> classes, for example, and each services preferred classes would be 
>>> honored.
>>>
>>> There is only one -dl.jar involved in the commonality aspects of the 
>>> involved Classes.  I could break the common classes into a separate 
>>> jar, but I have not done that, yet.
>>>
>>> This type of application must be restarted fairly rarely, so class 
>>> compatibility is a very key issue.
>>>
>>> Peter, can you provide a more specific example of when you think the 
>>> structure you are proposing would be valuable, i.e. the existing 
>>> mechanism would break?
>>>
>>> Gregg Wonderly
>>>
>>
>>
>
>



Mime
View raw message