river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter <j...@zeus.net.au>
Subject OSGi NP Complete Was: OSGi - deserialization remote invocation strategy
Date Sat, 11 Feb 2017 11:21:28 GMT
In a word, ServiceDiscoveryManager

ServiceDiscoveryManager is the solution.

ServiceDiscoveryManager performs discovery and looks up services from 
registrars based on filters.  ServiceDiscoveryManager then performs 
local filtering.  This allows time for proxy bundles to be installed, 
resolve, started and confirmed type compatible, prior to them being made 
available (via OSGi service registry if you so desire) for client use.

The new interfaces that are part of JGDMS that I'd like to see their way 
into River, found here:

https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/SafeServiceRegistrar.java

https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceAttributesAccessor.java
https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceCodebaseAccessor.java
https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceIDAccessor.java
https://github.com/pfirmstone/JGDMS/blob/Maven_build/modularize/JGDMS/jgdms-lib-dl/src/main/java/net/jini/lookup/ServiceProxyAccessor.java

ServiceCodebaseAccessor is also used as part of secure discovery, but 
the codebase string and certs are transferred as primitives over the 
network.

In this case codebase annotations don't need to be included in the 
stream, the JERI endpoints don't need them at all.

How so?

We can use ServiceItemFilter and ProxyPreparer to install, resolve and 
start out proxy codebase, before downloading the proxy.  The interfaces 
listed above allow an array bootstrap proxy's (java.lang.reflect.Proxy) 
to be obtained from SafeServiceRegistrar.

Firstly the bootstrap proxy's JERI endpoint will be loaded in the 
ServiceDiscoveryManager's ClassLoader, so after we've retrieved the 
codebase annotation and signers, created a bundle for the proxy, 
resolved it's dependencies (via OSGi resolution and repository 
services), we need to remarshall the bootstrap proxy into a 
MarshalledInstance, then unmarshall it using the ClassLoader of the 
recently started proxy bundle.  Then when we cast the bootstrap proxy to 
ServiceProxyAccessor and retrieve the smart proxy, it will be loaded 
into the same ClassLoader that the bootstrap proxy uses, our newly 
provisioned and loaded bundle (the correct ClassLoader) without need to 
serialize any annotations, then the smart proxy can have constraints 
applied etc and be registered as an OSGi service with the OSGi 
registrar, where client code can interact with the remote proxy.

Now if public Serializable classes that are imported by the proxy's 
bundle (service api) or private classes in the proxy's bundle can be 
deserialized and the JERI endpoint has a reference to the ClassLoader of 
the proxy.

This should be good enough so we don't require the "bundle stack" 
proposed earlier, which also saves the need to explain it and simplifies 
the solution (the intent of the bundle stack was to allow 
deserialization of private classes within other bundles whose packages 
have been imported by the proxy).

The client won't be able to pass a smart proxy to the service (like a 
Listener), but it can still pass a non smart proxy and it will still 
function.  So clients can still export their own remote service (albiet 
without a codebase, excluding smart proxy's), but it'll be good enough 
for a listener etc.

Cheers,

Peter.


On 8/02/2017 1:09 PM, Niclas Hedhman wrote:
> Maybe there are some misunderstanding somewhere... see below;
>
> On Wed, Feb 8, 2017 at 3:35 AM, Peter<jini@zeus.net.au>  wrote:
>> I'm currently only considering OSGi server ->  OSGi client.  Mick's
> investigating all four options.
>
> Ok, makes it a lot easier for me to follow.
>
>> Not expecting the client calling bundle to resolve everything, hence the
>> stack, so we have the full visibility of the bundle of the class that was
>> last resolved, so we can resolve its fields from it's bundle.  Eg it
>> might import packages the client does not.
> What is "client calling bundle"? I assume that it is the bundle that
> contains the service lookup...
>
> First of all, is the OSGi integration intending to be at "OSGi service
> level" ? If so, the Jini support bundle(s) in OSGi, will monitor Jini
> service registrations and register those in the local OSGi registry and all
> clients would end up being rather ignorant about the remote lookup. It is
> also possible to "be involved" in the service lookup (and registration)
> through the Service Hook API (I might got the name of it wrong), so that
> only when services are requested or service listeners are registered, does
> the Jini support kick in. And this can therefore work both ways...
>
> So, that said; I agree that the "client calling bundle" does not resolve
> anything. And I would go further and say; IF the client calling bundle
> looks up Jini services directly, all bets are off. This introduces some
> workable constraints of what could happen.
>
>
>> The "exact version" thing (only applies to the proxy bundle as we
>> expect the framework to load its deps) can be relaxed to compatible
>> versions to increase class sharing if you think it helps.  The proxy
> bundle
>> doesn't export anything at the client, only the server, it just seems to
>> make sense to keep the latest proxy communicating in case that last
>> bug fix release addresses a security issue.  All proxy classes are
>> implementation only classes.
> Yes, correct. The 'temporary' proxy bundle should not export any packages.
>
>> Because the proxy bundle manifest declares version import ranges,
>> I'm expecting the framework to favour already loaded bundles to
>> satisfy package import deps.
> Uhhhh... The OSGi framework doesn't "auto load" bundles. It is an explicit
> step. There are many "bundle loaders" around, such as Apache Karaf
> "Features" and "deploy/" directory. Most frameworks can also be instructed
> to load bundles at boot. So there is no "favor already loaded bundles". If
> there is no bundle satisfying the Import-Package, the bundle being resolved
> will not go to RESOLVED state. If there are many bundles (quite typical)
> satisfying an Import-Package, with all its additional contraints (versions,
> attributes, uses, ...), then you enter the NP-complete problem that Michal
> mentioned, finding a combination of wiring that satisfy as many bundles as
> possible. This is a problem mainly due to "uses" (since we seldom use
> attributes), where graphs of types must end up matching in the class space.
>
> See (especially) section 3.7 in OSGi 6.0 Core Specification. If anyone can
> get their head around those details in the first pass, it is you Peter. Not
> easy reading...
>
>
>> If the client is matching service api with the correct import package
>> version ranges (requirements defined by entry's), the proxy bundle
>> should find the service api and other imported packages are already
>> loaded.  Eg the client may use the requirements to use the resource
>> service or whatever the new bundle repository standard service is
>> called now to preload the requirements.  The client may also perform
>> upgrades before downloading a service.
> I think this is a misunderstanding as well. By doing what I wrote in the
> beginning (listen on Jini Service Registrations and register "something"
> (Remote proxy or a local proxy) in the OSGi registry) then the client
> bundle doesn't need to know anything. Also, the Jini support bundle gets
> plenty of information both from the OSGi Registry as well as the Jini
> Registry. So, when something disappears from Reggie, remove it from the
> local Registry and vice versa.
>
> Now, the deserialization of the Reggie proxy should detect version changes
> and update a cache. And I think that Paremus idea of "Bundle Garbage
> Collection" is sound, but something for later discussion. Point being; No
> need to figure out what can and can not be unloaded. ALSO, since OSGi
> mandates intermittent service availability, most OSGi applications are
> reasonably capable to handle that the Jini service will "disappear" and is
> required to release any held references to the object(s) so that regular GC
> can toss out the classes and classloader when bundle is unloaded.
>
>> In the majority of cases I don't think there's going to be much state
>> in the smart proxy that can't be loaded via the smart proxy bundle
>> and it's package imports, except for the odd handback, which the
>> client bundle should have the opportunity to resolve before
>> resorting to using an annotation.
> Sorry, I don't understand this statement.
>
>> I'm not quite ready to agree it's too complex and it's unsolveable, I
>> think we should at least explore it and understand it before we junk
>> the idea of supporting OSGi.
> If we are talking osgi<->osgi, I think there is reasonable chance to
> succeed.
>
>> Rather than utilise the Java2 class  loading I was planning to cast
>> ClassLoaders to BundleReference where appropriate and utilise
>> the Bundle.
> My gut says that this is not needed, if you go with my initial proposal. A
> Jini support bundle ends up having access to the BundleContext, from where
> everything else can be reached.
>
>> I did notice you're interpretation  of what I've written is different than
>> mine, so I think I need to put some effort into communicating more
>> effectively.  I think you're interpretation  of codebase annotation
>> "version is fixed" ignores that the annotation is only consulted after
>> determining that the current class is not available in our Bundles
>> currently participating in deserialization.   It doesn't apply to resolved
>> imported packages as annotations aren't used for them at all.
> OR, I have been burned in class resolution in OSGi enough times to have a
> feeling that it is more difficult than it seems. Any simple example I can
> think of would work...
>
>> For example, the first class we attempt to resolve during unmarshalling
>> belongs to a smart proxy, the client Bundle can't find the class. Ask the
>> framework to load the proxy bundle from the codebase annotation, it
>> does so and resolves all necessary package imports declared in its
>> manifest.  We now continue deserializing the smart proxy class fields
>> with the visibility of the smart proxy's bundle.  The smart proxy may
>> contain fields referencing objects resolved from its imports, we ensure
>> those classes deserialize their fields with the visibility of their own
> bundles.
>
> So, the smart proxy's "bundle" is a bundle on the server side as well? The
> smart proxy may also contain objects that are of a class that is not
> visible... And it may be N levels deep from the "field" in the smart proxy.
>
>> Every time we can't  resolve a class we first check if it's a handback
>> or parameter from a preceeding object in the graph, thus we walk
>> our graphs bundle stack.
> This is probably the bit I don't understand at all. On one hand you want to
> depend on OSGi framework to do the resolution, but OTOH you have something
> called a "bundle stack"? What is that?
>
>> If we still haven't resolved a class only then do we load a bundle from
>> it's codebase annotation url and check it can be cast to the field
>> before assigning it.  If it can't be cast to that field, we throw an
> exception.
>
> "assigning it"?  Doesn't the code in reality looks something like;
>
> private Map map;
>
> private void readObject( ObjectInputStream in ) {
>      map = (Map) in.readObject();
> }
>
> where the read object may be of a class not visible to neither the smart
> proxy's classloader nor any helper? Maybe you meant that it will implicitly
> thrown just by the above code.
>
>
>> In the case of a non smart proxy, there is no codebase, deserialization
>> will be loaded by and  rely completely on the visibility of the client
> bundle.
>> I think OSGi will be a lot less dependant on annotations than say a std
> env.
>
> Possibly... I think that largely depend on the usecase.
>
>> Still I guess wiring may be an an option, so as Michael suggests,
> annotate objects with their wiring graphs.
>
> Ok, here is what I see being the issue at hand; you think that it is
> possible to delay the bundle resolution of packages until deserialization
> itself. I think that is not possible. There is a need to bring all
> "non-available classes" into the client (and I think as a bundle is the
> correct solution). So, pick up the bundle reference of the smart proxy, and
> the bundle wiring graph of it. When the smart proxy arrives on the client,
> before doing anything else, load the bundles on the client. After that, it
> should be a "local JVM" problem, both to deserialize the smart proxy as
> well as every object communicated over the network.
>
>> What would we be considering if we hadn't been pre exposed to codebase
> annotations?
>> Standard deserialization uses one classpath, each bundle has its own
> unique classpath.
>
> It is not only about classpath. It is about class space and visibility as
> well. And serialization needs to bypass visibility, just like it bypassed
> other constraints before (such as bypassing constructors and initializing
> final fields), and that is a separate issue than the bundle loading
> mechanism.
>
>
> Cheers
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org  - New Energy for Java
>


Mime
View raw message