ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: OSGi migration may require a special marshaller
Date Tue, 03 Nov 2015 00:46:26 GMT
I think I can see Andrey’s point about the bundles. Having same classdef in
different bundles on sender and receiver sides sounds rather like a bug,
not a feature we should support.

Having read this thread, I think we need to have ClassLoaderResolver on the
sending and receiving sides. Something like this:

*public interface ClassLoaderResolver {*
*    // This method is invoked on the sender side to *
*    // marshal the information about the class loader.*
*    public byte[] writeClassLoader(Object o) throws IgniteException;*

*    // This method is invoked on the receiving side to*
*    // retrieve the class loader based on provided information.*
*    public ClassLoader readClassLoader(byte[]) throws IgniteException;*
*}*

This resolver should be a pluggable configuration for marshallers or
provided in IgniteConfiguration directly. Once we have such resolver, then
we can have a special OSGI implementation for it (perhaps a different one
for different OSGI containers).

Thoughts?

D.

On Mon, Nov 2, 2015 at 4:23 PM, Andrey Kornev <andrewkornev@hotmail.com>
wrote:

> Hey Raul,
>
> What kind of use case do you have in mind when you suggest that a classdef
> may be located in different bundles on different machines? What would one
> have to do to end up in such a predicament? It feels more like a
> configuration/deployment issue rather than a desirable feature.
>
> A more realistic assumption I believe is that the nodes will have (mostly)
> the same set of bundles deployed. (We even enforce it at runtime  by
> leveraging Ignite's PluginProvider.validateNewNode() method: we do not
> allow new nodes join the cluster if their OSGi environment is inconsistent.)
>
> The "bundle name" in my original email really meant "bundle symbolic name
> + bundle version". Sorry for not being clear.
>
> I'm not sure I understand why TCCL is even required for serialization?
>
> Finally, I didn't quite grok your thinking about the custom manifest
> header. Could you please elaborate? What type of information do you propose
> to store in the header? How will the serializer be able to provide that
> info, and how will the deserializer use the info to load the classes? Also,
> who is responsible for making sure that the custom header includes all
> classes to be serialized? And how/when/by whom would such a header be
> generated?
>
> Regards
> Andrey
>
> > Date: Mon, 2 Nov 2015 23:20:41 +0000
> > Subject: RE: OSGi migration may require a special marshaller
> > From: raul@evosent.com
> > To: dev@ignite.apache.org
> >
> > Hi Andrey,
> >
> > Thanks for the participation in this topic.
> >
> > I don't like the solution to incorporate the bundle symbolic name in the
> > serialised form. Nothing guarantees that the classdef will be located
> under
> > the same bundle in both source and target machines. We also have to take
> > into account that OSGi allows for multiple versions of the same
> > bundle/packages to co-exist in the same  container. So it becomes more
> > complex.
> >
> > Using the TCCL should work when serialising, but it'll probably be of no
> > use when deserialising on the other end.
> >
> > I need to enhance Ignite to:
> >
> > 1. Use the TCCL when marshalling on one end.
> > 2. Incorporate the package version of the class in the serialised form if
> > Ignite is running in an OSGi environment.
> > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > through a custom OSGi manifest header or the like, as I explained before.
> > Package version must be taken into account.
> >
> > What do you think?
> >
> > Raúl.
> > On 2 Nov 2015 17:25, "Andrey Kornev" <andrewkornev@hotmail.com> wrote:
> >
> > > Raul,
> > >
> > > The fundamental hurdle we need to jump over to make Ignite
> OSGi-enabled is
> > > the marshalling. More specifically the issue is with deserialization
> of the
> > > classes that are provided by the bundles other than the Ignite bundle
> > > itself.
> > >
> > > When the Ignite transport layer receives a message it needs to figure
> out
> > > how to deserialize the bytes and for that it needs to know the bundle
> that
> > > provides the class to be deserailized. At this point TCCL is of no
> use. To
> > > make things more complex, the class may contain other classes that come
> > > from other bundles, and so on recursively. This means that each object
> in
> > > the hierarchy must be serialized with its bundle name (or bundle id),
> so
> > > that the deserializer will then be able to correctly resolve the class
> > > while traversing the object hierarchy during deserialization.
> > >
> > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> plug
> > > in a custom class resolver. Romain's solution was to use Kryo that does
> > > provide a way to customize class resolution. It has solved the problem
> of
> > > capturing the bundle info and he was able to successfully run Ignite
> as a
> > > bundle in an OSGi container (after some repackaging and inclusion of
> the
> > > manifest). But Kryo-based marshalling introduced a lot of complexity
> to the
> > > code and incorrect use of Kryo's numerous serializers caused some weird
> > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> due
> > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > figure!). Overall the Kryo-based solution is brittle and hard to
> maintain.
> > >
> > > I feel the correct solution to OSGi problem would be to
> > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > original one) to be used in OSGi environment.
> > >
> > > Regards
> > > Andrey
> > >
> > > > From: raulk@apache.org
> > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Hi Romain,
> > > >
> > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> input.
> > > >
> > > > I'm thinking about the situation you describe and I wonder if you're
> > > > exporting Ignite as an OSGi service which is then consumed from other
> > > > bundles. Under this situation, it would be quite easy to reproduce
> the
> > > > behaviour you describe if Ignite is not resolving classes via the
> TCCL.
> > > > Need to dig deeper into that.
> > > >
> > > > Off the top of my head, there are two alternatives to solve it:
> > > >
> > > > 1. Use the TCCL for marshalling/unmarshalling (if not already used)
> – we
> > > > gotta be wary of possible regressions.
> > > > 2. Create a special OSGi header Ignite-Export-Package so that bundles
> > > > containing DTOs can expose packages to Ignite's marshallers.
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big
> Data and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> romain.gilles@misys.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm really interested in this issue:
> > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff
> to
> > > make
> > > > > it work in our osgi environment. The main issue for us now is the
> > > > > serialization. I think it you will have to rework the
> > > OptimizedMarshaller
> > > > > or any other marshaller that works with object that come from
> outside
> > > your
> > > > > class space.
> > > > >
> > > > > We have try kryo that works. Kryo provide an extension point in
> order
> > > to
> > > > > resolve the classes:
> > > > >
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > . With this extension we are able to solve the problem of external
> > > classes.
> > > > > The only issue with kryo is that some classes need a certain care
> in
> > > the
> > > > > serialization process and therefore a specialized serializer.
> > > > >
> > > > > So I would like to know from the community what do think of
> changing
> > > the
> > > > > way the optimized marshaller works or introducing the support of
> yet
> > > > > another marshaller based on a kryo like technology?
> > > > >
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Romain.
> > > > >
> > > > >
> > > > > PS: I'm ready to help in the both cases.
> > > > > "Misys" is the trade name of the Misys group of companies. This
> email
> > > and
> > > > > any attachments have been scanned for known viruses using multiple
> > > > > scanners. This email message is intended for the named recipient
> only.
> > > It
> > > > > may be privileged and/or confidential. If you are not the named
> > > recipient
> > > > > of this email please notify us immediately and do not copy it or
> use
> > > it for
> > > > > any purpose, nor disclose its contents to any other person. This
> email
> > > does
> > > > > not constitute the commencement of legal relations between you and
> > > Misys.
> > > > > Please refer to the executed contract between you and the relevant
> > > member
> > > > > of the Misys group for the identity of the contracting party with
> > > which you
> > > > > are dealing.
> > > > >
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message