ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Romain Gilles <romain.gil...@gmail.com>
Subject Re: OSGi migration may require a special marshaller
Date Tue, 03 Nov 2015 21:41:51 GMT
Hi Raul,
I had a look to what has been done in camel. I think here the use cases
came be different.
Let say I want to run a closure against the grid like this:

IgniteCompute compute  = ignite.compute();
// Execute closure on all cluster nodes.Collection<Integer> res = compute.apply(
    String::length,
    Arrays.asList("How many characters".split(" "))
);
     // Add all the word lengths received from cluster nodes.

int total = res.stream().mapToInt(Integer::intValue).sum();

Then I think, even in more complex and realistic use cases, you may don't
want to export the implementation details of your closure. But this closure
will be serialized by the Ignite marshalling framework.

Romain.

Le mar. 3 nov. 2015 à 20:18, Raul Kripalani <raulk@apache.org> a écrit :

> Hey guys,
>
> About the TCCL, I need to dig deeper into how serialisation is being done
> now. If, at any point, we are resolving a class through the classloader of
> a particular class, e.g. Ignition.getClass().getClassLoader(), etc. we are
> using the class space of the bundle containing that class. If we are using
> a class from Ignite, we obviously wouldn't be able to find a custom DTO
> provided by a user.
>
> In Camel we have found this problem several times and we've solved it by
> changing the TCCL, or using the TCCL [1] [2] to resolve classes.
>
> As I said, I need to look into this deeper and I'll have some time on
> Thursday, so I hope you don't mind waiting a little bit. I have already
> "bundle-fied" most of Ignite modules in the corresponding branch [3], so my
> next step is to test out the Ignite examples inside an OSGi environment
> (Karaf 4).
>
> [1] https://issues.apache.org/jira/browse/CAMEL-5748
> [2] https://issues.apache.org/jira/browse/CAMEL-5722
> [3] https://github.com/apache/ignite/tree/ignite-1527
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Tue, Nov 3, 2015 at 10:38 AM, Romain Gilles <romain.gilles@gmail.com>
> wrote:
>
> > Hi Raul,
> >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> >  - I like the point 2. Maybe for a graph of object that come from
> different
> > packages / bundles you may have to recursively capture the package
> version
> > of the individual element of your graph.
> >  - For the point 3 I wonder if it will not over-complexify the solution.
> As
> > a developer you will have to think about it. And it is not obvious in the
> > code where things are serialized or not. You may use lambda in your
> > application code therefore the current package become what you call a DTO
> > package. The current state of ignite does not enforce you to specify it
> for
> > "classical" classloading application. If you introduce this extra step
> for
> > OSGi ready application you will maybe discourage people to use OSGi.
> >
> > To comeback to our solution. We start we a strong assumption: our cluster
> > is homogeneous in term of code so, of course, it simplify our life :).
> >
> > BTW if you can introduce an extension point in the class resolution
> > mechanism it can be interesting for end user in order to allow them to
> > optimize it based on there specific use cases.
> >
> > Romain.
> >
> > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <raul@evosent.com> a écrit :
> >
> > > Hi Andrey,
> > >
> > > Thanks for the participation in this topic.
> > >
> > > I don't like the solution to incorporate the bundle symbolic name in
> the
> > > serialised form. Nothing guarantees that the classdef will be located
> > under
> > > the same bundle in both source and target machines. We also have to
> take
> > > into account that OSGi allows for multiple versions of the same
> > > bundle/packages to co-exist in the same  container. So it becomes more
> > > complex.
> > >
> > > Using the TCCL should work when serialising, but it'll probably be of
> no
> > > use when deserialising on the other end.
> > >
> > > I need to enhance Ignite to:
> > >
> > > 1. Use the TCCL when marshalling on one end.
> > > 2. Incorporate the package version of the class in the serialised form
> if
> > > Ignite is running in an OSGi environment.
> > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > through a custom OSGi manifest header or the like, as I explained
> before.
> > > Package version must be taken into account.
> > >
> > > What do you think?
> > >
> > > Raúl.
> > > On 2 Nov 2015 17:25, "Andrey Kornev" <andrewkornev@hotmail.com> wrote:
> > >
> > > > Raul,
> > > >
> > > > The fundamental hurdle we need to jump over to make Ignite
> OSGi-enabled
> > > is
> > > > the marshalling. More specifically the issue is with deserialization
> of
> > > the
> > > > classes that are provided by the bundles other than the Ignite bundle
> > > > itself.
> > > >
> > > > When the Ignite transport layer receives a message it needs to figure
> > out
> > > > how to deserialize the bytes and for that it needs to know the bundle
> > > that
> > > > provides the class to be deserailized. At this point TCCL is of no
> use.
> > > To
> > > > make things more complex, the class may contain other classes that
> come
> > > > from other bundles, and so on recursively. This means that each
> object
> > in
> > > > the hierarchy must be serialized with its bundle name (or bundle id),
> > so
> > > > that the deserializer will then be able to correctly resolve the
> class
> > > > while traversing the object hierarchy during deserialization.
> > > >
> > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> > > plug
> > > > in a custom class resolver. Romain's solution was to use Kryo that
> does
> > > > provide a way to customize class resolution. It has solved the
> problem
> > of
> > > > capturing the bundle info and he was able to successfully run Ignite
> > as a
> > > > bundle in an OSGi container (after some repackaging and inclusion of
> > the
> > > > manifest). But Kryo-based marshalling introduced a lot of complexity
> to
> > > the
> > > > code and incorrect use of Kryo's numerous serializers caused some
> weird
> > > > hard-to-debug issues in the Ignite core (like duplicate cache entries
> > due
> > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > > > figure!). Overall the Kryo-based solution is brittle and hard to
> > > maintain.
> > > >
> > > > I feel the correct solution to OSGi problem would be to
> > > > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > > > original one) to be used in OSGi environment.
> > > >
> > > > Regards
> > > > Andrey
> > > >
> > > > > From: raulk@apache.org
> > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: dev@ignite.apache.org
> > > > >
> > > > > Hi Romain,
> > > > >
> > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> > > input.
> > > > >
> > > > > I'm thinking about the situation you describe and I wonder if
> you're
> > > > > exporting Ignite as an OSGi service which is then consumed from
> other
> > > > > bundles. Under this situation, it would be quite easy to reproduce
> > the
> > > > > behaviour you describe if Ignite is not resolving classes via the
> > TCCL.
> > > > > Need to dig deeper into that.
> > > > >
> > > > > Off the top of my head, there are two alternatives to solve it:
> > > > >
> > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> used) –
> > > we
> > > > > gotta be wary of possible regressions.
> > > > > 2. Create a special OSGi header Ignite-Export-Package so that
> bundles
> > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > >
> > > > > Regards,
> > > > >
> > > > > *Raúl Kripalani*
> > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
Big
> Data
> > > and
> > > > > Messaging Engineer
> > > > > http://about.me/raulkripalani |
> > > http://www.linkedin.com/in/raulkripalani
> > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > >
> > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > romain.gilles@misys.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm really interested in this issue:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some
> stuff
> > to
> > > > make
> > > > > > it work in our osgi environment. The main issue for us now is
the
> > > > > > serialization. I think it you will have to rework the
> > > > OptimizedMarshaller
> > > > > > or any other marshaller that works with object that come from
> > outside
> > > > your
> > > > > > class space.
> > > > > >
> > > > > > We have try kryo that works. Kryo provide an extension point
in
> > order
> > > > to
> > > > > > resolve the classes:
> > > > > >
> > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > . With this extension we are able to solve the problem of
> external
> > > > classes.
> > > > > > The only issue with kryo is that some classes need a certain
care
> > in
> > > > the
> > > > > > serialization process and therefore a specialized serializer.
> > > > > >
> > > > > > So I would like to know from the community what do think of
> > changing
> > > > the
> > > > > > way the optimized marshaller works or introducing the support
of
> > yet
> > > > > > another marshaller based on a kryo like technology?
> > > > > >
> > > > > >
> > > > > > Thanks in advance,
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > >
> > > > > > Romain.
> > > > > >
> > > > > >
> > > > > > PS: I'm ready to help in the both cases.
> > > > > > "Misys" is the trade name of the Misys group of companies. This
> > email
> > > > and
> > > > > > any attachments have been scanned for known viruses using
> multiple
> > > > > > scanners. This email message is intended for the named recipient
> > > only.
> > > > It
> > > > > > may be privileged and/or confidential. If you are not the named
> > > > recipient
> > > > > > of this email please notify us immediately and do not copy it
or
> > use
> > > > it for
> > > > > > any purpose, nor disclose its contents to any other person.
This
> > > email
> > > > does
> > > > > > not constitute the commencement of legal relations between you
> and
> > > > Misys.
> > > > > > Please refer to the executed contract between you and the
> relevant
> > > > member
> > > > > > of the Misys group for the identity of the contracting party
with
> > > > which you
> > > > > > are dealing.
> > > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message