ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Romain Gilles <romain.gil...@gmail.com>
Subject Re: OSGi migration may require a special marshaller
Date Tue, 03 Nov 2015 22:01:04 GMT
Hi,
Maybe a missing point. I think but I'm not sure that in the
OptimizedMarshaller there is a caching of already serialized classes. Due
to the dynamic nature of OSGi this may lead to memory leak. In fact if a
bundle is refreshed, it will produce a new BundleRevision and therefore a
new classloader. And if you don't release the class from the previous
BundleRevision then you endup with memory leak. So maybe the Marshaller
interface or somewhere should provide a way to free those classes /
classloaders.

Regards,

Romain

Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <andrewkornev@hotmail.com> a
écrit :

> Romain/Dmitriy,
>
> I prefer Romain's approach, but just curious, in the API you guys are
> proposing why use a byte[] rather than OutputStream/InputStream? With a
> byte[], one would inevitably end up wrapping it into a byte stream class.
> Also, the stream-based interface would be more in line with the Marshaller
> API.
>
> Also for symmetry with the readClass() method, I suggest the writeClass()
> take a Class<?> rather than an object.
>
> Regards
> Andrey
>
> > From: romain.gilles@gmail.com
> > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > Subject: Re: OSGi migration may require a special marshaller
> > To: dev@ignite.apache.org
> >
> > Hi Dmitriy,
> > I think your solution is good. Maybe I will change it a little bit... :P
> > I think you should delegate the Class resolution to the resolver. Because
> > for example with your current solution the marshaller may before (or
> after)
> > store the fqn of the class (maybe only the first time it encounter it)
> but
> > in order to save the classloader context resolution the implementation of
> > the resolver may have to save the package name of the given object and
> some
> > extra info therefore the content package name will be serialized two
> times.
> > Ok, it's not a big deal.
> > But now if the resolver identify that it can reuse the same class loader
> > for a couple of classes. It will be interesting for it to have a
> > serialization context in order to save, let say, classloader/id mapping
> in
> > order to save the id instead of a more longer representation.
> > So I propose something like that:
> > *public interface ClassResolver {*
> > *    // This method is invoked on the sender side to *
> > *    // marshal the information about the class.*
> > *    // where the context is a map style object that is reset/new for
> each
> > object graph serialization.*
> > *    public byte[] writeClass(Object o, Context context) throws
> > IgniteException;*
> >
> > *    // This method is invoked on the receiving side to*
> > *    // retrieve the class based on provided information.*
> > *    // where the context is a map style object that is reset/new for
> each
> > object graph serialization.*
> > *    public Class<?> readClass(byte[], Context context) throws
> > IgniteException;*
> > *}*
> >
> > I think your proposal will solve our issue and maybe also open a door for
> > the osgi development.
> > Let me know what do you think about me proposal? :)
> >
> > Thanks,
> >
> > Romain
> >
> > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <dsetrakyan@apache.org> a
> > écrit :
> >
> > > Romain,
> > >
> > > Can you comment on the ClassLoaderResolver suggestion that I proposed
> > > earlier? Will it solve your problem?
> > >
> > > D.
> > >
> > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <romain.gilles@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Raul,
> > > >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes
> how?
> > > >  - I like the point 2. Maybe for a graph of object that come from
> > > different
> > > > packages / bundles you may have to recursively capture the package
> > > version
> > > > of the individual element of your graph.
> > > >  - For the point 3 I wonder if it will not over-complexify the
> solution.
> > > As
> > > > a developer you will have to think about it. And it is not obvious
> in the
> > > > code where things are serialized or not. You may use lambda in your
> > > > application code therefore the current package become what you call
> a DTO
> > > > package. The current state of ignite does not enforce you to specify
> it
> > > for
> > > > "classical" classloading application. If you introduce this extra
> step
> > > for
> > > > OSGi ready application you will maybe discourage people to use OSGi.
> > > >
> > > > To comeback to our solution. We start we a strong assumption: our
> cluster
> > > > is homogeneous in term of code so, of course, it simplify our life
> :).
> > > >
> > > > BTW if you can introduce an extension point in the class resolution
> > > > mechanism it can be interesting for end user in order to allow them
> to
> > > > optimize it based on there specific use cases.
> > > >
> > > > Romain.
> > > >
> > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <raul@evosent.com>
a
> écrit :
> > > >
> > > > > Hi Andrey,
> > > > >
> > > > > Thanks for the participation in this topic.
> > > > >
> > > > > I don't like the solution to incorporate the bundle symbolic name
> in
> > > the
> > > > > serialised form. Nothing guarantees that the classdef will be
> located
> > > > under
> > > > > the same bundle in both source and target machines. We also have
to
> > > take
> > > > > into account that OSGi allows for multiple versions of the same
> > > > > bundle/packages to co-exist in the same  container. So it becomes
> more
> > > > > complex.
> > > > >
> > > > > Using the TCCL should work when serialising, but it'll probably be
> of
> > > no
> > > > > use when deserialising on the other end.
> > > > >
> > > > > I need to enhance Ignite to:
> > > > >
> > > > > 1. Use the TCCL when marshalling on one end.
> > > > > 2. Incorporate the package version of the class in the serialised
> form
> > > if
> > > > > Ignite is running in an OSGi environment.
> > > > > 3. On the receiver end, discover cache entities / DTOs in all
> bundles
> > > > > through a custom OSGi manifest header or the like, as I explained
> > > before.
> > > > > Package version must be taken into account.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Raúl.
> > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <andrewkornev@hotmail.com>
> wrote:
> > > > >
> > > > > > Raul,
> > > > > >
> > > > > > The fundamental hurdle we need to jump over to make Ignite
> > > OSGi-enabled
> > > > > is
> > > > > > the marshalling. More specifically the issue is with
> deserialization
> > > of
> > > > > the
> > > > > > classes that are provided by the bundles other than the Ignite
> bundle
> > > > > > itself.
> > > > > >
> > > > > > When the Ignite transport layer receives a message it needs
to
> figure
> > > > out
> > > > > > how to deserialize the bytes and for that it needs to know the
> bundle
> > > > > that
> > > > > > provides the class to be deserailized. At this point TCCL is
of
> no
> > > use.
> > > > > To
> > > > > > make things more complex, the class may contain other classes
> that
> > > come
> > > > > > from other bundles, and so on recursively. This means that each
> > > object
> > > > in
> > > > > > the hierarchy must be serialized with its bundle name (or bundle
> id),
> > > > so
> > > > > > that the deserializer will then be able to correctly resolve
the
> > > class
> > > > > > while traversing the object hierarchy during deserialization.
> > > > > >
> > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the
> ability to
> > > > > plug
> > > > > > in a custom class resolver. Romain's solution was to use Kryo
> that
> > > does
> > > > > > provide a way to customize class resolution. It has solved the
> > > problem
> > > > of
> > > > > > capturing the bundle info and he was able to successfully run
> Ignite
> > > > as a
> > > > > > bundle in an OSGi container (after some repackaging and
> inclusion of
> > > > the
> > > > > > manifest). But Kryo-based marshalling introduced a lot of
> complexity
> > > to
> > > > > the
> > > > > > code and incorrect use of Kryo's numerous serializers caused
some
> > > weird
> > > > > > hard-to-debug issues in the Ignite core (like duplicate cache
> entries
> > > > due
> > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class
--
> go
> > > > > > figure!). Overall the Kryo-based solution is brittle and hard
to
> > > > > maintain.
> > > > > >
> > > > > > I feel the correct solution to OSGi problem would be to
> > > > > > 1) enhance the OptimizedMarshaller to allow custom class
> resolution.
> > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition
to
> the
> > > > > > original one) to be used in OSGi environment.
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > > From: raulk@apache.org
> > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: dev@ignite.apache.org
> > > > > > >
> > > > > > > Hi Romain,
> > > > > > >
> > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate
> your
> > > > > input.
> > > > > > >
> > > > > > > I'm thinking about the situation you describe and I wonder
if
> > > you're
> > > > > > > exporting Ignite as an OSGi service which is then consumed
from
> > > other
> > > > > > > bundles. Under this situation, it would be quite easy to
> reproduce
> > > > the
> > > > > > > behaviour you describe if Ignite is not resolving classes
via
> the
> > > > TCCL.
> > > > > > > Need to dig deeper into that.
> > > > > > >
> > > > > > > Off the top of my head, there are two alternatives to solve
it:
> > > > > > >
> > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already
> > > used) –
> > > > > we
> > > > > > > gotta be wary of possible regressions.
> > > > > > > 2. Create a special OSGi header Ignite-Export-Package so
that
> > > bundles
> > > > > > > containing DTOs can expose packages to Ignite's marshallers.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > *Raúl Kripalani*
> > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration,
> Big
> > > Data
> > > > > and
> > > > > > > Messaging Engineer
> > > > > > > http://about.me/raulkripalani |
> > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > >
> > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain <
> > > > > romain.gilles@misys.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'm really interested in this issue:
> > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270
. We some
> > > stuff
> > > > to
> > > > > > make
> > > > > > > > it work in our osgi environment. The main issue for
us now
> is the
> > > > > > > > serialization. I think it you will have to rework
the
> > > > > > OptimizedMarshaller
> > > > > > > > or any other marshaller that works with object that
come from
> > > > outside
> > > > > > your
> > > > > > > > class space.
> > > > > > > >
> > > > > > > > We have try kryo that works. Kryo provide an extension
point
> in
> > > > order
> > > > > > to
> > > > > > > > resolve the classes:
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > . With this extension we are able to solve the problem
of
> > > external
> > > > > > classes.
> > > > > > > > The only issue with kryo is that some classes need
a certain
> care
> > > > in
> > > > > > the
> > > > > > > > serialization process and therefore a specialized
serializer.
> > > > > > > >
> > > > > > > > So I would like to know from the community what do
think of
> > > > changing
> > > > > > the
> > > > > > > > way the optimized marshaller works or introducing
the
> support of
> > > > yet
> > > > > > > > another marshaller based on a kryo like technology?
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks in advance,
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > >
> > > > > > > > Romain.
> > > > > > > >
> > > > > > > >
> > > > > > > > PS: I'm ready to help in the both cases.
> > > > > > > > "Misys" is the trade name of the Misys group of companies.
> This
> > > > email
> > > > > > and
> > > > > > > > any attachments have been scanned for known viruses
using
> > > multiple
> > > > > > > > scanners. This email message is intended for the named
> recipient
> > > > > only.
> > > > > > It
> > > > > > > > may be privileged and/or confidential. If you are
not the
> named
> > > > > > recipient
> > > > > > > > of this email please notify us immediately and do
not copy
> it or
> > > > use
> > > > > > it for
> > > > > > > > any purpose, nor disclose its contents to any other
person.
> This
> > > > > email
> > > > > > does
> > > > > > > > not constitute the commencement of legal relations
between
> you
> > > and
> > > > > > Misys.
> > > > > > > > Please refer to the executed contract between you
and the
> > > relevant
> > > > > > member
> > > > > > > > of the Misys group for the identity of the contracting
party
> with
> > > > > > which you
> > > > > > > > are dealing.
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message