ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: OSGi migration may require a special marshaller
Date Wed, 04 Nov 2015 17:26:17 GMT
On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <romain.gilles@gmail.com>
wrote:

> Hi Dmitriy,
> I found this post that explain how to find a bundle based on its bundle
> name and version.
> This post explain for the past to now how to do it in the standard way with
> "pull" approach: https://www.eclipse.org/forums/index.php/t/205719/
> Regarding the version of OSGi you want to support then some solutions will
> works some others will not.
> There is an other way to do this stuff without using those "pull" style
> approach based on BundleTracker. If you want I can give you the code to do
> it with BundlTracker. I think with this solution you will support a wider
> range of OSGi version.
>

Romain, if you can provide a generic code sample to look up a ClassLoader
in OSGI based on manifest properties, would be great.


>
> Le mer. 4 nov. 2015 à 04:07, Dmitriy Setrakyan <dsetrakyan@apache.org> a
> écrit :
>
> > Andrey, et al,
> >
> > Can you provide a way on how to get a required Bundle object based on,
> say,
> > bundle symbolic name and version?
> >
> > I have reached the end of the internet trying to find a way in OSGI to
> look
> > up a Bundle based on something other than an actual Class belonging to
> that
> > bundle, but no luck.
> >
> > D.
> >
> > On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <andrewkornev@hotmail.com>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > I think your approach will work, but I let Romain respond.
> > >
> > > Also, in terms of the implementation, please keep in mind that the
> > > resolver must be called for each non-JDK and non-Ignite core class (it
> > > would probably make sense to eschew storing class loaders for such
> > classes
> > > in favor of compactness of the serialized representation -- see below).
> > > Also, it's worth keeping a cache of already resolved class loaders per
> > > marshaller invocation (this is probably the context that Romain has
> > > mentioned in his previous posting) to minimize the number of the
> resolver
> > > calls.
> > >
> > > In terms of the resolver's implementation, the simplest way to
> serialize
> > > the class loader would be by capturing two pieces of information (both
> > > strings): the bundle symbolic name and the bundle version. This
> approach
> > > however may result in bloating of the serialized representation: I'd
> > > roughly estimate the overhead per element to be at least 20-30 bytes
> (the
> > > length of the symbolic name string, plus the length of the version
> > string).
> > > There are way to reduce the overhead (like serializing the hash code of
> > the
> > > bundle id string rather than the string itself, and then come up with a
> > > clever way of resolving the hash collisions), but they all come at cost
> > of
> > > increased complexity...
> > >
> > > An alternative approach would be rely on the special bundle id which is
> > an
> > > integer and is generated by the OSGi container. But in this case, all
> > nodes
> > > must ensure that all the bundles have consistent ids (bundle A with id
> 42
> > > on node N1, has the same id on every other node) which is not easy --
> > while
> > > not entirely impossible -- to guarantee. As long as the nodes are
> > > homogeneous (have the same set of bundles deployed) the OSGi container
> is
> > > guaranteed to assign to the bundles the same ids.
> > >
> > > Thanks
> > > Andrey
> > >
> > > > From: dsetrakyan@apache.org
> > > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > > Subject: Re: OSGi migration may require a special marshaller
> > > > To: dev@ignite.apache.org
> > > >
> > > > Romain,
> > > >
> > > > In the upcoming release we will be deprecating the
> OptimizedMarshaller
> > > and
> > > > will be switching to a default internal marshaller (which is based on
> > the
> > > > new PortableMarshaller donated by GridGain).
> > > >
> > > > Having said that, we may be able to pass BinaryWriter and
> BinaryReader
> > > > instead of byte arrays. This will be pretty close to passing the
> > stream,
> > > as
> > > > suggested by Andrey.
> > > >
> > > > Also, I still think that we should only resolve class loaders and not
> > the
> > > > class itself. The main reason is that Ignite will encode class names
> > into
> > > > an integer hash code and will store the integer->class-fqn mapping
in
> > > > internal replicated cache. I doubt users will get more efficient than
> > an
> > > > integer (4 bytes) for a class name.
> > > >
> > > > On the receiving side, once we are able to get the right class
> loader,
> > we
> > > > can easily get the proper class by calling
> > > ClassLoader.findClass(class-fqn).
> > > >
> > > > Thoughts?
> > > >
> > > > D.
> > > >
> > > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <
> romain.gilles@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Maybe a missing point. I think but I'm not sure that in the
> > > > > OptimizedMarshaller there is a caching of already serialized
> classes.
> > > Due
> > > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> > if
> > > a
> > > > > bundle is refreshed, it will produce a new BundleRevision and
> > > therefore a
> > > > > new classloader. And if you don't release the class from the
> previous
> > > > > BundleRevision then you endup with memory leak. So maybe the
> > Marshaller
> > > > > interface or somewhere should provide a way to free those classes
/
> > > > > classloaders.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Romain
> > > > >
> > > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <
> andrewkornev@hotmail.com
> > >
> > > a
> > > > > écrit :
> > > > >
> > > > > > Romain/Dmitriy,
> > > > > >
> > > > > > I prefer Romain's approach, but just curious, in the API you
guys
> > are
> > > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > > With a
> > > > > > byte[], one would inevitably end up wrapping it into a byte
> stream
> > > class.
> > > > > > Also, the stream-based interface would be more in line with
the
> > > > > Marshaller
> > > > > > API.
> > > > > >
> > > > > > Also for symmetry with the readClass() method, I suggest the
> > > writeClass()
> > > > > > take a Class<?> rather than an object.
> > > > > >
> > > > > > Regards
> > > > > > Andrey
> > > > > >
> > > > > > > From: romain.gilles@gmail.com
> > > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > > To: dev@ignite.apache.org
> > > > > > >
> > > > > > > Hi Dmitriy,
> > > > > > > I think your solution is good. Maybe I will change it a
little
> > > bit...
> > > > > :P
> > > > > > > I think you should delegate the Class resolution to the
> resolver.
> > > > > Because
> > > > > > > for example with your current solution the marshaller may
> before
> > > (or
> > > > > > after)
> > > > > > > store the fqn of the class (maybe only the first time it
> > encounter
> > > it)
> > > > > > but
> > > > > > > in order to save the classloader context resolution the
> > > implementation
> > > > > of
> > > > > > > the resolver may have to save the package name of the given
> > object
> > > and
> > > > > > some
> > > > > > > extra info therefore the content package name will be
> serialized
> > > two
> > > > > > times.
> > > > > > > Ok, it's not a big deal.
> > > > > > > But now if the resolver identify that it can reuse the
same
> class
> > > > > loader
> > > > > > > for a couple of classes. It will be interesting for it
to have
> a
> > > > > > > serialization context in order to save, let say, classloader/id
> > > mapping
> > > > > > in
> > > > > > > order to save the id instead of a more longer representation.
> > > > > > > So I propose something like that:
> > > > > > > *public interface ClassResolver {*
> > > > > > > *    // This method is invoked on the sender side to *
> > > > > > > *    // marshal the information about the class.*
> > > > > > > *    // where the context is a map style object that is
> reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > *    public byte[] writeClass(Object o, Context context)
throws
> > > > > > > IgniteException;*
> > > > > > >
> > > > > > > *    // This method is invoked on the receiving side to*
> > > > > > > *    // retrieve the class based on provided information.*
> > > > > > > *    // where the context is a map style object that is
> reset/new
> > > for
> > > > > > each
> > > > > > > object graph serialization.*
> > > > > > > *    public Class<?> readClass(byte[], Context context)
throws
> > > > > > > IgniteException;*
> > > > > > > *}*
> > > > > > >
> > > > > > > I think your proposal will solve our issue and maybe also
open
> a
> > > door
> > > > > for
> > > > > > > the osgi development.
> > > > > > > Let me know what do you think about me proposal? :)
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Romain
> > > > > > >
> > > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > > a
> > > > > > > écrit :
> > > > > > >
> > > > > > > > Romain,
> > > > > > > >
> > > > > > > > Can you comment on the ClassLoaderResolver suggestion
that I
> > > proposed
> > > > > > > > earlier? Will it solve your problem?
> > > > > > > >
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > > romain.gilles@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Raul,
> > > > > > > > >  - Do you plan to use the TCCL when marshalling
in OSGi
> env?
> > > If yes
> > > > > > how?
> > > > > > > > >  - I like the point 2. Maybe for a graph of object
that
> come
> > > from
> > > > > > > > different
> > > > > > > > > packages / bundles you may have to recursively
capture the
> > > package
> > > > > > > > version
> > > > > > > > > of the individual element of your graph.
> > > > > > > > >  - For the point 3 I wonder if it will not over-complexify
> > the
> > > > > > solution.
> > > > > > > > As
> > > > > > > > > a developer you will have to think about it.
And it is not
> > > obvious
> > > > > > in the
> > > > > > > > > code where things are serialized or not. You
may use lambda
> > in
> > > your
> > > > > > > > > application code therefore the current package
become what
> > you
> > > call
> > > > > > a DTO
> > > > > > > > > package. The current state of ignite does not
enforce you
> to
> > > > > specify
> > > > > > it
> > > > > > > > for
> > > > > > > > > "classical" classloading application. If you
introduce this
> > > extra
> > > > > > step
> > > > > > > > for
> > > > > > > > > OSGi ready application you will maybe discourage
people to
> > use
> > > > > OSGi.
> > > > > > > > >
> > > > > > > > > To comeback to our solution. We start we a strong
> assumption:
> > > our
> > > > > > cluster
> > > > > > > > > is homogeneous in term of code so, of course,
it simplify
> our
> > > life
> > > > > > :).
> > > > > > > > >
> > > > > > > > > BTW if you can introduce an extension point in
the class
> > > resolution
> > > > > > > > > mechanism it can be interesting for end user
in order to
> > allow
> > > them
> > > > > > to
> > > > > > > > > optimize it based on there specific use cases.
> > > > > > > > >
> > > > > > > > > Romain.
> > > > > > > > >
> > > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani
<
> > raul@evosent.com>
> > > a
> > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > Hi Andrey,
> > > > > > > > > >
> > > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > > >
> > > > > > > > > > I don't like the solution to incorporate
the bundle
> > symbolic
> > > name
> > > > > > in
> > > > > > > > the
> > > > > > > > > > serialised form. Nothing guarantees that
the classdef
> will
> > be
> > > > > > located
> > > > > > > > > under
> > > > > > > > > > the same bundle in both source and target
machines. We
> also
> > > have
> > > > > to
> > > > > > > > take
> > > > > > > > > > into account that OSGi allows for multiple
versions of
> the
> > > same
> > > > > > > > > > bundle/packages to co-exist in the same
 container. So it
> > > becomes
> > > > > > more
> > > > > > > > > > complex.
> > > > > > > > > >
> > > > > > > > > > Using the TCCL should work when serialising,
but it'll
> > > probably
> > > > > be
> > > > > > of
> > > > > > > > no
> > > > > > > > > > use when deserialising on the other end.
> > > > > > > > > >
> > > > > > > > > > I need to enhance Ignite to:
> > > > > > > > > >
> > > > > > > > > > 1. Use the TCCL when marshalling on one
end.
> > > > > > > > > > 2. Incorporate the package version of the
class in the
> > > serialised
> > > > > > form
> > > > > > > > if
> > > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > > 3. On the receiver end, discover cache entities
/ DTOs in
> > all
> > > > > > bundles
> > > > > > > > > > through a custom OSGi manifest header or
the like, as I
> > > explained
> > > > > > > > before.
> > > > > > > > > > Package version must be taken into account.
> > > > > > > > > >
> > > > > > > > > > What do you think?
> > > > > > > > > >
> > > > > > > > > > Raúl.
> > > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > > andrewkornev@hotmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Raul,
> > > > > > > > > > >
> > > > > > > > > > > The fundamental hurdle we need to jump
over to make
> > Ignite
> > > > > > > > OSGi-enabled
> > > > > > > > > > is
> > > > > > > > > > > the marshalling. More specifically
the issue is with
> > > > > > deserialization
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > classes that are provided by the bundles
other than the
> > > Ignite
> > > > > > bundle
> > > > > > > > > > > itself.
> > > > > > > > > > >
> > > > > > > > > > > When the Ignite transport layer receives
a message it
> > > needs to
> > > > > > figure
> > > > > > > > > out
> > > > > > > > > > > how to deserialize the bytes and for
that it needs to
> > know
> > > the
> > > > > > bundle
> > > > > > > > > > that
> > > > > > > > > > > provides the class to be deserailized.
At this point
> TCCL
> > > is of
> > > > > > no
> > > > > > > > use.
> > > > > > > > > > To
> > > > > > > > > > > make things more complex, the class
may contain other
> > > classes
> > > > > > that
> > > > > > > > come
> > > > > > > > > > > from other bundles, and so on recursively.
This means
> > that
> > > each
> > > > > > > > object
> > > > > > > > > in
> > > > > > > > > > > the hierarchy must be serialized with
its bundle name
> (or
> > > > > bundle
> > > > > > id),
> > > > > > > > > so
> > > > > > > > > > > that the deserializer will then be
able to correctly
> > > resolve
> > > > > the
> > > > > > > > class
> > > > > > > > > > > while traversing the object hierarchy
during
> > > deserialization.
> > > > > > > > > > >
> > > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller
is lacking
> > the
> > > > > > ability to
> > > > > > > > > > plug
> > > > > > > > > > > in a custom class resolver. Romain's
solution was to
> use
> > > Kryo
> > > > > > that
> > > > > > > > does
> > > > > > > > > > > provide a way to customize class resolution.
It has
> > solved
> > > the
> > > > > > > > problem
> > > > > > > > > of
> > > > > > > > > > > capturing the bundle info and he was
able to
> successfully
> > > run
> > > > > > Ignite
> > > > > > > > > as a
> > > > > > > > > > > bundle in an OSGi container (after
some repackaging and
> > > > > > inclusion of
> > > > > > > > > the
> > > > > > > > > > > manifest). But Kryo-based marshalling
introduced a lot
> of
> > > > > > complexity
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > code and incorrect use of Kryo's numerous
serializers
> > > caused
> > > > > some
> > > > > > > > weird
> > > > > > > > > > > hard-to-debug issues in the Ignite
core (like duplicate
> > > cache
> > > > > > entries
> > > > > > > > > due
> > > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > > class --
> > > > > > go
> > > > > > > > > > > figure!). Overall the Kryo-based solution
is brittle
> and
> > > hard
> > > > > to
> > > > > > > > > > maintain.
> > > > > > > > > > >
> > > > > > > > > > > I feel the correct solution to OSGi
problem would be to
> > > > > > > > > > > 1) enhance the OptimizedMarshaller
to allow custom
> class
> > > > > > resolution.
> > > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller
(in
> > > addition to
> > > > > > the
> > > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Andrey
> > > > > > > > > > >
> > > > > > > > > > > > From: raulk@apache.org
> > > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47
+0000
> > > > > > > > > > > > Subject: Re: OSGi migration may
require a special
> > > marshaller
> > > > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Romain,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm working on the OSGi compatibility
of Ignite. I
> > > appreciate
> > > > > > your
> > > > > > > > > > input.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm thinking about the situation
you describe and I
> > > wonder if
> > > > > > > > you're
> > > > > > > > > > > > exporting Ignite as an OSGi service
which is then
> > > consumed
> > > > > from
> > > > > > > > other
> > > > > > > > > > > > bundles. Under this situation,
it would be quite easy
> > to
> > > > > > reproduce
> > > > > > > > > the
> > > > > > > > > > > > behaviour you describe if Ignite
is not resolving
> > > classes via
> > > > > > the
> > > > > > > > > TCCL.
> > > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > > >
> > > > > > > > > > > > Off the top of my head, there
are two alternatives to
> > > solve
> > > > > it:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling
(if not
> > > already
> > > > > > > > used) –
> > > > > > > > > > we
> > > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > > 2. Create a special OSGi header
Ignite-Export-Package
> > so
> > > that
> > > > > > > > bundles
> > > > > > > > > > > > containing DTOs can expose packages
to Ignite's
> > > marshallers.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > >
> > > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > > PMC & Committer @ Apache Ignite,
Apache Camel |
> > > Integration,
> > > > > > Big
> > > > > > > > Data
> > > > > > > > > > and
> > > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > > http://about.me/raulkripalani
|
> > > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > > http://blog.raulkr.net | twitter:
@raulvk
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM,
Gilles, Romain <
> > > > > > > > > > romain.gilles@misys.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm really interested in
this issue:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270
> .
> > We
> > > > > some
> > > > > > > > stuff
> > > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > > > it work in our osgi environment.
The main issue for
> > us
> > > now
> > > > > > is the
> > > > > > > > > > > > > serialization. I think it
you will have to rework
> the
> > > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > > or any other marshaller that
works with object that
> > > come
> > > > > from
> > > > > > > > > outside
> > > > > > > > > > > your
> > > > > > > > > > > > > class space.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have try kryo that works.
Kryo provide an
> > extension
> > > > > point
> > > > > > in
> > > > > > > > > order
> > > > > > > > > > > to
> > > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > > . With this extension we
are able to solve the
> > problem
> > > of
> > > > > > > > external
> > > > > > > > > > > classes.
> > > > > > > > > > > > > The only issue with kryo
is that some classes need
> a
> > > > > certain
> > > > > > care
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > serialization process and
therefore a specialized
> > > > > serializer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So I would like to know from
the community what do
> > > think of
> > > > > > > > > changing
> > > > > > > > > > > the
> > > > > > > > > > > > > way the optimized marshaller
works or introducing
> the
> > > > > > support of
> > > > > > > > > yet
> > > > > > > > > > > > > another marshaller based
on a kryo like technology?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Romain.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > PS: I'm ready to help in
the both cases.
> > > > > > > > > > > > > "Misys" is the trade name
of the Misys group of
> > > companies.
> > > > > > This
> > > > > > > > > email
> > > > > > > > > > > and
> > > > > > > > > > > > > any attachments have been
scanned for known viruses
> > > using
> > > > > > > > multiple
> > > > > > > > > > > > > scanners. This email message
is intended for the
> > named
> > > > > > recipient
> > > > > > > > > > only.
> > > > > > > > > > > It
> > > > > > > > > > > > > may be privileged and/or
confidential. If you are
> not
> > > the
> > > > > > named
> > > > > > > > > > > recipient
> > > > > > > > > > > > > of this email please notify
us immediately and do
> not
> > > copy
> > > > > > it or
> > > > > > > > > use
> > > > > > > > > > > it for
> > > > > > > > > > > > > any purpose, nor disclose
its contents to any other
> > > person.
> > > > > > This
> > > > > > > > > > email
> > > > > > > > > > > does
> > > > > > > > > > > > > not constitute the commencement
of legal relations
> > > between
> > > > > > you
> > > > > > > > and
> > > > > > > > > > > Misys.
> > > > > > > > > > > > > Please refer to the executed
contract between you
> and
> > > the
> > > > > > > > relevant
> > > > > > > > > > > member
> > > > > > > > > > > > > of the Misys group for the
identity of the
> > contracting
> > > > > party
> > > > > > with
> > > > > > > > > > > which you
> > > > > > > > > > > > > are dealing.
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message