activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martyn Taylor <mtay...@redhat.com>
Subject Re: [DISCUSS] Custom Object Serialisation Support
Date Fri, 02 Jun 2017 09:26:51 GMT
So, I could originally see a requirement for controlling the
(de)serialization process for performance or security reasons, whilst also
controlling data format.  I still think having something light weight that
gives users control over this (outside of overriding the java serialization
methods) may be useful.  It would only take a couple of lines of code in
the client to do it.

But, if this thread is really only about integrating multiple technologies,
by controlling bytes that are sent over the wire then I have to agree with
Tim, in that Camel does seem to be a good fit for this problem.  Michael, I
can see your point re: the success of the Kafka model, if you feel that
this is largely down to the API and the abstraction of the serialization
process, how about just wrapping a Camel context?

On Fri, Jun 2, 2017 at 3:28 AM, Clebert Suconic <clebert.suconic@gmail.com>
wrote:

> I have a mixed feeling about this.
>
> At the same time this would fix it for both options. As it wouldn't be part
> of the project so other people would have to find your library.
>
>
>
> However this common tooling is an issue in other cases as well.  Like I
> know some users need a pooled connection factory for both core and qpid
> JMS.
>
>
>
> So if we could create a JMS-tools package either in Artemis or somewhere
> else.  We could have this as part of the tooling.
>
>
>
> But let me sleep over anyways.  And perhaps someone will have a better idea
> as I sleep.
>
>
> On Thu, Jun 1, 2017 at 6:51 PM Michael André Pearce <
> michael.andre.pearce@me.com> wrote:
>
> > I have been looking at an alternative approach (i think its a little less
> > optimal), but in thoughts it might be more acceptable for those not
> wishing
> > this into the core jms client.
> > It means the custom object serialisation is a optional add-on rather than
> > coded into the core jms client, similar idea to the way spring wraps
> > connection factories for caching connection, and others.
> >
> >
> > https://github.com/michaelandrepearce/activemq-artemis/tree/
> SerdesExtension/artemis-jms-client-extensions
> > <
> > https://github.com/michaelandrepearce/activemq-artemis/tree/
> SerdesExtension/artemis-jms-client-extensions
> > >
> >
> > It essentially is wrapping at the JMS api level, so actually would/should
> > work for any JMS Client not just artemis, e.g. could wrap QPID client.
> >
> > Thoughts on this approach rather than native integration to the clients?
> >
> >
> >
> > > On 1 Jun 2017, at 21:54, Matt Pavlovich <mattrpav@gmail.com> wrote:
> > >
> > > I understand the usefulness of adding this to the Client Connection
> > Factory vs kicking all the way up to an integration stack (like Camel).
> In
> > my eyes, its about solving for cross-platform object serialization using
> > byte payloads instead of the problems associated with non-portable/cross
> > platform ObjectMessage and MapMessage JMS message types. Tibco-RV was
> > really good for that as well.
> > >
> > > IMHO— plugging in protobuf, Avro, or other binary-serialization
> > flavor-of-the-month at the connection factory level has real value.
> > >
> > >> On Jun 1, 2017, at 2:23 PM, Clebert Suconic <
> clebert.suconic@gmail.com>
> > wrote:
> > >>
> > >> I honestly don't see an issue on making the write and readObject a
> > >> pluggable operation. It's a simple change on the API at
> > >> ActiveMQConnectionFactory.
> > >>
> > >> On Thu, Jun 1, 2017 at 1:09 PM, Michael André Pearce
> > >> <michael.andre.pearce@me.com> wrote:
> > >>> This is why the proposal is for a pluggable interface to declared to
> > convert from Object to byte[] and back not saying one moment Artemis owns
> > or has all the implementations.
> > >>>
> > >>> Anyhow your points are taken on board, we def need to agree on a well
> > defined and clean api, to avoid exactly that situation.
> > >>>
> > >>> Sent from my iPhone
> > >>>
> > >>>> On 1 Jun 2017, at 17:59, Timothy Bish <tabish121@gmail.com>
wrote:
> > >>>>
> > >>>>> On 06/01/2017 12:49 PM, Michael André Pearce wrote:
> > >>>>> Not at all that's the point
> > >>>>>
> > >>>>> Application code uses JMS Api. The serdes are just defined/declared
> > into the connection factory typically the only part of the app exposed to
> > any particulars about the broker implementation detail.
> > >>>>>
> > >>>>> Yes we can do converter.convert(object) in code, this is just
> manual
> > code in the app space.
> > >>>>>
> > >>>>> but like with kafka and I have to stress it's successfulness
is
> that
> > that converter is in it's api of the product. Which means a lot of
> > companies reuse a few single implementations and a good eco system occurs
> > like with schema registry (Eg we use confluents serdes we don't code out
> > own)
> > >>>>
> > >>>> Which is exactly what camel can solve and without starting down
the
> > slippery slop of packing the same data format conversions Camel can
> already
> > handle into the Artemis code base as will happen as each user wants their
> > own preferred data format.
> > >>>>
> > >>>> Similar things were tried in the ActiveMQ 5.x code and abandoned
> over
> > time so I'd like to avoid that if possible.
> > >>>>
> > >>>> Anyway, I've raised my objection.
> > >>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Sent from my iPhone
> > >>>>>
> > >>>>>>> On 1 Jun 2017, at 17:39, Timothy Bish <tabish121@gmail.com>
> wrote:
> > >>>>>>>
> > >>>>>>> On 06/01/2017 12:19 PM, Michael André Pearce wrote:
> > >>>>>>> Agreed it does as an EAI pattern or flow, But I have
to
> > code/define into Camel's dsl, it does JMS as much as our custom app code
> > would it consumes the JMS api.
> > >>>>>> So you still need to code / define the serialization then
and
> > include that in you application which means the difference between some
> > connection.setSerializationThing() vs
> > producer.send(Converter.convert(payload)); so I don't see how that is a
> > real value add to the JMS client.
> > >>>>>>
> > >>>>>>> What we propose here is just providing a clean way
to define the
> > JMS ObjectMessage internal serialisation. If Java serialisation isn't
> your
> > cup of tea. (Which for many reasons isn't for us, and I'm sure it's
> similar
> > for others)
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>>>> On 1 Jun 2017, at 16:58, Timothy Bish <tabish121@gmail.com>
> > wrote:
> > >>>>>>>>>
> > >>>>>>>>> On 06/01/2017 11:50 AM, Michael André Pearce
wrote:
> > >>>>>>>>> Really this is much more about how an ObjectMessage
serializes
> > the Object. As we have C++ clients etc that obviously won't be able to
> > understand Java serialized object.
> > >>>>>>>>>
> > >>>>>>>>> We use Avro and a schema repo for our dto transfer
over the
> > wire, it's been a real performance boost , and removed some core data
> > issues, and really like to use it over the JMS land.
> > >>>>>>>>>
> > >>>>>>>>> One can argue that you could manually code
this that you
> > serialize the data manually first and then just manually send a
> > BytesMessage.
> > >>>>>>>>>
> > >>>>>>>>> But I think taking some inspiration from other
places where a
> > serdes pattern is done has really helped (Kafka), from a corporation user
> > approach wiring some prebuilt serdes into a factory is very easy, having
> > duplicated code in many many apps leaves for issues, and implementation
> > divergence.
> > >>>>>>>>>
> > >>>>>>>>> The one downside of Kafka is it's lack of spec
api, this is one
> > big sell of artemis as it's JMS compliant. Coding against JMS api for
> Java
> > estate is a huge win, this is suggesting taking some of the good bits :).
> > >>>>>>>>>
> > >>>>>>>>> Does camel expose this as some sort of JMS
API wrapper? I
> > thought it was much more an EAI solution.
> > >>>>>>>>>
> > >>>>>>>>> Cheers
> > >>>>>>>>> Mike
> > >>>>>>>> Camel does JMS transport: http://camel.apache.org/sjms.html
> > >>>>>>>> Camel does AVRO: http://camel.apache.org/avro.html
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> Sent from my iPhone
> > >>>>>>>>>
> > >>>>>>>>>>>> On 1 Jun 2017, at 15:18, Martyn
Taylor <mtaylor@redhat.com>
> > wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Jun 1, 2017 at 2:45 PM,
Timothy Bish <
> > tabish121@gmail.com> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On 06/01/2017 09:34 AM, Martyn
Taylor wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Thu, Jun 1, 2017 at
2:32 PM, Timothy Bish <
> > tabish121@gmail.com> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 06/01/2017 08:51 AM,
Martyn Taylor wrote:
> > >>>>>>>>>>>>>> I get the use case for
using JSON/XML, particularly for
> > cross language
> > >>>>>>>>>>>>>> communication.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> One way users get around
this problem right now is just to
> > serialize
> > >>>>>>>>>>>>>> to/from XML/JSON at the
client application level and just
> > use JMS
> > >>>>>>>>>>>>>> TextMessages to send the
data. I guess the idea here to
> > remove that
> > >>>>>>>>>>>>>> complexity from the client
application and into the client
> > via these
> > >>>>>>>>>>>>>> pluggable serializer objects?
 Removing the serizliation
> > logic out of
> > >>>>>>>>>>>>>> code
> > >>>>>>>>>>>>>> and into configuration.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Providing I've understood
this properly, it seems like a
> > good idea to
> > >>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>> so +1.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> This problem has already
been solved via frameworks like
> > Apache Camel,
> > >>>>>>>>>>>>> putting such complexity into
the JMS client is solving a
> > problem that's
> > >>>>>>>>>>>>> already been solved and in
much more flexible and
> > configurable ways.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks Tim.  I am not a Camel expert
in any shape or form,
> > how much
> > >>>>>>>>>>>> additional complexity/configuration
would be required to do
> > something
> > >>>>>>>>>>>> similar with Camel?  My understanding
of the proposal here
> is
> > really just
> > >>>>>>>>>>>> to give control back to the user
in terms of how their
> > objects are
> > >>>>>>>>>>>> serialized.  I'd expect this to
be pretty light weight, just
> > allow a user
> > >>>>>>>>>>>> to configure a class to do the
serialization.
> > >>>>>>>>>>>>
> > >>>>>>>>>>> Camel offers conversions for a number
of data formats
> > >>>>>>>>>> Sure.   Though, one of the drivers (mentioned
in this thread)
> > for having
> > >>>>>>>>>> control over the de/serialization process
was for performance.
> > Converting
> > >>>>>>>>>> to another format is going to obviously
make this much worse.
> > >>>>>>>>>>
> > >>>>>>>>>>> as well as routing amongst numerous
protocols, have a look at
> > the
> > >>>>>>>>>>> supported data formats page:
> > http://camel.apache.org/data-format.html and
> > >>>>>>>>>>> the transports http://camel.apache.org/transport.html
> > >>>>>>>>>>> This doesn't seem to be doing much
more for the user than
> > moving the work
> > >>>>>>>>>>> they need to do around,
> > >>>>>>>>>> Well, it abstracts the de/serialization
process out of
> > application code.
> > >>>>>>>>>>
> > >>>>>>>>>>> they still have to implement or configure
the mechanics of
> the
> > >>>>>>>>>>> transformation of the data format to
the appropriate JMS
> > message type and
> > >>>>>>>>>>> back again.  Even if you bake in something
to the client to
> > handle some
> > >>>>>>>>>>> common formats you will quickly find
that it doesn't meet
> > everyone's needs
> > >>>>>>>>>>> and you'll end up implementing a poor
mans Camel inside a JMS
> > API
> > >>>>>>>>>>> restricted client which seems less
than ideal.
> > >>>>>>>>>> I agree reinventing the wheel (badly) is
not a good idea.  So,
> > if Camel is
> > >>>>>>>>>> able to provide us with a solution to the
problem, that
> > addresses the
> > >>>>>>>>>> issues outlined here.  Then, we should
certainly look into it.
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers.
> > >>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Jun 1, 2017 at 7:44 AM,
Michael André Pearce <
> > >>>>>>>>>>>>>> michael.andre.pearce@me.com>
wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think i might be getting
the problem, use case you want
> > to go for,
> > >>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> is to possible serialise
to JSON or XML, because they're
> > supported well
> > >>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> other languages like
c++, which won't read a java
> > serialised object,
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>> say for XML you generate
objects via an XSD which by
> > default aren't
> > >>>>>>>>>>>>>>> serialisable, so you
cannot simply add Serializable to
> the
> > object, as
> > >>>>>>>>>>>>>>> its
> > >>>>>>>>>>>>>>> generated at build.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Is this the problem
we need to solve? If so:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> To get around this
normally the tools that generate
> > objects for
> > >>>>>>>>>>>>>>> serialisation from
schema such as XSD do support a way to
> > toggle or
> > >>>>>>>>>>>>>>> change
> > >>>>>>>>>>>>>>> the generation slightly
for some common use cases.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> In case of XSD, where
using jaxb it would be to add
> > something like the
> > >>>>>>>>>>>>>>> below to jaxb global
bindings:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> <xs:annotation>
> > >>>>>>>>>>>>>>> <xs:appinfo>
> > >>>>>>>>>>>>>>> <jaxb:globalBindings
generateIsSetMethod="true">
> > >>>>>>>>>>>>>>> <xjc:serializable
uid="12343"/>
> > >>>>>>>>>>>>>>> </jaxb:globalBindings>
> > >>>>>>>>>>>>>>> </xs:appinfo>
> > >>>>>>>>>>>>>>> </xs:annotation>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> like wise if you are
generating POJO's from a jsonschema
> > using for say
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> tool jsonschema2pojo
 there is a toggle in the maven
> plugin
> > >>>>>>>>>>>>>>> serializable
> > >>>>>>>>>>>>>>> which you can switch
to true.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Obviously if you hand
crank your DTO Pojo's then it's a
> > case of simply
> > >>>>>>>>>>>>>>> add
> > >>>>>>>>>>>>>>> implement  Serializable
to the class.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Cheers
> > >>>>>>>>>>>>>>> Mike
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Sent from my iPhone
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 1 Jun 2017, at 06:57,
Michael André Pearce <
> > >>>>>>>>>>>>>>> michael.andre.pearce@me.com>
wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> we could but then it
wouldn't work via jms api. Typically
> > if using jms
> > >>>>>>>>>>>>>>>> the only custom
or specific broker object is the
> > connection factory
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> rest you code to Jms.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Sent from my iPhone
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 1 Jun 2017,
at 04:10, Clebert Suconic <
> > clebert.suconic@gmail.com>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>> On Wed, May 31,
2017 at 10:47 PM Michael André Pearce <
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> michael.andre.pearce@me.com>
wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Jms api dictates
class set in object message to be
> > serializable.
> > >>>>>>>>>>>>>>>>> We could make
an extension. It could be an extra
> message
> > this
> > >>>>>>>>>>>>>>>>> actually.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 31 May 2017,
at 22:37, Timothy Nodine <
> > timgnodine@gmail.com>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Should the
interface require the underlying class to be
> > Serializable?
> > >>>>>>>>>>>>>>>>> One use case
might be to provide serialization to
> > classes that aren't
> > >>>>>>>>>>>>>>>>>> natively
serializable.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Michael
André Pearce wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> To
help discussion,
> > >>>>>>>>>>>>>>>>>>>>
A very very basic implementation just to simulate
> the
> > idea.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > https://github.com/michaelandrepearce/activemq-artemis/tree/
> > >>>>>>>>>>>>>>>>>> CustomSerialisation
> > >>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > https://github.com/michaelandrepearce/activemq-artemis/tree/
> > >>>>>>>>>>>>>>>>>> CustomSerialisation
> > >>>>>>>>>>>>>>>> n.b. doesn’t
fully compile is just pseudo impl, nor
> > doesn’t include
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> bits as discussed
below like map/change type to a byte
> > message for
> > >>>>>>>>>>>>>>>>>> compatibility,
nor media type idea.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Cheers
> > >>>>>>>>>>>>>>>>>>>>
Mike
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
--
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Clebert
Suconic
> > >>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>> Tim Bish
> > >>>>>>>>>>>>> twitter: @tabish121
> > >>>>>>>>>>>>> blog: http://timbish.blogspot.com/
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Tim Bish
> > >>>>>>>>>>> twitter: @tabish121
> > >>>>>>>>>>> blog: http://timbish.blogspot.com/
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Tim Bish
> > >>>>>>>> twitter: @tabish121
> > >>>>>>>> blog: http://timbish.blogspot.com/
> > >>>>>>>>
> > >>>>>> --
> > >>>>>> Tim Bish
> > >>>>>> twitter: @tabish121
> > >>>>>> blog: http://timbish.blogspot.com/
> > >>>>>>
> > >>>>
> > >>>> --
> > >>>> Tim Bish
> > >>>> twitter: @tabish121
> > >>>> blog: http://timbish.blogspot.com/
> > >>>>
> > >>
> > >>
> > >>
> > >> --
> > >> Clebert Suconic
> > >
> >
> > --
> Clebert Suconic
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message