activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael André Pearce <michael.andre.pea...@me.com>
Subject Re: [DISCUSS] Custom Object Serialisation Support
Date Thu, 01 Jun 2017 22:51:30 GMT
I have been looking at an alternative approach (i think its a little less optimal), but in
thoughts it might be more acceptable for those not wishing this into the core jms client.
It means the custom object serialisation is a optional add-on rather than coded into the core
jms client, similar idea to the way spring wraps connection factories for caching connection,
and others.

https://github.com/michaelandrepearce/activemq-artemis/tree/SerdesExtension/artemis-jms-client-extensions
<https://github.com/michaelandrepearce/activemq-artemis/tree/SerdesExtension/artemis-jms-client-extensions>

It essentially is wrapping at the JMS api level, so actually would/should work for any JMS
Client not just artemis, e.g. could wrap QPID client.

Thoughts on this approach rather than native integration to the clients?



> On 1 Jun 2017, at 21:54, Matt Pavlovich <mattrpav@gmail.com> wrote:
> 
> I understand the usefulness of adding this to the Client Connection Factory vs kicking
all the way up to an integration stack (like Camel). In my eyes, its about solving for cross-platform
object serialization using byte payloads instead of the problems associated with non-portable/cross
platform ObjectMessage and MapMessage JMS message types. Tibco-RV was really good for that
as well.
> 
> IMHO— plugging in protobuf, Avro, or other binary-serialization flavor-of-the-month
at the connection factory level has real value.
> 
>> On Jun 1, 2017, at 2:23 PM, Clebert Suconic <clebert.suconic@gmail.com> wrote:
>> 
>> I honestly don't see an issue on making the write and readObject a
>> pluggable operation. It's a simple change on the API at
>> ActiveMQConnectionFactory.
>> 
>> On Thu, Jun 1, 2017 at 1:09 PM, Michael André Pearce
>> <michael.andre.pearce@me.com> wrote:
>>> This is why the proposal is for a pluggable interface to declared to convert
from Object to byte[] and back not saying one moment Artemis owns or has all the implementations.
>>> 
>>> Anyhow your points are taken on board, we def need to agree on a well defined
and clean api, to avoid exactly that situation.
>>> 
>>> Sent from my iPhone
>>> 
>>>> On 1 Jun 2017, at 17:59, Timothy Bish <tabish121@gmail.com> wrote:
>>>> 
>>>>> On 06/01/2017 12:49 PM, Michael André Pearce wrote:
>>>>> Not at all that's the point
>>>>> 
>>>>> Application code uses JMS Api. The serdes are just defined/declared into
the connection factory typically the only part of the app exposed to any particulars about
the broker implementation detail.
>>>>> 
>>>>> Yes we can do converter.convert(object) in code, this is just manual
code in the app space.
>>>>> 
>>>>> but like with kafka and I have to stress it's successfulness is that
that converter is in it's api of the product. Which means a lot of companies reuse a few single
implementations and a good eco system occurs like with schema registry (Eg we use confluents
serdes we don't code out own)
>>>> 
>>>> Which is exactly what camel can solve and without starting down the slippery
slop of packing the same data format conversions Camel can already handle into the Artemis
code base as will happen as each user wants their own preferred data format.
>>>> 
>>>> Similar things were tried in the ActiveMQ 5.x code and abandoned over time
so I'd like to avoid that if possible.
>>>> 
>>>> Anyway, I've raised my objection.
>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> Sent from my iPhone
>>>>> 
>>>>>>> On 1 Jun 2017, at 17:39, Timothy Bish <tabish121@gmail.com>
wrote:
>>>>>>> 
>>>>>>> On 06/01/2017 12:19 PM, Michael André Pearce wrote:
>>>>>>> Agreed it does as an EAI pattern or flow, But I have to code/define
into Camel's dsl, it does JMS as much as our custom app code would it consumes the JMS api.
>>>>>> So you still need to code / define the serialization then and include
that in you application which means the difference between some connection.setSerializationThing()
vs producer.send(Converter.convert(payload)); so I don't see how that is a real value add
to the JMS client.
>>>>>> 
>>>>>>> What we propose here is just providing a clean way to define
the JMS ObjectMessage internal serialisation. If Java serialisation isn't your cup of tea.
(Which for many reasons isn't for us, and I'm sure it's similar for others)
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>>>> On 1 Jun 2017, at 16:58, Timothy Bish <tabish121@gmail.com>
wrote:
>>>>>>>>> 
>>>>>>>>> On 06/01/2017 11:50 AM, Michael André Pearce wrote:
>>>>>>>>> Really this is much more about how an ObjectMessage serializes
the Object. As we have C++ clients etc that obviously won't be able to understand Java serialized
object.
>>>>>>>>> 
>>>>>>>>> We use Avro and a schema repo for our dto transfer over
the wire, it's been a real performance boost , and removed some core data issues, and really
like to use it over the JMS land.
>>>>>>>>> 
>>>>>>>>> One can argue that you could manually code this that
you serialize the data manually first and then just manually send a BytesMessage.
>>>>>>>>> 
>>>>>>>>> But I think taking some inspiration from other places
where a serdes pattern is done has really helped (Kafka), from a corporation user approach
wiring some prebuilt serdes into a factory is very easy, having duplicated code in many many
apps leaves for issues, and implementation divergence.
>>>>>>>>> 
>>>>>>>>> The one downside of Kafka is it's lack of spec api, this
is one big sell of artemis as it's JMS compliant. Coding against JMS api for Java estate is
a huge win, this is suggesting taking some of the good bits :).
>>>>>>>>> 
>>>>>>>>> Does camel expose this as some sort of JMS API wrapper?
I thought it was much more an EAI solution.
>>>>>>>>> 
>>>>>>>>> Cheers
>>>>>>>>> Mike
>>>>>>>> Camel does JMS transport: http://camel.apache.org/sjms.html
>>>>>>>> Camel does AVRO: http://camel.apache.org/avro.html
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Sent from my iPhone
>>>>>>>>> 
>>>>>>>>>>>> On 1 Jun 2017, at 15:18, Martyn Taylor <mtaylor@redhat.com>
wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Jun 1, 2017 at 2:45 PM, Timothy Bish
<tabish121@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 06/01/2017 09:34 AM, Martyn Taylor
wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Jun 1, 2017 at 2:32 PM, Timothy
Bish <tabish121@gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 06/01/2017 08:51 AM, Martyn Taylor
wrote:
>>>>>>>>>>>>>> I get the use case for using JSON/XML,
particularly for cross language
>>>>>>>>>>>>>> communication.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> One way users get around this problem
right now is just to serialize
>>>>>>>>>>>>>> to/from XML/JSON at the client application
level and just use JMS
>>>>>>>>>>>>>> TextMessages to send the data. I
guess the idea here to remove that
>>>>>>>>>>>>>> complexity from the client application
and into the client via these
>>>>>>>>>>>>>> pluggable serializer objects?  Removing
the serizliation logic out of
>>>>>>>>>>>>>> code
>>>>>>>>>>>>>> and into configuration.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Providing I've understood this properly,
it seems like a good idea to
>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>> so +1.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This problem has already been solved
via frameworks like Apache Camel,
>>>>>>>>>>>>> putting such complexity into the JMS
client is solving a problem that's
>>>>>>>>>>>>> already been solved and in much more
flexible and configurable ways.
>>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks Tim.  I am not a Camel expert in any
shape or form, how much
>>>>>>>>>>>> additional complexity/configuration would
be required to do something
>>>>>>>>>>>> similar with Camel?  My understanding of
the proposal here is really just
>>>>>>>>>>>> to give control back to the user in terms
of how their objects are
>>>>>>>>>>>> serialized.  I'd expect this to be pretty
light weight, just allow a user
>>>>>>>>>>>> to configure a class to do the serialization.
>>>>>>>>>>>> 
>>>>>>>>>>> Camel offers conversions for a number of data
formats
>>>>>>>>>> Sure.   Though, one of the drivers (mentioned in
this thread) for having
>>>>>>>>>> control over the de/serialization process was for
performance.  Converting
>>>>>>>>>> to another format is going to obviously make this
much worse.
>>>>>>>>>> 
>>>>>>>>>>> as well as routing amongst numerous protocols,
have a look at the
>>>>>>>>>>> supported data formats page: http://camel.apache.org/data-format.html
and
>>>>>>>>>>> the transports http://camel.apache.org/transport.html
>>>>>>>>>>> This doesn't seem to be doing much more for the
user than moving the work
>>>>>>>>>>> they need to do around,
>>>>>>>>>> Well, it abstracts the de/serialization process out
of application code.
>>>>>>>>>> 
>>>>>>>>>>> they still have to implement or configure the
mechanics of the
>>>>>>>>>>> transformation of the data format to the appropriate
JMS message type and
>>>>>>>>>>> back again.  Even if you bake in something to
the client to handle some
>>>>>>>>>>> common formats you will quickly find that it
doesn't meet everyone's needs
>>>>>>>>>>> and you'll end up implementing a poor mans Camel
inside a JMS API
>>>>>>>>>>> restricted client which seems less than ideal.
>>>>>>>>>> I agree reinventing the wheel (badly) is not a good
idea.  So, if Camel is
>>>>>>>>>> able to provide us with a solution to the problem,
that addresses the
>>>>>>>>>> issues outlined here.  Then, we should certainly
look into it.
>>>>>>>>>> 
>>>>>>>>>> Cheers.
>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Jun 1, 2017 at 7:44 AM, Michael André
Pearce <
>>>>>>>>>>>>>> michael.andre.pearce@me.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I think i might be getting the problem,
use case you want to go for,
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> is to possible serialise to JSON
or XML, because they're supported well
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> other languages like c++, which
won't read a java serialised object,
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> say for XML you generate objects
via an XSD which by default aren't
>>>>>>>>>>>>>>> serialisable, so you cannot simply
add Serializable to the object, as
>>>>>>>>>>>>>>> its
>>>>>>>>>>>>>>> generated at build.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Is this the problem we need to
solve? If so:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> To get around this normally the
tools that generate objects for
>>>>>>>>>>>>>>> serialisation from schema such
as XSD do support a way to toggle or
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>> the generation slightly for some
common use cases.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In case of XSD, where using jaxb
it would be to add something like the
>>>>>>>>>>>>>>> below to jaxb global bindings:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> <xs:annotation>
>>>>>>>>>>>>>>> <xs:appinfo>
>>>>>>>>>>>>>>> <jaxb:globalBindings generateIsSetMethod="true">
>>>>>>>>>>>>>>> <xjc:serializable uid="12343"/>
>>>>>>>>>>>>>>> </jaxb:globalBindings>
>>>>>>>>>>>>>>> </xs:appinfo>
>>>>>>>>>>>>>>> </xs:annotation>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> like wise if you are generating
POJO's from a jsonschema using for say
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> tool jsonschema2pojo  there is
a toggle in the maven plugin
>>>>>>>>>>>>>>> serializable
>>>>>>>>>>>>>>> which you can switch to true.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Obviously if you hand crank your
DTO Pojo's then it's a case of simply
>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>>> implement  Serializable to the
class.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 1 Jun 2017, at 06:57, Michael
André Pearce <
>>>>>>>>>>>>>>> michael.andre.pearce@me.com>
wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> we could but then it wouldn't
work via jms api. Typically if using jms
>>>>>>>>>>>>>>>> the only custom or specific
broker object is the connection factory
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> rest you code to Jms.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 1 Jun 2017, at 04:10,
Clebert Suconic <clebert.suconic@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> On Wed, May 31, 2017 at 10:47
PM Michael André Pearce <
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> michael.andre.pearce@me.com>
wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jms api dictates class
set in object message to be serializable.
>>>>>>>>>>>>>>>>> We could make an extension.
It could be an extra message this
>>>>>>>>>>>>>>>>> actually.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 31 May 2017, at 22:37,
Timothy Nodine <timgnodine@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Should the interface
require the underlying class to be Serializable?
>>>>>>>>>>>>>>>>> One use case might be
to provide serialization to classes that aren't
>>>>>>>>>>>>>>>>>> natively serializable.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Michael André
Pearce wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> To help discussion,
>>>>>>>>>>>>>>>>>>>> A very very
basic implementation just to simulate the idea.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> https://github.com/michaelandrepearce/activemq-artemis/tree/
>>>>>>>>>>>>>>>>>> CustomSerialisation
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> https://github.com/michaelandrepearce/activemq-artemis/tree/
>>>>>>>>>>>>>>>>>> CustomSerialisation
>>>>>>>>>>>>>>>> n.b. doesn’t fully compile
is just pseudo impl, nor doesn’t include
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> bits as discussed below
like map/change type to a byte message for
>>>>>>>>>>>>>>>>>> compatibility, nor
media type idea.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Clebert Suconic
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>> Tim Bish
>>>>>>>>>>>>> twitter: @tabish121
>>>>>>>>>>>>> blog: http://timbish.blogspot.com/
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Tim Bish
>>>>>>>>>>> twitter: @tabish121
>>>>>>>>>>> blog: http://timbish.blogspot.com/
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> --
>>>>>>>> Tim Bish
>>>>>>>> twitter: @tabish121
>>>>>>>> blog: http://timbish.blogspot.com/
>>>>>>>> 
>>>>>> --
>>>>>> Tim Bish
>>>>>> twitter: @tabish121
>>>>>> blog: http://timbish.blogspot.com/
>>>>>> 
>>>> 
>>>> --
>>>> Tim Bish
>>>> twitter: @tabish121
>>>> blog: http://timbish.blogspot.com/
>>>> 
>> 
>> 
>> 
>> -- 
>> Clebert Suconic
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message