activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martyn Taylor <mtay...@redhat.com>
Subject Re: [DISCUSS] AMQP Large Messages
Date Thu, 16 Nov 2017 11:10:48 GMT
Clebert,

We need to distinguish between streamed messages and large messages.  For
the basic large message case, i.e. a single large message sent to the
broker.  I agree with what you have here.

Streamed large messages (i.e. messages that are received in chunks) allows
us to store the message without having to keep the whole thing in broker
memory.  This get's complicated with cross protocol as:

1. Not all protocols support streamed messages.
2. Cross protocol may require message conversions

Both 1. and 2. would likely mean that we'd likely need to read the whole
message into memory and I'm not sure we really want to do this?  It defeats
one of the main purposes of streamed messages.  (The others being saving
client memory and reducing the amount of data to resend on connection drop.)

For 1. I wonder whether or not we want to support this at all, or if we do,
make it configurable.
For 2. Is it possible to transform the message via a stream?  If not then
perhaps we treat this as a raw bytes message?

Cheers

On Wed, Nov 15, 2017 at 11:33 PM, Clebert Suconic <clebert.suconic@gmail.com
> wrote:

> It would be possible... look at
> https://github.com/apache/activemq-artemis/pull/1655
>
> I will keep the same current semantic. That is.. you would read the
> whole body.. and convert in memory.
> That's because I wouldn't know how to convert the body without having
> it into memory.. for things like string reading.
>
> Possible to do it without using memory.. but not very practical at
> this point... we could make this a separate task.. to convert large
> messages without requiring full memory read.
>
>
>
> I would add a note to docs saying that large messages will be read
> into memory before converting protocol barriers.
>
> On Wed, Nov 15, 2017 at 6:24 PM, Michael André Pearce
> <michael.andre.pearce@me.com> wrote:
> > Looks good for AMQP to AMQP client.
> >
> > One question is in a poly protocol supported setup. Eg senders and
> receivers one uses Core and AMQP.
> >
> > It’s not clear if this should now be possible with the new
> LargeMessageBody abstraction you discuss.
> >
> > Cheers
> > Mike
> >
> > Sent from my iPhone
> >
> >> On 16 Nov 2017, at 02:10, Clebert Suconic <clebert.suconic@gmail.com>
> wrote:
> >>
> >> I am currently implementing Large Message support for AMQP Messages.
> >>
> >> There's currently no support for Large Messages in AMQP, although I
> >> have recently implemented a dum copy and convert, but if you send a
> >> large message, say with 1 G, the whole body would be read into memory
> >> before a conversion to AMQP.. what won't fit in most cases of course.
> >> (even so I can always make a case where it wouldn't fit).
> >>
> >> LargeMessages are always converted as Core currently. With this change
> >> they will be kept as AMQP all the way through, adding full support for
> >> large messages, which is nice!
> >>
> >>
> >> Anyway... the design I got so far is the following:
> >>
> >>
> >> - LargeServerMessageImpl will delegate its implementation to another
> >> class called LargeMessageBody
> >>
> >> - LargeMessageBody would then be used by AMQPMessage when incomplete
> >> messages are sent through the wire.
> >>
> >> - When sending messages on the server, I will use sender.send multiple
> >> times until the whole buffer is read. I'm not 100% sure qpid.Proton
> >> will work fine with that approach. I will start with that, and deal
> >> with further bugs if they come along.
> >>
> >>
> >> - When receiving messages on the server. Messages are marked as
> >> incomplete. So, I will create the file using the LargeMessageBody. on
> >> that case I will just use the current recv method we already have
> >> available on Proton multiple times. This seems to work fine already.
>
>
>
> --
> Clebert Suconic
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message