qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcel Meulemans <m.meulem...@tkhinnovations.com>
Subject Re: High memory footprint of proton-j in ActiveMQ
Date Fri, 06 Jun 2014 14:44:08 GMT
Maintaining both strategies sounds like a good idea (and a fun exercise so
I might give it a go over the weekend :p).

Jira item created: https://issues.apache.org/jira/browse/PROTON-597

-- 
Marcel


On Fri, Jun 6, 2014 at 4:00 PM, Rafael Schloming <rhs@alum.mit.edu> wrote:

> That sounds like a worthy improvement to me. We could possibly put in a
> threshold to let people choose their own tradeoff, e.g. below  X buffer
> size we use one strategy, and above it we use another. Either way, I'd go
> ahead and file a JIRA with a pointer to your commit.
>
> --Rafael
>
>
> On Thu, Jun 5, 2014 at 4:48 PM, Marcel Meulemans <
> m.meulemans@tkhinnovations.com> wrote:
>
> > I have been experiencing high (resident) memory usage when using ActiveMQ
> > with large amounts of AMQP connections.
> >
> > After some investigation I have traced this memory usage back to
> proton-j.
> > The cause is that by default ActiveMQ defines a maxFrameSize of 1MB for
> the
> > AmqpWireFormat and in proton-j there are two classes
> (TrasportOutputAdaptor
> > and FrameParser) that hold a ByteBuffer of maxFrameSize throughout their
> > lifetime. In the case of the 1MB frame size that means that each
> transport
> > has a fairly high memory footprint (2MB) considering that this memory is
> > hardly ever used, i.e. only when there is a frame to transfer in/out and
> in
> > most cases the frame will not be maxFrameSize. Because there is a
> Transport
> > for every connection the memory usage can become quite high when dealing
> > with many connections with large maxFrameSize, e.g. 2GB for only 1000
> > connections (I was aiming for 10000 connections so you see my problem).
> >
> > Generally the buffer allocation may not be a problem but in my case they
> > are so I decided to modify proton so that it does not require these
> buffers
> > to exist at all times. The change I made are a classical tradeoff between
> > memory usage and cpu cycles. The memory usage has been reduced and the
> cpu
> > usage increased (more frequent allocations). The buffers are only
> allocated
> > when needed and are released as soon as they hold no more data. The
> changes
> > (which pass all tests and have the desired results) can be found here:
> > https://github.com/marcelmeulemans/qpid-proton
> >
> > Because this may not be considered a bug or even an improvement I am
> > sharing it here so you guys can decide is this is in anyway useful.
> >
> > --
> > Marcel
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message