qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Schloming <...@alum.mit.edu>
Subject Re: High memory footprint of proton-j in ActiveMQ
Date Fri, 06 Jun 2014 14:00:53 GMT
That sounds like a worthy improvement to me. We could possibly put in a
threshold to let people choose their own tradeoff, e.g. below  X buffer
size we use one strategy, and above it we use another. Either way, I'd go
ahead and file a JIRA with a pointer to your commit.

--Rafael


On Thu, Jun 5, 2014 at 4:48 PM, Marcel Meulemans <
m.meulemans@tkhinnovations.com> wrote:

> I have been experiencing high (resident) memory usage when using ActiveMQ
> with large amounts of AMQP connections.
>
> After some investigation I have traced this memory usage back to proton-j.
> The cause is that by default ActiveMQ defines a maxFrameSize of 1MB for the
> AmqpWireFormat and in proton-j there are two classes (TrasportOutputAdaptor
> and FrameParser) that hold a ByteBuffer of maxFrameSize throughout their
> lifetime. In the case of the 1MB frame size that means that each transport
> has a fairly high memory footprint (2MB) considering that this memory is
> hardly ever used, i.e. only when there is a frame to transfer in/out and in
> most cases the frame will not be maxFrameSize. Because there is a Transport
> for every connection the memory usage can become quite high when dealing
> with many connections with large maxFrameSize, e.g. 2GB for only 1000
> connections (I was aiming for 10000 connections so you see my problem).
>
> Generally the buffer allocation may not be a problem but in my case they
> are so I decided to modify proton so that it does not require these buffers
> to exist at all times. The change I made are a classical tradeoff between
> memory usage and cpu cycles. The memory usage has been reduced and the cpu
> usage increased (more frequent allocations). The buffers are only allocated
> when needed and are released as soon as they hold no more data. The changes
> (which pass all tests and have the desired results) can be found here:
> https://github.com/marcelmeulemans/qpid-proton
>
> Because this may not be considered a bug or even an improvement I am
> sharing it here so you guys can decide is this is in anyway useful.
>
> --
> Marcel
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message