qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcel Meulemans <m.meulem...@tkhinnovations.com>
Subject High memory footprint of proton-j in ActiveMQ
Date Thu, 05 Jun 2014 20:48:08 GMT
I have been experiencing high (resident) memory usage when using ActiveMQ
with large amounts of AMQP connections.

After some investigation I have traced this memory usage back to proton-j.
The cause is that by default ActiveMQ defines a maxFrameSize of 1MB for the
AmqpWireFormat and in proton-j there are two classes (TrasportOutputAdaptor
and FrameParser) that hold a ByteBuffer of maxFrameSize throughout their
lifetime. In the case of the 1MB frame size that means that each transport
has a fairly high memory footprint (2MB) considering that this memory is
hardly ever used, i.e. only when there is a frame to transfer in/out and in
most cases the frame will not be maxFrameSize. Because there is a Transport
for every connection the memory usage can become quite high when dealing
with many connections with large maxFrameSize, e.g. 2GB for only 1000
connections (I was aiming for 10000 connections so you see my problem).

Generally the buffer allocation may not be a problem but in my case they
are so I decided to modify proton so that it does not require these buffers
to exist at all times. The change I made are a classical tradeoff between
memory usage and cpu cycles. The memory usage has been reduced and the cpu
usage increased (more frequent allocations). The buffers are only allocated
when needed and are released as soon as they hold no more data. The changes
(which pass all tests and have the desired results) can be found here:
https://github.com/marcelmeulemans/qpid-proton

Because this may not be considered a bug or even an improvement I am
sharing it here so you guys can decide is this is in anyway useful.

--
Marcel

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message