Mailing-List: contact users-help@qpid.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@qpid.apache.org
Received-SPF: pass (athena.apache.org: domain of
 prvs=1234a63865=rhs@alum.mit.edu designates 18.7.68.18 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAM4xzrXOD9j8br_QnRp=imur_zdeRx=RgUNYem6p6Y5=7XpBoA@mail.gmail.com>
References: 
 <CAM4xzrXOD9j8br_QnRp=imur_zdeRx=RgUNYem6p6Y5=7XpBoA@mail.gmail.com>
Date: Fri, 6 Jun 2014 10:00:53 -0400
Message-ID: 
 <CACL1BnAV9HG6fTrekYtSy+edme=8ssewQfgUBYp+ssT1pxy3ow@mail.gmail.com>
Subject: Re: High memory footprint of proton-j in ActiveMQ
From: Rafael Schloming <rhs@alum.mit.edu>
To: "users@qpid.apache.org" <users@qpid.apache.org>
Content-Type: multipart/alternative; boundary=047d7b86d55c2c3d7804fb2b4a00

--047d7b86d55c2c3d7804fb2b4a00
Content-Type: text/plain; charset=UTF-8

That sounds like a worthy improvement to me. We could possibly put in a
threshold to let people choose their own tradeoff, e.g. below  X buffer
size we use one strategy, and above it we use another. Either way, I'd go
ahead and file a JIRA with a pointer to your commit.

--Rafael


On Thu, Jun 5, 2014 at 4:48 PM, Marcel Meulemans <
m.meulemans@tkhinnovations.com> wrote:

> I have been experiencing high (resident) memory usage when using ActiveMQ
> with large amounts of AMQP connections.
>
> After some investigation I have traced this memory usage back to proton-j.
> The cause is that by default ActiveMQ defines a maxFrameSize of 1MB for the
> AmqpWireFormat and in proton-j there are two classes (TrasportOutputAdaptor
> and FrameParser) that hold a ByteBuffer of maxFrameSize throughout their
> lifetime. In the case of the 1MB frame size that means that each transport
> has a fairly high memory footprint (2MB) considering that this memory is
> hardly ever used, i.e. only when there is a frame to transfer in/out and in
> most cases the frame will not be maxFrameSize. Because there is a Transport
> for every connection the memory usage can become quite high when dealing
> with many connections with large maxFrameSize, e.g. 2GB for only 1000
> connections (I was aiming for 10000 connections so you see my problem).
>
> Generally the buffer allocation may not be a problem but in my case they
> are so I decided to modify proton so that it does not require these buffers
> to exist at all times. The change I made are a classical tradeoff between
> memory usage and cpu cycles. The memory usage has been reduced and the cpu
> usage increased (more frequent allocations). The buffers are only allocated
> when needed and are released as soon as they hold no more data. The changes
> (which pass all tests and have the desired results) can be found here:
> https://github.com/marcelmeulemans/qpid-proton
>
> Because this may not be considered a bug or even an improvement I am
> sharing it here so you guys can decide is this is in anyway useful.
>
> --
> Marcel
>

--047d7b86d55c2c3d7804fb2b4a00--