Mailing-List: contact users-help@qpid.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@qpid.apache.org
Received-SPF: pass (athena.apache.org: domain of jarmstrong@avvasi.com
 designates 216.16.254.35 as permitted sender)
From: Jeff Armstrong <jarmstrong@avvasi.com>
To: "users@qpid.apache.org" <users@qpid.apache.org>
Date: Thu, 8 Sep 2011 16:44:00 -0400
Subject: RE: qpidd using approx 10x memory
Thread-Topic: qpidd using approx 10x memory
Thread-Index: AcxuZrScj2qVNtciTuuZEBMyBPEXyQAAVTWT
Message-ID: <02AF59B4F3B6F242A5F08CCACBBF9BD40E90C36BCC@air.MATTER.LOCAL>
References: <02AF59B4F3B6F242A5F08CCACBBF9BD40E90C36BBC@air.MATTER.LOCAL>
 <02AF59B4F3B6F242A5F08CCACBBF9BD40E90C36BC8@air.MATTER.LOCAL>,<4E68EC8C.80606@redhat.com>
 <02AF59B4F3B6F242A5F08CCACBBF9BD40E90C36BC9@air.MATTER.LOCAL>,<4E6926D5.5010306@redhat.com>
In-Reply-To: <4E6926D5.5010306@redhat.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Using gcc version 4.4.3.
I couldn't find anything about thread pooling for Ubuntu. Are you sure that=
 this is done at the OS level and not at the application level? Also, could=
 this cause memory bloat relative to the number of messages if it is thread=
s that are pooled? This seems unlikely to me.

Jeff

________________________________________
From: Carl Trieloff [cctrieloff@redhat.com]
Sent: Thursday, September 08, 2011 4:34 PM
To: users@qpid.apache.org
Subject: Re: qpidd using approx 10x memory

what is the gcc version? or do you know if memory thread pooling is
enabled in 10.04

turning that off should resolve the memory bloat. On Fedora / RHEL it is
an env var, should be able to google it for Ubuntu

Carl.


On 09/08/2011 01:30 PM, Jeff Armstrong wrote:
> Ubuntu 10.04
> ________________________________________
> From: Carl Trieloff [cctrieloff@redhat.com]
> Sent: Thursday, September 08, 2011 12:25 PM
> To: users@qpid.apache.org
> Subject: Re: qpidd using approx 10x memory
>
> What OS version?
>
> Carl.
>
>
> On 09/08/2011 11:53 AM, Jeff Armstrong wrote:
>> Does anyone have thoughts on this? Should this be submitted in the issue=
 tracker?
>> ________________________________________
>> From: Jeff Armstrong [jarmstrong@avvasi.com]
>> Sent: Friday, August 26, 2011 3:08 PM
>> To: users@qpid.apache.org
>> Subject: qpidd using approx 10x memory
>>
>> When filling up some queues, I noticed that the qpidd process was using =
about 10x the memory than was actually being used by the queues. To do a si=
mple test on this I wrote a simple client that sends a bunch of messages th=
at get stored in the queue, then compared the total byte depth with the mem=
ory used by qpidd.
>>
>> After sending 1 million messages, the byte count (using qpid-stat -q) sh=
owed 105MB, but checking the memory usage I saw 1.4GB (checked with htop an=
d pmap). The messages were approximately 100 bytes large, so the byte count=
 from qpid-stat makes sense. What doesn't make sense is that the total memo=
ry usage is over 10x as big. I checked the memory usage of qpidd before con=
necting with the client to send the messages and it was around 200MB, so th=
ere is still a discrepancy of about 1.1GB. These results also match with th=
e original problem with an application I'm working on where qpidd used up a=
ll my memory (16GB) even though the queue sizes added up to about 1GB.
>>
>> I'm using version 0.10. Is this a known issue? I couldn't find it in the=
 issue tracker. Any workarounds?
>>
>>
>> Here is how I created the exchange/queues:
>> $ ./qpid-config add exchange direct qvue
>> $ ./qpid-config add queue testQueue
>> $ ./qpid-config bind qvue testQueue testBind
>>
>>
>> Here is the client code I used:
>> #include <iostream>
>> #include <qpid/client/Message.h>
>> #include <qpid/client/Connection.h>
>> #include <qpid/client/Session.h>
>>
>> using namespace std;
>> using namespace qpid::client;
>>
>> int main(int argc, char** argv) {
>>     Connection connection;
>>     connection.open("127.0.0.1");
>>     Session session =3D connection.newSession();
>>
>>     int count =3D 0;
>>     while(true) {
>>         Message msg;
>>         msg.setData("Hello World! 0123456789 abcdefghijklmnopqrstuvwxyz =
weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
>>         msg.getDeliveryProperties().setRoutingKey("testBind");
>>         session.messageTransfer(qpid::client::arg::content=3Dmsg, qpid::=
client::arg::destination=3D"testQueue");
>>         count++;
>>         if (count % 1000 =3D=3D 0) {
>>             cout << count << " messages sent" << endl;
>>         }
>>     }
>>     return 0;
>> }
>>
>>
>> Thanks,
>> Jeff
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project:      http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project:      http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org