Return-Path: Delivered-To: apmail-qpid-users-archive@www.apache.org Received: (qmail 95759 invoked from network); 24 Sep 2010 07:41:12 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 24 Sep 2010 07:41:12 -0000 Received: (qmail 19617 invoked by uid 500); 24 Sep 2010 07:41:12 -0000 Delivered-To: apmail-qpid-users-archive@qpid.apache.org Received: (qmail 19310 invoked by uid 500); 24 Sep 2010 07:41:09 -0000 Mailing-List: contact users-help@qpid.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@qpid.apache.org Delivered-To: mailing list users@qpid.apache.org Received: (qmail 19302 invoked by uid 99); 24 Sep 2010 07:41:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Sep 2010 07:41:07 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [213.41.18.43] (HELO vm-abcsmtp.abc-arbitrage.com) (213.41.18.43) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Sep 2010 07:40:59 +0000 Received: from vm-abcexch2003.abc-arbitrage.fr (unverified [192.168.191.14]) by vm-abcsmtp.abc-arbitrage.com (Clearswift SMTPRS 5.4.0) with ESMTP id ; Fri, 24 Sep 2010 09:40:28 +0200 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message Subject: RE: Qpid C++ broker memory leak? MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Date: Fri, 24 Sep 2010 09:40:37 +0200 Message-ID: <3BC56299C1B27B4FA2FF39164EE02087BF1F31@vm-abcexch2003.abc-arbitrage.fr> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Qpid C++ broker memory leak? Thread-Index: ActbtVYy8Ienq43eRRmT/Mh88DhsFAAAkgYg References: <3BC56299C1B27B4FA2FF39164EE02087BF1F15@vm-abcexch2003.abc-arbitrage.fr> <4C9B8C2B.8090600@redhat.com> From: "Julien Lavigne du Cadet" To: "Alan Conway" , X-Virus-Checked: Checked by ClamAV on apache.org Here's the output I have for a few run. The memory consumption pattern seem= s quite different from yours but it doesn't increase linearly abcqpid@abcQpid:~/qpid-0.5/cpp/src/tests$ while ps up `pgrep qpidd`; do ./p= erftest -s --username guest --password guest123 --size 64 --count 1000000 ;= done USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 0.0 0.0 244104 6020 ? Ssl 09:08 0:00 qpidd -d 123551 111219 223287 13.6284 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 31.2 2.1 487524 174900 ? Ssl 09:08 0:16 qpidd -d 122093 101291 203355 12.4118 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 53.6 3.5 562036 287208 ? Ssl 09:08 0:33 qpidd -d 127229 111748 223542 13.6439 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 69.5 3.4 562036 285792 ? Ssl 09:08 0:50 qpidd -d 128199 105704 212375 12.9623 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 82.4 5.3 758748 441024 ? Rsl 09:08 1:06 qpidd -d 123956 109960 219983 13.4267 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 92.7 4.1 627676 340588 ? Ssl 09:08 1:23 qpidd -d 129984 112129 224308 13.6907 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 100 5.0 693212 414732 ? Ssl 09:08 1:39 qpidd -d 107284 106511 213869 13.0535 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 106 4.1 693212 340740 ? Ssl 09:08 1:56 qpidd -d 101739 100153 200296 12.2251 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 112 4.3 693212 354076 ? Ssl 09:08 2:13 qpidd -d 128088 108798 217643 13.2839 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 117 5.5 776172 455168 ? Ssl 09:08 2:30 qpidd -d 129048 112187 225225 13.7466 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 121 4.7 758748 391288 ? Ssl 09:08 2:46 qpidd -d 128988 112805 225689 13.7749 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 124 5.2 758880 430636 ? Ssl 09:08 3:02 qpidd -d 131103 112403 225844 13.7845 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 128 5.9 824416 487504 ? Ssl 09:08 3:18 qpidd -d 129258 113177 226364 13.8162 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 130 4.9 758880 403288 ? Ssl 09:08 3:34 qpidd -d 120588 109785 219620 13.4046 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 20167 133 5.3 758880 439848 ? Ssl 09:08 3:50 qpidd -d -----Message d'origine----- De=A0: Alan Conway [mailto:aconway@redhat.com]=20 Envoy=E9=A0: jeudi 23 septembre 2010 19:20 =C0=A0: users@qpid.apache.org Cc=A0: Julien Lavigne du Cadet Objet=A0: Re: Qpid C++ broker memory leak? On 09/23/2010 08:43 AM, Julien Lavigne du Cadet wrote: > Hi, > > > > I noticed something that looks like a memory leak with both qpid 0.5 and > 0.6 on ubuntu (hardy) and I would like your feedback. > > > > I have a test application that publishes small non persistent messages > as fast as possible. > > Another application connects to the broker and creates a queue with auto > delete set to true. I don't read the message in order to fill the queue > with hundreds of thousands of messages. After a few seconds, the broker > consumes 2GB of memory and I kill both clients. I can see in qmf that > the queue is deleted correctly, however qpid will release only a very > small portion of the memory (for 2GB, it will be around 200MB). Waiting > doesn't change a thing. If I start both clients again, qpid memory > consumption will instantly increase. However, if I decide to read the > message (so that the message depth stay close to 0), the memory > consumption will stay low whatever happens. > > > > I can reproduce this behaviour with both qpid 0.5 and qpid 0.6, with > durable and non-durable queues on 2 different servers running ubuntu. > However, on a windows server qpid releases the memory instantly when I > kill both clients. > > > > I also see some similar behaviour when running perftests: > > > > After running "./perftest --size 64 --count 1000000", qpid consumes > 168Mb of memory (463MB virtual) > > After running "./perftest --size 64 --count 1000000 -qt 10", qpid > consumes 832Mb of memory (1637MB virtual) > I don't see the same results as you, if I run perftest in a loop qpidd rema= ins=20 stable at about 400Mb virtual memory (this is on latest trunk). If you run= =20 perftest repeatedly do you see continuous growth over time? Note that freed= =20 memory is not immediately returned to the OS, so some growth in qpidd size = is to=20 be expected even if all messages have been removed from queues. However if = you=20 see continuous growth over time then there probably is a leak to be investi= gated. $ while ps up `pgrep qpidd`; do qpid-perftest -s --size 64 --count 1000000 = ; done USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 37.2 0.3 401544 52256 ? Ssl 12:59 1:26 qpidd --au= th=3Dno -d 43269.3 42904.7 86263.5 5.26511 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 50.2 0.2 398156 37440 ? Ssl 12:59 2:07 qpidd --au= th=3Dno -d 40394.2 40248.7 80641.5 4.92196 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 61.8 0.2 397716 37856 ? Ssl 12:59 2:52 qpidd --au= th=3Dno -d 41445.9 41308.1 82771.1 5.05194 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 70.9 0.2 395784 37304 ? Ssl 12:59 3:35 qpidd --au= th=3Dno -d 40487.1 40336.8 80821.4 4.93294 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 79.1 0.2 395784 37304 ? Ssl 12:59 4:20 qpidd --au= th=3Dno -d 40717.5 40552.7 81255.8 4.95946 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 85.9 0.2 395964 35260 ? Ssl 12:59 5:04 qpidd --au= th=3Dno -d 40110.3 40026 80195.4 4.89474 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 92.1 0.2 396868 37332 ? Ssl 12:59 5:49 qpidd --au= th=3Dno -d 41388.6 41231.4 82746.6 5.05045 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 97.5 0.2 395784 37840 ? Ssl 12:59 6:32 qpidd --au= th=3Dno -d 42590 42440 85032.8 5.18999 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND aconway 10902 101 0.2 396824 38880 ? Ssl 12:59 7:14 qpidd --au= th=3Dno -d P Please consider your environmental responsibility before printing this e= mail ***************************************************************************= ****** Ce message peut contenir des informations confidentielles.=20 Les idees et opinions presentees dans ce message=20 sont celles de son auteur, et ne representent pas necessairement celles du groupe ABC arbitrage. Au cas ou il ne vous serait pas destine, merci d'en aviser l'expediteur immediatement et de le supprimer. This message may contain confidential information.=20 Any views or opinions presented are solely those of its author=20 and do not necessarily represent those of ABC arbitrage.=20 If you are not the intended recipient, please notify the sender immediately and delete it. ***************************************************************************= ****** --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscribe@qpid.apache.org