Mailing-List: contact etch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: etch-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of ThomasMarsh@gamestop.com
 designates 12.25.107.29 as permitted sender)
From: Thomas Marsh <ThomasMarsh@gamestop.com>
To: "etch-dev@incubator.apache.org" <etch-dev@incubator.apache.org>
Subject: RE: Etch/C Memory Consumption
Thread-Topic: Etch/C Memory Consumption
Thread-Index: AcxxivBi5BxEJeFwTuaXOAvn2ve90gAqoZ8AAAESxwAABdE7kAAYXA3gAEQlqoA=
Date: Thu, 15 Sep 2011 19:41:16 +0000
Message-ID: <9711F93A3E3A3C458276A476FFA91AECC22DF3@GV1HQPEX03.babgsetc.pvt>
References: <9711F93A3E3A3C458276A476FFA91AECC22BAF@GV1HQPEX03.babgsetc.pvt>
 <CAB63Y-enNBEOU3Y0vwfJSqqu_KW0nyKED6MEzXFDiM-HkyLzkg@mail.gmail.com>
 <CAB63Y-cTwnkB-mepLQqfmpeUnDPwFWnDRda4pvUUWk3UgY=eqA@mail.gmail.com>
 <9711F93A3E3A3C458276A476FFA91AECC22C15@GV1HQPEX03.babgsetc.pvt>
 <66BD268F973E3544A665E5F503FB38B70C257F4A92@DC01.bmw-carit.intra>
In-Reply-To: <66BD268F973E3544A665E5F503FB38B70C257F4A92@DC01.bmw-carit.intra>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hello Michael,

I have started to investigate the memory consumption issue in more detail, =
and can confirm it is leaking around 32 bytes per call of say_hello(). That=
 is about the size of the User helloworld_object and the user->name wchar_t=
 which is allocated in the client loop, but clearly must be a copy of that =
data if this is the issue. The leak traceback is below (for a run with a lo=
op length of 1000):

32,768 bytes in 4 blocks are possibly lost in loss record 567 of 567
    at  malloc (vg_replace_malloc.c:195)
    by apr_pool_create_ex (apr_pools.c:344)
    by new_queue (etch_queue.c:59)
    by new_mailbox_a (etch_plain_mailbox.c:119)
    by new_mailbox (etch_plain_mailbox.c:82)
    by pmboxmgr_transport_call (etch_plain_mailbox_manager.c:543)
    by tcpdelsvc_begincall (etch_transport.c:525)
    by etchremote_begincall (etch_remote.c:129)
    by helloworld_remote_begin_server_say_hello (helloworld_remote_server.c=
:217)
    by helloworld_remote_server_say_hello (helloworld_remote_server.c:276)
    by main (helloworld_client_main.c:183)
=20
The leak size at this location varies depending on the number of times I it=
erate over the say_hello() invocation. Here are the leak sizes from this ro=
utine at various loop durations:

500 -> 16,384 bytes lost
1,000 -> 32,768 bytes lost
5,000 -> 155,648 bytes lost
10,000 -> 319,488 bytes lost

However, this memory is released upon a call to etch_runtime_shutdown(), so=
 you cannot see the leak unless you interrupt the program execution (or wat=
ch the memory grow...).

As for my platform:
- Linux CentOS 5.4 (32 bit i686)
- libapr-1.4.5
- libapr-util-1.3.12
- libapr-iconv-1.2.1
- etch (latest from SVN)
- gcc 4.1.2

I hope that the traceback may give you some clue as to the source of the le=
ak. Otherwise, I will dig deeper when I find more time to focus on this aga=
in.

Viele Gruesse,

--thomas
=20
-----Original Message-----
From: Michael Fitzner [mailto:Michael.Fitzner@bmw-carit.de]=20
Sent: Wednesday, September 14, 2011 6:34 AM
To: etch-dev@incubator.apache.org
Subject: AW: Etch/C Memory Consumption

Hi Thomas,
Your code should be correct and no memory leaks in this part. Could you pro=
vide me with some more information about what systems do you use (Linux, Wi=
ndows) and a call stack of you memory leak if available. I will also try to=
 reproduce your test.

Thanks
Michael

-----Urspr=FCngliche Nachricht-----
Von: Thomas Marsh [mailto:ThomasMarsh@gamestop.com]
Gesendet: Dienstag, 13. September 2011 16:47
An: etch-dev@incubator.apache.org
Betreff: RE: Etch/C Memory Consumption

Hello Martijn,

Thanks for your response. However, this is not the source of the leak. Ther=
e is a clear etch_object_destroy in the helloworld_remote_begin_server_say_=
hello() which deallocates any parameters to the methods (meaning you will g=
et a segfault if you try to reuse the parameters which now no longer point =
to valid memory). This usage semantic is also clearly stated in the C Bindi=
ng notes

>From http://incubator.apache.org/etch/c-binding-tips-tricks.html:

	The C Binding for Etch has the following memory management rules:

	Implementation side: Parameters of functions have to be destroyed by the f=
unction implementation using etch_object_destroy.
	Caller side: Result Objects have to be freed by the caller using etch_obje=
ct_destroy. Parameters of calls will be freed by the runtime automatically.

To reiterate, program memory usage in this very simple usage scenario _must=
_ be stable. I suspect an ever growing hash table, or some other similar cu=
lprit within the runtime. Can any of the Etch/C binding developers comment?

Thanks, and best regards,

--thomas

-----Original Message-----
From: Martijn Dashorst [mailto:martijn.dashorst@gmail.com]
Sent: Tuesday, September 13, 2011 7:23 AM
To: etch-dev@incubator.apache.org
Subject: Re: Etch/C Memory Consumption

And  user->name =3D new_stringw(L"User"); will do so as well

Martijn

On Tue, Sep 13, 2011 at 1:52 PM, Martijn Dashorst <martijn.dashorst@gmail.c=
om> wrote:
> user =3D new_helloworld_user()
>
> will allocate memory for each pass through the loop.
>
> Martijn
>
> On Mon, Sep 12, 2011 at 11:19 PM, Thomas Marsh <ThomasMarsh@gamestop.com>=
 wrote:
>> Hello all,
>>
>> I have a question about memory consumption in the Etch/C runtime based o=
n behavior we are seeing within our C client. I have modified the C impleme=
ntation of the HelloWorld example in the distribution to run an infinite lo=
op of requests:
>>
>> In the main() routine of helloworld_client_main.c:
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0...
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0while (1) {
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0user =3D =
new_helloworld_user();
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0user->id =
=3D 5;
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0user->nam=
e =3D new_stringw(L"User");
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0result =
=3D remote->say_hello(remote,=20
>> user);
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (is_et=
ch_exception(result)) {
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0...
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0printf("%=
S\n", result->v.valw);
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0etch_obje=
ct_destroy(result);
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0...
>>
>> While running this, I see that the memory consumption of the client cont=
inually grows. (In this example, it grows by about 1 mB every 5 seconds.) I=
 cannot see the memory leak when testing with valgrind, so it would suggest=
 that the leak is in Etch managed memory which is cleared at exit.
>>
>> My understanding is that the call to remote->say_hello() should delegate=
 responsibility of deallocation of the parameters to the Etch runtime, and =
that the client code is only responsible for deallocating the result object=
. The memory use should be stable within this tight loop. Can anyone commen=
t on the potential cause of the memory consumption?
>>
>> Thanks, and best regards,
>>
>> --thomas
>>
>
>
>
> --
> Become a Wicket expert, learn from the best: http://wicketinaction.com
>


--
Become a Wicket expert, learn from the best: http://wicketinaction.com