Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9F736200BBB for ; Thu, 10 Nov 2016 22:59:33 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 9E398160B01; Thu, 10 Nov 2016 21:59:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 96A56160AF7 for ; Thu, 10 Nov 2016 22:59:32 +0100 (CET) Received: (qmail 83348 invoked by uid 500); 10 Nov 2016 21:59:31 -0000 Mailing-List: contact users-help@trafficserver.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@trafficserver.apache.org Delivered-To: mailing list users@trafficserver.apache.org Received: (qmail 83331 invoked by uid 99); 10 Nov 2016 21:59:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Nov 2016 21:59:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 41C42C21C8 for ; Thu, 10 Nov 2016 21:59:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.818 X-Spam-Level: X-Spam-Status: No, score=-1.818 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id eS0mX_bc_2IZ for ; Thu, 10 Nov 2016 21:59:29 +0000 (UTC) Received: from nm22-vm4.bullet.mail.ne1.yahoo.com (nm22-vm4.bullet.mail.ne1.yahoo.com [98.138.91.182]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 7D28660DCA for ; Thu, 10 Nov 2016 21:59:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1478815135; bh=kRMugdQRoa8nRnyG3EWKrZgN3xS82SAD1+l2L2k1dIc=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject; b=VMBldO39iUeCR01ApagIX8kk6mU6E/qk3hLQ5P4HutfvojmlHV+MQNXVg9/miLPBn+JQVFOhKkPY4b57rXF/ZC7KF7AnJSz7j1NTNfe+pF4qTIQLPBfghojJQGt2RB8tE+eIA37vIkoLHDO0XlYUpDHGr8vDfHBd3CtwHSQoOTEiVqqGWnW4839S8HU10OT3YsoaJAlxUsVypv5C8baJZm08s9dzP7aN38/Mo1pipaiRY3Wy2hsl+XtVkxfhjW6PiC/lOJbYgYZT9J6wFKgC+YMAzGeHLkFmb2gvjbz2bRpZSMhvl9aopITOXsquq8QfuQk5fQk5WY2dLzeWGKqmVQ== Received: from [98.138.100.112] by nm22.bullet.mail.ne1.yahoo.com with NNFMP; 10 Nov 2016 21:58:55 -0000 Received: from [98.138.84.40] by tm103.bullet.mail.ne1.yahoo.com with NNFMP; 10 Nov 2016 21:58:55 -0000 Received: from [127.0.0.1] by smtp108.mail.ne1.yahoo.com with NNFMP; 10 Nov 2016 21:58:55 -0000 X-Yahoo-Newman-Id: 248015.34819.bm@smtp108.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: d6HH4R8VM1ktOwb1xx3Y5tt2FmDEdVBgdlTYARNWRUNKgoO lyLx6TpJQH49BT78ugos1BgUhunEhEZcSGhLW_nBp365dAYkfEumG6H3kgfk UreNI7pUdxW4v6tSsURxt.RoD0IBt4_PtbIpy0AwNYx77i6cOYD9DHhrMl2p y_AxkaTe5RJ4OAIyAiHsNB8N2shl_vh5ImnQrQFuXeQn3cEYRf3D1fxTQ4fJ 3SSJOHSmOsfsPri2uY7_9ydFkFwpxVWXyfyjPOtvE5bjmx.gjSMGcsatER1X qCW_XrNVtw4OtQL0NGlozGy73Q6ZZ98OfyKQ1NL_3Bsy6qy_l7f8zMUS5.8v J.L1J5gJjUnZajte_bzyLVaOSyq0DwG96bFI0JjeX0_Q8OGswGXq4wpx_HYJ pamEA5yRWrt_C12S3Ho_jfQHym8v3wb8tDPO1XFAiwxpi5E4raInDByjICZo OQNDOQ1VuBQlljwvOkSQhegwIke4RT48fJ8z.GTOBUAsmh8E2QGBzsFCAzlJ IoSwDPt54p2Ig5p4_8sXDlhetSRyJtEcha5qsX17yuZK0T24_zvrrtg-- X-Yahoo-SMTP: CK8LyZSswBAjH8yIxfpBQZ9YszDcPEFK9OoeLT8- Content-Type: multipart/alternative; boundary=Apple-Mail-DC82F0FA-20A8-4153-A4BA-B3E35E3E73E2 Mime-Version: 1.0 (1.0) Subject: Re: Address Sanitizer issue in detecting memory leak. From: Sudheer Vinukonda X-Mailer: iPhone Mail (14B100) In-Reply-To: <4FD346F0-C1DB-4421-91B2-C4B9BD2FF73A@neumob.com> Date: Thu, 10 Nov 2016 13:58:53 -0800 Cc: dev@trafficserver.apache.org Content-Transfer-Encoding: 7bit Message-Id: <52E17D7C-E234-4E2D-B30F-1767B620AAF7@yahoo.com> References: <4FD346F0-C1DB-4421-91B2-C4B9BD2FF73A@neumob.com> To: users@trafficserver.apache.org archived-at: Thu, 10 Nov 2016 21:59:33 -0000 --Apple-Mail-DC82F0FA-20A8-4153-A4BA-B3E35E3E73E2 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Not sure if the *leak* in your test code below is detectable..since your pro= gram is exiting after allocating the memory (system should automatically rec= laim the memory on process exit). You may try to modify it to something more explicit like the below -=20 > #include > void dummy() { > char *c =3D (char*) malloc(20); c =3D (char*) malloc(100); > } > int main() { > dummy(); > while(1) {} > } > On Nov 10, 2016, at 1:43 PM, Vamsi Ambati wrote: >=20 >=20 > I am trying to resolve a memory leak issue with ATS 7.0 version. > We are using ATS from the CDN perspective and developed 3 plugins. > DevOps complains that traffic_server process gradually consumes lot of me= mory and does release back which is a sign of memory leak. > I tried with Valgrind but it didn=E2=80=99t work because of LUA and one of= our ATS community member suggested to use Address Sanitizer. > I followed exactly the steps mentioned Debugging Traffic Server using AS= AN. >=20 > I have build the traffic server with CXXFLAGS=3D-fno-omit-frame-pointer -f= sanitize=3Daddress > Verified that traffic_server is build with ASAN library by 'ldd bin/traff= ic_server=E2=80=99 > Restarted the traffic server with 'service nm-trafficserver restart'. >=20 > I have adopted three approaches to catch the memory leak >=20 > Approach 1:=20 > Just running the traffic server(with -f option) and continuously sending H= TTP requests using wrk (from another machine) by attaching=20 > to a gdb . > ' gdb $(pidof traffic_server)=E2=80=99 > break __asan_report_error. >=20 > Issue: I didn=E2=80=99t hit this function from the memory leak perspective= . >=20 > Approach 2: > I have run the traffic server with PROXY_AUTO_EXIT and sending the http re= quests=20 > PROXY_AUTO_EXIT=3D30 ASAN_OPTIONS=3Ddetect_leaks=3D1:verbosity=3D2 bin/tr= affic_server -f >=20 > Issue: Tool didn=E2=80=99t complain any thing about the memory leak. >=20 >=20 > Approach 3: > I have created simple code explicitly introducing the memory leak >=20 > #include > void dummy() { > malloc(20); > } > int main() { > dummy(); > return 0; > } >=20 > Compiled with g++ -fsanitize=3Daddress add.cpp=20 > Executed like ASAN_OPTIONS=3Ddetect_leaks=3D1 ./a.out >=20 > Issue: Tool didn=E2=80=99t detect this memory leak. >=20 >=20 > Production System Configuration: >=20 > 16Gb Ram with 8 cores VM. > gcc --version > gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 >=20 > Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did t= hey resolve it ? > Any other suggestions how to narrow down the memory leak issue with ATS ? >=20 >=20 >=20 > Regards > Vamsi >=20 > PS: I am new to the community and new to Apache traffic server open source= module too. >=20 >=20 >=20 >=20 --Apple-Mail-DC82F0FA-20A8-4153-A4BA-B3E35E3E73E2 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Not sure if the *leak* in y= our test code below is detectable..since your program is exiting after alloc= ating the memory (system should automatically reclaim the memory on process e= xit).

You may try to modify it to something more ex= plicit like the below - 

#include <stdlib.h>
void dummy() {
  char *c =3D (char*) malloc(= 20);
   c =3D (char*) ma= lloc(100);
}
int main() {
  du= mmy();
  while(1) {}
}


On Nov 10, 2016, at 1:43 PM, Vamsi Amb= ati <vamsi@neumob.com> wrote:<= br>

I am trying to resolve a memory leak issue with ATS 7.0 version.
We are using ATS from the CDN perspective and developed 3 plugins.
<= div class=3D"">DevOps complains that  traffic_server process gradually c= onsumes lot of memory and does release back which is a sign of memory leak.<= /div>
I tried with Valgrind but it didn=E2=80=99t work becaus= e of LUA and one of our ATS community member suggested to  use Address S= anitizer.
I followed exactly the steps mentioned  =  Debugging Traffic Server using ASAN.

I have build the traffic server= with CXXFLAGS=3D-fno-omit-frame-pointer -fsanitize=3Daddr= ess
Verified that traffic_server is build with A= SAN library by  'ldd bin/tr= affic_server=E2=80=99
Restarted the traffic server with 'service nm-trafficserver restart'.=

I have adopted= three approaches to catch the memory leak

Approa= ch 1: 
Just running the traffic server(with -f option) and continuously sending HT= TP requests using wrk  (from another machine) by attaching =
to a gdb .
' gdb $(pidof traffic_ser= ver)=E2=80=99
b= reak __asan_report_error.

Issue: I didn=E2=80=99t hit this function from the memory leak perspective.

Approach 2:
I have run the traffic server with PROXY= _AUTO_EXIT and sending the http requests 
<= span style=3D"font-family: Menlo; font-size: 11px; font-variant-ligatures: n= o-common-ligatures;" class=3D"">PROXY_AUTO_EXIT=3D30 ASAN_OPTIONS=3Ddetect_l= eaks=3D1:verbosity=3D2  bin/traffic_server -f

Issue: Tool d= idn=E2=80=99t complain any thing about the  memory leak.


App= roach 3:
I have created simple code explicitly intr= oducing the memory leak

#include <stdlib.h>
void dummy() {
  malloc(20);
}
int main() {
  dummy();
  return 0;
}

Compiled with g++ -fsanitize=3Daddress add.cpp 
Executed like&n= bsp;ASAN_OPTIONS=3Ddetect_leaks=3D1 ./a.out

Issue: Tool didn=E2= =80=99t detect this memory leak.

=
Production System Configuration:

  • 16Gb Ram with 8 cores VM.
  • gcc --version
gcc (= Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

Did any one encounter this  ASAN &= nbsp;issue on Ubuntu 14.04  ? If so, how did they resolve it ?
Any other su= ggestions how to narrow down the memory leak issue with ATS ?



Regards
&nb= sp;  Vamsi

PS= : I am new to the community and new to Apache traffic server open source mod= ule too.




= --Apple-Mail-DC82F0FA-20A8-4153-A4BA-B3E35E3E73E2--