Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E3772200BDE for ; Fri, 11 Nov 2016 03:38:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E1487160B10; Fri, 11 Nov 2016 02:38:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id AE066160B01 for ; Fri, 11 Nov 2016 03:37:59 +0100 (CET) Received: (qmail 25417 invoked by uid 500); 11 Nov 2016 02:37:58 -0000 Mailing-List: contact users-help@trafficserver.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@trafficserver.apache.org Delivered-To: mailing list users@trafficserver.apache.org Received: (qmail 25397 invoked by uid 99); 11 Nov 2016 02:37:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Nov 2016 02:37:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 10574185F06; Fri, 11 Nov 2016 02:37:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.999 X-Spam-Level: * X-Spam-Status: No, score=1.999 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id AJ95m2So4nDc; Fri, 11 Nov 2016 02:37:56 +0000 (UTC) Received: from mail-pf0-f177.google.com (mail-pf0-f177.google.com [209.85.192.177]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 8D1F25F65A; Fri, 11 Nov 2016 02:37:55 +0000 (UTC) Received: by mail-pf0-f177.google.com with SMTP id i88so2994819pfk.2; Thu, 10 Nov 2016 18:37:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=7xPVABMyCt7mUWbKmaWmbgzVC0unA8IH2aqsHmcse9o=; b=LAlRmjMIaCHD1wy6RGa90ZkvhJP7L/MpLZmsb/RLnNvZMCbhRGz7RlW7H3Z0dCN8wH sOmjuwkO2wBgukR0jLtrRpMM/fEnpzXTOJVAaU1+KK6XqBOWEXEqvxO30kTXcDE2OooN a8VGewCaxnwNhhphc5l5OEjPzHJcEZ+HdD9XKITbFVA6neW8rF2Twy4zFACD5ZymhsS1 JYN4UpHoMHm1KlZzzf1qqo3NhI3dCImOl4wUA7rTdxANz+zAeC6TTkA1O+xOo9OhYKOe 32BuLl0FP/aBwm87nRUVmUT6Wc0tWzNuayYjqP4xTtv7tz6BZCRqxb5rGje6LbR73bmp wBUA== X-Gm-Message-State: ABUngvd/cegrx/6AQFGryeprqwczJjAlp+Kuii4/7RKHopb7Kxhwwux0geKLTqPEw6bl8g== X-Received: by 10.98.50.5 with SMTP id y5mr1862984pfy.141.1478831869411; Thu, 10 Nov 2016 18:37:49 -0800 (PST) Received: from mac-bryan.bryancall.com (70-59-138-58.ptld.qwest.net. [70.59.138.58]) by smtp.gmail.com with ESMTPSA id d15sm10255117pfl.46.2016.11.10.18.37.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 10 Nov 2016 18:37:48 -0800 (PST) From: Bryan Call Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_5AC7AE25-418F-4689-878A-E51ABDEEBD62" Mime-Version: 1.0 (Mac OS X Mail 10.1 \(3251\)) Subject: Re: Address Sanitizer issue in detecting memory leak. Date: Thu, 10 Nov 2016 18:37:47 -0800 In-Reply-To: <36F1C85E-6C40-435F-8E38-661A04A8C9D0@neumob.com> Cc: dev@trafficserver.apache.org To: users@trafficserver.apache.org References: <4FD346F0-C1DB-4421-91B2-C4B9BD2FF73A@neumob.com> <52E17D7C-E234-4E2D-B30F-1767B620AAF7@yahoo.com> <36F1C85E-6C40-435F-8E38-661A04A8C9D0@neumob.com> X-Mailer: Apple Mail (2.3251) archived-at: Fri, 11 Nov 2016 02:38:01 -0000 --Apple-Mail=_5AC7AE25-418F-4689-878A-E51ABDEEBD62 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 It is working on my Fedora 24. You might need to upgrade your compiler = or OS. [bcall@homer ~]$ cat test.cc #include void dummy() { void *x =3D malloc(20); } int main() { dummy(); return 0; } [bcall@homer ~]$ g++ test.cc -fno-omit-frame-pointer -fsanitize=3Daddress [bcall@homer ~]$ ./a.out =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D9344=3D=3DERROR: LeakSanitizer: detected memory leaks Direct leak of 20 byte(s) in 1 object(s) allocated from: #0 0x7f00500b3e60 in malloc (/lib64/libasan.so.3+0xc6e60) #1 0x400777 in dummy() (/home/bcall/a.out+0x400777) #2 0x400787 in main (/home/bcall/a.out+0x400787) #3 0x7f004f3a3730 in __libc_start_main (/lib64/libc.so.6+0x20730) SUMMARY: AddressSanitizer: 20 byte(s) leaked in 1 allocation(s). -Bryan > On Nov 10, 2016, at 2:22 PM, Vamsi Ambati wrote: >=20 > Hi Sudheer, >=20 > Thanks for the quick response. > I tried with your suggested code snippet but Address sanitizer = doesn=E2=80=99t detect the leak. > I even tried with gdb by introducing break point at = __asan_report_error. >=20 >=20 > Vamsi >=20 >> On Nov 10, 2016, at 1:58 PM, Sudheer Vinukonda = > wrote: >>=20 >> Not sure if the *leak* in your test code below is detectable..since = your program is exiting after allocating the memory (system should = automatically reclaim the memory on process exit). >>=20 >> You may try to modify it to something more explicit like the below -=20= >>=20 >>> #include >>> void dummy() { >>> char *c =3D (char*) malloc(20); >> c =3D (char*) malloc(100); >>> } >>> int main() { >>> dummy(); >>> while(1) {} >>> } >>=20 >>=20 >>=20 >> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati > wrote: >>=20 >>>=20 >>> I am trying to resolve a memory leak issue with ATS 7.0 version. >>> We are using ATS from the CDN perspective and developed 3 plugins. >>> DevOps complains that traffic_server process gradually consumes lot = of memory and does release back which is a sign of memory leak. >>> I tried with Valgrind but it didn=E2=80=99t work because of LUA and = one of our ATS community member suggested to use Address Sanitizer. >>> I followed exactly the steps mentioned Debugging Traffic Server = using ASAN = . >>>=20 >>> I have build the traffic server with = CXXFLAGS=3D-fno-omit-frame-pointer -fsanitize=3Daddress >>> Verified that traffic_server is build with ASAN library by 'ldd = bin/traffic_server=E2=80=99 >>> Restarted the traffic server with 'service nm-trafficserver = restart'. >>>=20 >>> I have adopted three approaches to catch the memory leak >>>=20 >>> Approach 1:=20 >>> Just running the traffic server(with -f option) and continuously = sending HTTP requests using wrk (from another machine) by attaching=20 >>> to a gdb . >>> ' gdb $(pidof traffic_server)=E2=80=99 >>> break __asan_report_error. >>>=20 >>> Issue: I didn=E2=80=99t hit this function from the memory leak = perspective. >>>=20 >>> Approach 2: >>> I have run the traffic server with PROXY_AUTO_EXIT and sending the = http requests=20 >>> PROXY_AUTO_EXIT=3D30 ASAN_OPTIONS=3Ddetect_leaks=3D1:verbosity=3D2 = bin/traffic_server -f >>>=20 >>> Issue: Tool didn=E2=80=99t complain any thing about the memory = leak. >>>=20 >>>=20 >>> Approach 3: >>> I have created simple code explicitly introducing the memory leak >>>=20 >>> #include >>> void dummy() { >>> malloc(20); >>> } >>> int main() { >>> dummy(); >>> return 0; >>> } >>>=20 >>> Compiled with g++ -fsanitize=3Daddress add.cpp=20 >>> Executed like ASAN_OPTIONS=3Ddetect_leaks=3D1 ./a.out >>>=20 >>> Issue: Tool didn=E2=80=99t detect this memory leak. >>>=20 >>>=20 >>> Production System Configuration: >>>=20 >>> 16Gb Ram with 8 cores VM. >>> gcc --version >>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 >>>=20 >>> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, = how did they resolve it ? >>> Any other suggestions how to narrow down the memory leak issue with = ATS ? >>>=20 >>>=20 >>>=20 >>> Regards >>> Vamsi >>>=20 >>> PS: I am new to the community and new to Apache traffic server open = source module too. >>>=20 >>>=20 >>>=20 >>>=20 >=20 --Apple-Mail=_5AC7AE25-418F-4689-878A-E51ABDEEBD62 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 It is working on my Fedora 24.  You might need to = upgrade your compiler or OS.

[bcall@homer ~]$ cat test.cc
#include = <stdlib.h>
void dummy() {
  void *x =3D malloc(20);
}
int main() {
  dummy();
  return 0;
}
[bcall@homer ~]$ g++ test.cc -fno-omit-frame-pointer = -fsanitize=3Daddress
[bcall@homer ~]$ = ./a.out

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=3D=3D9344=3D=3DERROR: LeakSanitizer: detected memory = leaks

Direct = leak of 20 byte(s) in 1 object(s) allocated from:
    #0 0x7f00500b3e60 in malloc = (/lib64/libasan.so.3+0xc6e60)
    #1 = 0x400777 in dummy() (/home/bcall/a.out+0x400777)
    #2 0x400787 in main = (/home/bcall/a.out+0x400787)
    #3 = 0x7f004f3a3730 in __libc_start_main (/lib64/libc.so.6+0x20730)

SUMMARY: = AddressSanitizer: 20 byte(s) leaked in 1 allocation(s).

-Bryan




On Nov 10, 2016, at 2:22 PM, Vamsi Ambati <vamsi@neumob.com> = wrote:

Hi = Sudheer,

Thanks for = the quick response.
I tried with your suggested = code snippet but Address sanitizer doesn=E2=80=99t detect the = leak.
I even tried with gdb by introducing break = point at __asan_report_error.


Vamsi

On Nov = 10, 2016, at 1:58 PM, Sudheer Vinukonda <sudheervinukonda@yahoo.com> wrote:

Not sure if the *leak* in your test code below is = detectable..since your program is exiting after allocating the memory = (system should automatically reclaim the memory on process = exit).

You may = try to modify it to something more explicit like the below = - 

#include = <stdlib.h>
void dummy() = {
  char *c =3D (char*) = malloc(20);
   c =3D = (char*) malloc(100);
}
int main() = {
  dummy();
  while(1) = {}
}


On Nov = 10, 2016, at 1:43 PM, Vamsi Ambati <vamsi@neumob.com> wrote:


I am = trying to resolve a memory leak issue with ATS 7.0 version.
We are using ATS from the CDN perspective and developed 3 = plugins.
DevOps complains that  traffic_server = process gradually consumes lot of memory and does release back which is = a sign of memory leak.
I tried with Valgrind but it = didn=E2=80=99t work because of LUA and one of our ATS community member = suggested to  use Address Sanitizer.
I = followed exactly the steps mentioned   Debugging Traffic Server using ASAN.

I have build the traffic = server with CXXFLAGS=3D-fno-omit-frame-pointer = -fsanitize=3Daddress
Verified that = traffic_server is build with ASAN library by  'ldd bin/traffic_server=E2=80=99
Restarted the = traffic server with 'service nm-trafficserver restart'.

I have adopted three approaches to catch the memory = leak

Approach 1: 
Just running the traffic server(with -f = option) and continuously sending HTTP requests using wrk  (from = another machine) by attaching 
to a gdb .
' gdb $(pidof = traffic_server)=E2=80=99
break = __asan_report_error.

Issue: I didn=E2=80=99t hit this function = from the memory leak perspective.

Approach 2:
I have run the traffic server with = PROXY_AUTO_EXIT and sending the http requests 
PROXY_AUTO_EXIT=3D30 ASAN_OPTIONS=3Ddetect_leaks=3D1:verbosity=3D= 2  bin/traffic_server -f

Issue: Tool = didn=E2=80=99t complain any thing about the  memory leak.


Approach 3:
I have = created simple code explicitly introducing the memory leak

#include <stdlib.h>
void dummy() = {
  malloc(20);
}
int main() {
  = dummy();
  return 0;
}

Compiled with g++ -fsanitize=3Daddress add.cpp 
Executed = like ASAN_OPTIONS=3Ddetect_leaks=3D1 = ./a.out

Issue: Tool didn=E2=80=99t detect= this memory leak.


Production System = Configuration:

  • 16Gb Ram with 8 = cores VM.
  • gcc --version
= gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

Did any one encounter this  ASAN  issue on Ubuntu = 14.04  ? If so, how did they resolve it ?
Any other = suggestions how to narrow down the memory leak issue with ATS = ?



Regards
   Vamsi

PS: I am new to the = community and new to Apache traffic server open source module = too.






= --Apple-Mail=_5AC7AE25-418F-4689-878A-E51ABDEEBD62--