Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 109A7200BC7 for ; Thu, 10 Nov 2016 22:44:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 0F5C5160B01; Thu, 10 Nov 2016 21:44:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 09D78160AF7 for ; Thu, 10 Nov 2016 22:43:58 +0100 (CET) Received: (qmail 36240 invoked by uid 500); 10 Nov 2016 21:43:58 -0000 Mailing-List: contact users-help@trafficserver.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@trafficserver.apache.org Delivered-To: mailing list users@trafficserver.apache.org Received: (qmail 36225 invoked by uid 99); 10 Nov 2016 21:43:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Nov 2016 21:43:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A05E61A0725 for ; Thu, 10 Nov 2016 21:43:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.398 X-Spam-Level: ** X-Spam-Status: No, score=2.398 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=neumob.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id GNu_dxyPIzYM for ; Thu, 10 Nov 2016 21:43:56 +0000 (UTC) Received: from mail-pf0-f177.google.com (mail-pf0-f177.google.com [209.85.192.177]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id C38B05F2F2 for ; Thu, 10 Nov 2016 21:43:55 +0000 (UTC) Received: by mail-pf0-f177.google.com with SMTP id n85so152651055pfi.1 for ; Thu, 10 Nov 2016 13:43:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=neumob.com; s=google; h=from:subject:date:message-id:cc:to:mime-version; bh=m/qVSpjLSq8DKHkgcnlHL3x5tZ8lMu9DMl9lvhN8RQQ=; b=gu0qq6m+sn8Gx1dYBWgh6ZUoXW2AM5Cn2sYeUmgo9UlsFfFXuMVbXtQjDh/9parb4y 4fFG1Nrg6PJDqi93TpenE7aCN8zX1KLKOmteVzajIy814OU7X6VNf9loI0tPrHDcgFBS azOMxJLRYUGDiPluTS+bVxq+TpXMf70GgLqF0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:subject:date:message-id:cc:to:mime-version; bh=m/qVSpjLSq8DKHkgcnlHL3x5tZ8lMu9DMl9lvhN8RQQ=; b=RUS6TaE7ehDeNpwcrUb0t5FtGc8Qasz4HFz2U+tTAU35+KJLU59f0XzK378RYQ5agB vRlNjjNN7EMk4kNdXBojoPcLV3vSfswCwTujPLQuX7+CCzybg5Gv4pLUyfip+v4sFjst f6+OW+PGDoKI+GgjA+rZliOPYYuyBmFw/Ac83sDN3kPVi6rtvLXYJ5PZmgobiIdWL7SG Let8TCHgoaGR93QPmltolk2jC8DmmJdVsaivA84ZFr3jtzd1XyWR2Kc33Bst8dX+dvjK UeQalx33QIXPqE6qYAbgboIzD0RlxERwothd32nR+LVu0EX8LFrNJ96aVOlAhD+IMckj t3Rg== X-Gm-Message-State: ABUngvcN8qktpUH14/oyxEAWhMdHTr/ZvG6sP6A3F0rYNsXtwdG6YnSBdjlvfA4OYgUWl/zU X-Received: by 10.98.22.196 with SMTP id 187mr14002722pfw.34.1478814234851; Thu, 10 Nov 2016 13:43:54 -0800 (PST) Received: from [10.178.5.114] ([67.21.0.132]) by smtp.gmail.com with ESMTPSA id a66sm9463063pfa.64.2016.11.10.13.43.54 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 10 Nov 2016 13:43:54 -0800 (PST) From: Vamsi Ambati Content-Type: multipart/alternative; boundary="Apple-Mail=_F79DFB2F-D4C9-4E39-A70A-1B741C25DDAB" Subject: Address Sanitizer issue in detecting memory leak. Date: Thu, 10 Nov 2016 13:43:52 -0800 Message-Id: <4FD346F0-C1DB-4421-91B2-C4B9BD2FF73A@neumob.com> Cc: users@trafficserver.apache.org To: dev@trafficserver.apache.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) archived-at: Thu, 10 Nov 2016 21:44:00 -0000 --Apple-Mail=_F79DFB2F-D4C9-4E39-A70A-1B741C25DDAB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I am trying to resolve a memory leak issue with ATS 7.0 version. We are using ATS from the CDN perspective and developed 3 plugins. DevOps complains that traffic_server process gradually consumes lot of = memory and does release back which is a sign of memory leak. I tried with Valgrind but it didn=E2=80=99t work because of LUA and one = of our ATS community member suggested to use Address Sanitizer. I followed exactly the steps mentioned Debugging Traffic Server using = ASAN = . I have build the traffic server with CXXFLAGS=3D-fno-omit-frame-pointer = -fsanitize=3Daddress Verified that traffic_server is build with ASAN library by 'ldd = bin/traffic_server=E2=80=99 Restarted the traffic server with 'service nm-trafficserver restart'. I have adopted three approaches to catch the memory leak Approach 1:=20 Just running the traffic server(with -f option) and continuously sending = HTTP requests using wrk (from another machine) by attaching=20 to a gdb . ' gdb $(pidof traffic_server)=E2=80=99 break __asan_report_error. Issue: I didn=E2=80=99t hit this function from the memory leak = perspective. Approach 2: I have run the traffic server with PROXY_AUTO_EXIT and sending the http = requests=20 PROXY_AUTO_EXIT=3D30 ASAN_OPTIONS=3Ddetect_leaks=3D1:verbosity=3D2 = bin/traffic_server -f Issue: Tool didn=E2=80=99t complain any thing about the memory leak. Approach 3: I have created simple code explicitly introducing the memory leak #include void dummy() { malloc(20); } int main() { dummy(); return 0; } Compiled with g++ -fsanitize=3Daddress add.cpp=20 Executed like ASAN_OPTIONS=3Ddetect_leaks=3D1 ./a.out Issue: Tool didn=E2=80=99t detect this memory leak. Production System Configuration: 16Gb Ram with 8 cores VM. gcc --version gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how = did they resolve it ? Any other suggestions how to narrow down the memory leak issue with ATS = ? Regards Vamsi PS: I am new to the community and new to Apache traffic server open = source module too. --Apple-Mail=_F79DFB2F-D4C9-4E39-A70A-1B741C25DDAB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

I am trying to resolve a = memory leak issue with ATS 7.0 version.
We are using ATS = from the CDN perspective and developed 3 plugins.
DevOps complains that  traffic_server process gradually = consumes lot of memory and does release back which is a sign of memory = leak.
I tried with Valgrind but it didn=E2=80=99t = work because of LUA and one of our ATS community member suggested to =  use Address Sanitizer.
I followed exactly the = steps mentioned   Debugging Traffic Server using ASAN.

I have build the traffic = server with CXXFLAGS=3D-fno-omit-frame-pointer = -fsanitize=3Daddress
Verified that = traffic_server is build with ASAN library by  'ldd bin/traffic_server=E2=80=99
Restarted the = traffic server with 'service nm-trafficserver restart'.

I have adopted three approaches to catch the memory = leak

Approach 1: 
Just running the traffic server(with -f = option) and continuously sending HTTP requests using wrk  (from = another machine) by attaching 
to a gdb .
' gdb $(pidof = traffic_server)=E2=80=99
break = __asan_report_error.

Issue: I didn=E2=80=99t hit this function = from the memory leak perspective.

Approach 2:
I have run the traffic server with = PROXY_AUTO_EXIT and sending the http requests 
PROXY_AUTO_EXIT=3D30 ASAN_OPTIONS=3Ddetect_leaks=3D1:verbosity=3D= 2  bin/traffic_server -f

Issue: Tool = didn=E2=80=99t complain any thing about the  memory leak.


Approach 3:
I have = created simple code explicitly introducing the memory leak

#include <stdlib.h>
void dummy() = {
  malloc(20);
}
int main() {
  = dummy();
  return 0;
}

Compiled with g++ -fsanitize=3Daddress add.cpp 
Executed = like ASAN_OPTIONS=3Ddetect_leaks=3D1 = ./a.out

Issue: Tool didn=E2=80=99t detect= this memory leak.


Production System = Configuration:

  • 16Gb Ram with 8 = cores VM.
  • gcc --version
= gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

Did any one encounter this  ASAN  issue on Ubuntu = 14.04  ? If so, how did they resolve it ?
Any other = suggestions how to narrow down the memory leak issue with ATS = ?



Regards
   Vamsi

PS: I am new to the = community and new to Apache traffic server open source module = too.




= --Apple-Mail=_F79DFB2F-D4C9-4E39-A70A-1B741C25DDAB--