From users-return-7325-archive-asf-public=cust-asf.ponee.io@trafficserver.apache.org Wed Oct 31 17:00:33 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E962218065D for ; Wed, 31 Oct 2018 17:00:32 +0100 (CET) Received: (qmail 77661 invoked by uid 500); 31 Oct 2018 16:00:31 -0000 Mailing-List: contact users-help@trafficserver.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@trafficserver.apache.org Delivered-To: mailing list users@trafficserver.apache.org Received: (qmail 77651 invoked by uid 99); 31 Oct 2018 16:00:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Oct 2018 16:00:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6862C1A02F3 for ; Wed, 31 Oct 2018 16:00:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.888 X-Spam-Level: * X-Spam-Status: No, score=1.888 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=oath.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id FD8Y_o1zCfDr for ; Wed, 31 Oct 2018 16:00:28 +0000 (UTC) Received: from mail-it1-f170.google.com (mail-it1-f170.google.com [209.85.166.170]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E78B65F3B7 for ; Wed, 31 Oct 2018 16:00:27 +0000 (UTC) Received: by mail-it1-f170.google.com with SMTP id e74-v6so18976700ita.2 for ; Wed, 31 Oct 2018 09:00:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oath.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=hUtHCCZt59hqBW1r51EIrJeFsYlvHd1PKbOGwIcVghQ=; b=RMC8n42JYAe14Fmfn9co6JXgA1JhpEOmC7liJTvsp9I63qMvqR3CWernATpY1cOdxG 8j8DBlnIbrtbfnTVDTNE4LMXmc3y222F79njTGdup72XlouRvFpPy5yFXZL1u0D32Dv3 j01ASG0l/gqbTGO8AxqL03uEig1ndfzZEuDETAOi8hbsb6DwXp9WLwd3IpAoTrxfI9Rp d23mdTWPZc1uKF43MDOxWkTncLJW58C1FVPwDfIrckkaduEa5eRlR9/Qo5ALv2lX5GFQ M/etIA5YDiAXxd6U1Wo9i26+wzvdJxaBLtTuBDkl5bs9CgtKovPIG1sea9pqoeYarFiQ GopQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=hUtHCCZt59hqBW1r51EIrJeFsYlvHd1PKbOGwIcVghQ=; b=asJ/iJuR8OkjJflcaLJsMul6O2PIA9rWV/mxjUCodCVP1LFHnT7pJ9pK/WuaqcWqzK SXC+931jGfSbQVMN9SWJIRjoOMVxZhYbCXoQxWN//5lVuOAxye8NfLNLAeEHhKbAdwGE Fel9IFajofh2LrnMlTeYVtAqGaUvZgeoZeSn07UKJ9GUfzrWDx6MNoTUeiT1w3JK/hqF hagu5vmk4MYDFy7099KoleDYjc8Dhh6iHyg23Wq99NgXhYh169aPYGfJKE3KfvpEvS3L VONQDyPH+xVxAzrIP8sCd2uNHmk+c5WEtvhwCn7NO3Wsm3Lo9SNXL1aT7e+hYuK1ma43 Ojgw== X-Gm-Message-State: AGRZ1gK8sgJJO4UtbOBkmXC5oCgFEaBxBBmVH2mqzhR41rKrAI+g/L43 j93GikabPv4XI+q8nXkpuK7KUA8m/p/F5WZBatJcNSl8 X-Google-Smtp-Source: AJdET5dhgK7G9+gPnBNuCq9wEjrcQpwUNYl/cq2OCiG583nS8c8LYpUp2j0xhWy+AzGEcmGy3AnoEFs0XPT7CKY2Fow= X-Received: by 2002:a24:d807:: with SMTP id b7-v6mr1661092itg.100.1541001619988; Wed, 31 Oct 2018 09:00:19 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Alan Carroll Date: Wed, 31 Oct 2018 11:00:08 -0500 Message-ID: Subject: Re: Memory usage - Virtual Machine - 7.1.x To: users@trafficserver.apache.org Content-Type: multipart/alternative; boundary="000000000000288ea4057988664d" --000000000000288ea4057988664d Content-Type: text/plain; charset="UTF-8" Looking at the memory dump, my first guess would be you have a lot of stalled transactions that never got cleaned up. This is based on the ioBufAllocator[5], which IIRC is the default size for the initial read. The hdrStrHeap and hdrHeap are used for storing request / response headers in memory. Those being so large seems to indicate that data isn't getting cleaned up. On Wed, Oct 17, 2018 at 2:00 PM Steve Malenfant wrote: > I tried the stats_over_http plugin today and it's behaving the same. > memory/hdrHeap and memory/hdrStrHeap are increasing on every requests. > > Will file an issue. Not only a Virtual Machine issue. > > Tried to get jemalloc to help us but I believe we failed. No output. > > How to reproduce? > > #!/bin/bash > > while true; do > > curl http:///_stats >/dev/null > > done > > Steve > > On Mon, Sep 10, 2018 at 7:21 AM Steve Malenfant > wrote: > >> Ever since we upgraded to 6.x or 7.x, it seems like we have issues with >> memory utilization climbing over time in a 4GB VM. It takes in average >> about 5 days before it starts swapping. >> >> Current version: 7.1.4 >> >> There is only a few requests getting to those VMs and they are creating >> quite a problem on the VM hosts due to excessive swapping (IO). Most of >> those requests, if not all are using the astats_over_http plugin for >> Traffic Control. >> >> I've condensed the memory dump just to show what's being used. >> Seems like hdrStrHeap seems to be using quite a bit of memory and keeps >> increasing over time. >> >> Is there anything specific I would need to look into? >> >> >> ----------------------------------------------------------------------------------------- >> Allocated | In-Use | Type Size | Free List Name >> >> --------------------|--------------------|------------|---------------------------------- >> 2097152 | 0 | 32768 | >> memory/ioBufAllocator[8] >> 524288 | 0 | 8192 | >> memory/ioBufAllocator[6] >> 96993280 | 96661504 | 4096 | >> memory/ioBufAllocator[5] >> 16384 | 128 | 128 | >> memory/ioBufAllocator[0] >> 73728 | 45312 | 96 | >> memory/eventAllocator >> 1407600 | 1399440 | 80 | >> memory/mutexAllocator >> 24576 | 22016 | 64 | >> memory/ioBlockAllocator >> 1150560 | 1145328 | 48 | >> memory/ioDataAllocator >> 65280 | 62400 | 240 | memory/ioAllocator >> 97760 | 25568 | 752 | >> memory/netVCAllocator >> 114400 | 0 | 880 | >> memory/sslNetVCAllocator >> 67712 | 33856 | 33856 | >> memory/dnsBufAllocator >> 163840 | 0 | 1280 | >> memory/dnsEntryAllocator >> 4096 | 16 | 16 | >> memory/expiryQueueEntry >> 8192 | 64 | 64 | >> memory/refCountCacheHashingValueAllocator >> 296960 | 2320 | 2320 | >> memory/hostDBContAllocator >> 1162870784 | 1162797056 | 2048 | memory/hdrStrHeap >> 1162870784 | 1162862592 | 2048 | memory/hdrHeap >> 24576 | 6144 | 192 | >> memory/httpServerSessionAllocator >> 1990656 | 0 | 7776 | >> memory/httpSMAllocator >> 114688 | 30464 | 896 | >> memory/http1ClientSessionAllocator >> 24576 | 6624 | 96 | >> memory/INKContAllocator >> 4096 | 224 | 32 | >> memory/apiHookAllocator >> 262144 | 0 | 1024 | memory/ArenaBlock >> 2431268112 | 2425101056 | | TOTAL >> >> ----------------------------------------------------------------------------------------- >> >> Steve >> > -- *Beware the fisherman who's casting out his line in to a dried up riverbed.* *Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.* *It's easier that way. *- Genesis : Duke : VI 25-28 --000000000000288ea4057988664d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Looking at the memory dump, my first guess would be you ha= ve a lot of stalled transactions that never got cleaned up. This is based o= n the ioBufAllocator[5], which IIRC is the default size for the initial rea= d. The hdrStrHeap and hdrHeap are used for storing request / response heade= rs in memory. Those being so large seems to indicate that data isn't ge= tting cleaned up.

On W= ed, Oct 17, 2018 at 2:00 PM Steve Malenfant <smalenfant@gmail.com> wrote:
I tried the stats_over_http plugin today and= it's behaving the same. memory/hdrHeap and memory/hdrStrHeap are incre= asing on every requests.

Will file an issue. Not only a = Virtual Machine issue.

Tried to get jemalloc to he= lp us but I believe we failed. No output.

How to r= eproduce?

#!/bin/bash

while true; do

=C2=A0=C2=A0 curl http://<ip>/_stats >/dev/null

done


Steve

=
On Mon, Sep 10, 2018 at 7:21 AM= Steve Malenfant <smalenfant@gmail.com> wrote:
Ever since we upgraded to 6.x or 7.x, it seems like w= e have issues with memory utilization climbing over time in a 4GB VM. It ta= kes in average about 5 days before it starts swapping.

C= urrent version: 7.1.4

There is only a few requests g= etting to those VMs and they are creating quite a problem on the VM hosts d= ue to excessive swapping (IO). Most of those requests, if not all are using= the astats_over_http plugin for Traffic Control.

= I've condensed the memory dump just to show what's being used.
Seems like hdrStrHeap seems to be using quite a bit of memory and kee= ps increasing over time.

Is there anything spe= cific I would need to look into?=C2=A0

-----------= ---------------------------------------------------------------------------= ---
=C2=A0 =C2=A0 =C2=A0Allocated=C2=A0 =C2=A0 =C2=A0 |= =C2=A0 =C2=A0 =C2=A0 =C2=A0 In-Use=C2=A0 =C2=A0 =C2=A0 | Type Size=C2=A0 |= =C2=A0 =C2=A0Free List Name
--------------------|----------------= ----|------------|----------------------------------
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 2097152 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 |=C2=A0 =C2=A0 =C2=A0 32768 | memory/ioBufAl= locator[8]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0524288= |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 |=C2=A0 = =C2=A0 =C2=A0 =C2=A08192 | memory/ioBufAllocator[6]
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A096993280 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A096661504 |=C2=A0 =C2=A0 =C2=A0 =C2=A04096 | memory/ioBufAllocator[5]<= /div>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 16384 |=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 128 |=C2=A0 =C2=A0 =C2=A0 = =C2=A0 128 | memory/ioBufAllocator[0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 73728 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 45312 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A096 | memory/eventAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1407600 |=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 1399440 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A080 | me= mory/mutexAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 24576 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 22016 |=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A064 | memory/ioBlockAllocator
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1150560 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 1145328 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A048 | memory/ioDataAll= ocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 65280 |= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 62400 |=C2=A0 =C2=A0 =C2= =A0 =C2=A0 240 | memory/ioAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 97760 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 25568 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 752 | memory/netVCAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0114400 |=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 |=C2=A0 =C2=A0 =C2=A0 =C2= =A0 880 | memory/sslNetVCAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 67712 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 33856 |=C2=A0 =C2=A0 =C2=A0 33856 | memory/dnsBufAllocator
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0163840 |=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 |=C2=A0 =C2=A0 =C2=A0 =C2= =A01280 | memory/dnsEntryAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A04096 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A016 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A016 | memory/expir= yQueueEntry
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A08192 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A064 |= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A064 | memory/refCountCacheHashingValueAllo= cator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0296960 |=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02320 |=C2=A0 =C2=A0 =C2= =A0 =C2=A02320 | memory/hostDBContAllocator
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A01162870784 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01162797056 |=C2= =A0 =C2=A0 =C2=A0 =C2=A02048 | memory/hdrStrHeap
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A01162870784 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0116286259= 2 |=C2=A0 =C2=A0 =C2=A0 =C2=A02048 | memory/hdrHeap
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 24576 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A06144 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 192 | memory/http= ServerSessionAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 1990656 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 |= =C2=A0 =C2=A0 =C2=A0 =C2=A07776 | memory/httpSMAllocator
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0114688 |=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 30464 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 896 | memory/ht= tp1ClientSessionAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 24576 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A066= 24 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A096 | memory/INKContAllocator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04096 |=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 224 |=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A032 | memory/apiHookAllocator
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0262144 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 0 |=C2=A0 =C2=A0 =C2=A0 =C2=A01024 | memory/ArenaBlock=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02431268112 |=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A02425101056 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | TO= TAL
-------------------------------------------------------------= ----------------------------

Steve


--
Beware the fisherman who's casting out his line in to a dried up= riverbed.
Oh don't try to tell him 'cause he won't = believe. Throw some bread to the ducks instead.
It's e= asier that way. - Genesis : Duke : VI 25-28
--000000000000288ea4057988664d--