From users-return-6937-archive-asf-public=cust-asf.ponee.io@trafficserver.apache.org Tue Jan 23 09:36:53 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 2CD77180621 for ; Tue, 23 Jan 2018 09:36:53 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 1CF94160C39; Tue, 23 Jan 2018 08:36:53 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3BAC9160C17 for ; Tue, 23 Jan 2018 09:36:52 +0100 (CET) Received: (qmail 26826 invoked by uid 500); 23 Jan 2018 08:36:51 -0000 Mailing-List: contact users-help@trafficserver.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@trafficserver.apache.org Delivered-To: mailing list users@trafficserver.apache.org Received: (qmail 26816 invoked by uid 99); 23 Jan 2018 08:36:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Jan 2018 08:36:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E0AAB180162 for ; Tue, 23 Jan 2018 08:36:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id QHQfgpszV_rP for ; Tue, 23 Jan 2018 08:36:48 +0000 (UTC) Received: from mail-qt0-f171.google.com (mail-qt0-f171.google.com [209.85.216.171]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 3B2BF5F2EE for ; Tue, 23 Jan 2018 08:36:47 +0000 (UTC) Received: by mail-qt0-f171.google.com with SMTP id s39so27780098qth.7 for ; Tue, 23 Jan 2018 00:36:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=8XmTiuYu/H+HV8BLZTWenZke95oLCuoBxxg59Bz3qrM=; b=MeoK5ZVMeZ4t/tAqVrPdgi+Yy+1LUqREAB4VFYJAPoYPKroE7iqrUyZJQOW1+n+/XY 2qb8nFisY/K19xB2HYZdUJ793lw+FPfRQ+SrR8vMXiOkcWiNEzP53GJoabBihKR1PZrI bvCZtskUGzwq13Lr14JniH2NVaKizgGCzC5+Y3dFrzRc5Y2zTfYEWa+lrWHNS9jGOPVB qyAERSWPryj4hznw+01Gt1BNgj7GlmJO4l6XaPXV9FsES3L8vT2PbDqT9UnTM0lBn4z2 H0R9NOF5cFrG9Q021DbtgxwUQbvHpgS7dgR1Hq7x8Faa7EWUa3emPiyGZ3/ZKI3oz5Qc hjMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=8XmTiuYu/H+HV8BLZTWenZke95oLCuoBxxg59Bz3qrM=; b=NHiKqjsQZzpfXDu6924Hd4NmFtn7QhPlPdevsua3XpySot6cNmJRl1bVaun5GDDvB+ NWdWcJj8X/AolK3DnfRRHkvpNCpy2U/+D9pN5xQKeWtGfwwKabd5hQ9/hWiD22BASiwu om2/ck8YGRv7PCEMmwzZEf+62yhJ56MVnmZ9oQGQjbZoXCmsrAV/DWFHMJm4V3gQTS1w fl9sn/kNlsp5zPXXVH4VKwxTxbNWf4TYKleM0E7jyu65j6Qgj7fV6HkSzw8srQYaiJSa KvVywm9w4PDgSEc+zwNHTh6eybf4pvlyehS2YmYJ6hZjrpECXJW7WK4V34eqXLVo2Py3 +ngQ== X-Gm-Message-State: AKwxytdwOVcaheMczn+R7kR3JVbzI3YZbHxWfWsmbepE6VX48KJCTybS 9amd3TuSbmkuysHGKEwH5GjoohYcQeLFUKDw/SI= X-Google-Smtp-Source: AH8x227jRAOMAXV650ufPUeM2pOKGTtXD90j3eitHUAjdgbQMAkzWr1y+CKYKIe9A7p188mPIWzyrBHn+3d5pLVyyPY= X-Received: by 10.55.175.7 with SMTP id y7mr2487827qke.2.1516696605994; Tue, 23 Jan 2018 00:36:45 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.201.15 with HTTP; Tue, 23 Jan 2018 00:36:45 -0800 (PST) In-Reply-To: References: From: Veiko Kukk Date: Tue, 23 Jan 2018 10:36:45 +0200 Message-ID: Subject: Re: Understanding ATS memory usage To: users@trafficserver.apache.org Content-Type: multipart/alternative; boundary="94eb2c06f72c6eb3db05636d727a" --94eb2c06f72c6eb3db05636d727a Content-Type: text/plain; charset="UTF-8" Hi again, During that mysterious task that happens after ~ 50-51 minutes causes requests/responses to slow down very much, even time out. Requests that usually take few hundred milliseconds are now taking over 30s and timing out. This happens only during that time when memory consumption is suddenly dropped by ATS. Happens for both bypassed urls and for hits. ATS version is 7.1.1 and this looks like serious bug for me. Regards, Veiko 2017-12-19 18:28 GMT+02:00 Alan Carroll : > It's a complex subject hard to put in an email. A few notes: > > 1) You shouldn't let an ATS box swap. That almost always ends badly. > Adding more ram or adjusting the configuration to avoid it is better. I > think we set swappiness to 0. > > 2) The cache directory takes memory independent of the ram cache. This is > 10 bytes per directory entry. The number of directory entries is roughly > the cache disk size divided by the average object size as set in > records.config. > > 3) ATS does not use the kernel page cache for its own cache operations. > > 4) A larger ram cache almost always creates better performance, but the > yield curve can differ quite a lot. What the ram cache does is enable > cached data to be served from ram instead of disk. However, once the ram > cache covers most of the working set, additional ram yields marginal > benefits. E.g. putting an object fetched once a day in ram cache is better, > but not very much. > > 5) I think what your'e seeing with your graph is cache directory > synchronization to disk. To do that, ATS allocates memory for a copy of the > cache directory, copies the directory there, then writes it out. It should > be doing that somewhat peicemeal because a full duplicate of cache > directory can be very large. > > > On Tue, Dec 19, 2017 at 3:04 AM, Veiko Kukk wrote: > >> Hi, >> >> Really nobody knows how ATS uses memory? >> >> Veiko >> >> 2017-12-12 14:44 GMT+02:00 Veiko Kukk : >> >>> Hi, >>> >>> I'm confused about ATS memory configuration. I have a server with CentOS >>> 7, ATS 7.1.1, 64GB memory and ~ 10TB disk. >>> traffic_server process takes ~ 23GB memory with the configuration option >>> (8GB) >>> CONFIG proxy.config.cache.ram_cache.size INT 8589934592 >>> <(858)%20993-4592> >>> ATS is using raw partition on HDD. >>> >>> * Why does it swap when there is page cache that's basically free memory >>> that could be used before swapping. vm.swappiness is 10, i had it set to 0 >>> too, then system does not swap. >>> * Considering ATS is using O_DIRECT with raw partitions and it's own >>> memory management for disk cache, would that mean that ATS is not using >>> kernel page cache at all? >>> * Would ATS benefit from larger RAM cache considering it has it's own >>> disk buffer management. >>> >>> Also, most strange is that there are frequent memory usage drops of >>> traffic_server process. After around 50 minutes, 10GB memory is released >>> and immediately consumed again. Attaching screenshot. >>> >>> Regards, >>> Veiko >>> >>> >> > --94eb2c06f72c6eb3db05636d727a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi again,

During that mysterious task t= hat happens after ~ 50-51 minutes causes requests/responses to slow down ve= ry much, even time out.
Requests that usually take few hundred mi= lliseconds are now taking over 30s and timing out. This happens only during= that time when memory consumption is suddenly dropped by ATS. Happens for = both bypassed urls and for hits.
ATS version is 7.1.1 and this lo= oks like serious bug for me.

Regards,
Ve= iko


2017-12-19 18:28 GMT+02:00 Alan Carroll <= solidwallofco= de@oath.com>:
It's a complex subject hard to put in an email. A few notes:
1) You shouldn't let an ATS box swap. That almost always e= nds badly. Adding more ram or adjusting the configuration to avoid it is be= tter. I think we set swappiness to 0.

2) The cache= directory takes memory independent of the ram cache. This is 10 bytes per = directory entry. The number of directory entries is roughly the cache disk = size divided by the average object size as set in records.config.

3) ATS does not use the kernel page cache for its own cache= operations.

4) A larger ram cache almost always c= reates better performance, but the yield curve can differ quite a lot. What= the ram cache does is enable cached data to be served from ram instead of = disk. However, once the ram cache covers most of the working set, additiona= l ram yields marginal benefits. E.g. putting an object fetched once a day i= n ram cache is better, but not very much.

5) I thi= nk what your'e seeing with your graph is cache directory synchronizatio= n to disk. To do that, ATS allocates memory for a copy of the cache directo= ry, copies the directory there, then writes it out. It should be doing that= somewhat peicemeal because a full duplicate of cache directory can be very= large.

<= div class=3D"gmail_extra">
On Tue, Dec 19, 20= 17 at 3:04 AM, Veiko Kukk <veiko.kukk@gmail.com> wrote:
Hi,

Re= ally nobody knows how ATS uses memory?

Veiko
=

2017-12-12 14:44 GMT+02:00 Veiko Kukk <veiko.kukk@gmail.com><= /span>:
Hi,

I'm confused about ATS memory configuration. I have a server wit= h CentOS 7, ATS 7.1.1, 64GB memory and ~ 10TB disk.
traffic_serve= r process takes ~ 23GB memory with the configuration option (8GB)
CONFIG proxy.config.cache.ram_cache.size INT 8589934592
<= div>ATS is using raw partition on HDD.

* Why does = it swap when there is page cache that's basically free memory that coul= d be used before swapping. vm.swappiness is 10, i had it set to 0 too, then= system does not swap.
* Considering ATS is using O_DIRECT with r= aw partitions and it's own memory management for disk cache, would that= mean that ATS is not using kernel page cache at all?
* Would ATS= benefit=C2=A0 from larger RAM cache considering it has it's own disk b= uffer management.

Also, most strange is that there= are frequent memory usage drops of traffic_server process. After around 50= minutes, 10GB memory is released and immediately consumed again. Attaching= screenshot.

Regards,
Veiko



--94eb2c06f72c6eb3db05636d727a--