Hi again,

During that mysterious task that happens after ~ 50-51 minutes causes requests/responses to slow down very much, even time out.
Requests that usually take few hundred milliseconds are now taking over 30s and timing out. This happens only during that time when memory consumption is suddenly dropped by ATS. Happens for both bypassed urls and for hits.
ATS version is 7.1.1 and this looks like serious bug for me.

Regards,
Veiko


2017-12-19 18:28 GMT+02:00 Alan Carroll <solidwallofcode@oath.com>:
It's a complex subject hard to put in an email. A few notes:

1) You shouldn't let an ATS box swap. That almost always ends badly. Adding more ram or adjusting the configuration to avoid it is better. I think we set swappiness to 0.

2) The cache directory takes memory independent of the ram cache. This is 10 bytes per directory entry. The number of directory entries is roughly the cache disk size divided by the average object size as set in records.config.

3) ATS does not use the kernel page cache for its own cache operations.

4) A larger ram cache almost always creates better performance, but the yield curve can differ quite a lot. What the ram cache does is enable cached data to be served from ram instead of disk. However, once the ram cache covers most of the working set, additional ram yields marginal benefits. E.g. putting an object fetched once a day in ram cache is better, but not very much.

5) I think what your'e seeing with your graph is cache directory synchronization to disk. To do that, ATS allocates memory for a copy of the cache directory, copies the directory there, then writes it out. It should be doing that somewhat peicemeal because a full duplicate of cache directory can be very large.


On Tue, Dec 19, 2017 at 3:04 AM, Veiko Kukk <veiko.kukk@gmail.com> wrote:
Hi,

Really nobody knows how ATS uses memory?

Veiko

2017-12-12 14:44 GMT+02:00 Veiko Kukk <veiko.kukk@gmail.com>:
Hi,

I'm confused about ATS memory configuration. I have a server with CentOS 7, ATS 7.1.1, 64GB memory and ~ 10TB disk.
traffic_server process takes ~ 23GB memory with the configuration option (8GB)
CONFIG proxy.config.cache.ram_cache.size INT 8589934592
ATS is using raw partition on HDD.

* Why does it swap when there is page cache that's basically free memory that could be used before swapping. vm.swappiness is 10, i had it set to 0 too, then system does not swap.
* Considering ATS is using O_DIRECT with raw partitions and it's own memory management for disk cache, would that mean that ATS is not using kernel page cache at all?
* Would ATS benefitĀ  from larger RAM cache considering it has it's own disk buffer management.

Also, most strange is that there are frequent memory usage drops of traffic_server process. After around 50 minutes, 10GB memory is released and immediately consumed again. Attaching screenshot.

Regards,
Veiko