Looking at the memory dump, my first guess would be you have a lot of stalled transactions that never got cleaned up. This is based on the ioBufAllocator[5], which IIRC is the default size for the initial read. The hdrStrHeap and hdrHeap are used for storing request / response headers in memory. Those being so large seems to indicate that data isn't getting cleaned up.

On Wed, Oct 17, 2018 at 2:00 PM Steve Malenfant <smalenfant@gmail.com> wrote:
I tried the stats_over_http plugin today and it's behaving the same. memory/hdrHeap and memory/hdrStrHeap are increasing on every requests.

Will file an issue. Not only a Virtual Machine issue.

Tried to get jemalloc to help us but I believe we failed. No output.

How to reproduce?

#!/bin/bash

while true; do

   curl http://<ip>/_stats >/dev/null

done


Steve

On Mon, Sep 10, 2018 at 7:21 AM Steve Malenfant <smalenfant@gmail.com> wrote:
Ever since we upgraded to 6.x or 7.x, it seems like we have issues with memory utilization climbing over time in a 4GB VM. It takes in average about 5 days before it starts swapping.

Current version: 7.1.4

There is only a few requests getting to those VMs and they are creating quite a problem on the VM hosts due to excessive swapping (IO). Most of those requests, if not all are using the astats_over_http plugin for Traffic Control.

I've condensed the memory dump just to show what's being used.
Seems like hdrStrHeap seems to be using quite a bit of memory and keeps increasing over time.

Is there anything specific I would need to look into? 

-----------------------------------------------------------------------------------------
     Allocated      |        In-Use      | Type Size  |   Free List Name
--------------------|--------------------|------------|----------------------------------
            2097152 |                  0 |      32768 | memory/ioBufAllocator[8]
             524288 |                  0 |       8192 | memory/ioBufAllocator[6]
           96993280 |           96661504 |       4096 | memory/ioBufAllocator[5]
              16384 |                128 |        128 | memory/ioBufAllocator[0]
              73728 |              45312 |         96 | memory/eventAllocator
            1407600 |            1399440 |         80 | memory/mutexAllocator
              24576 |              22016 |         64 | memory/ioBlockAllocator
            1150560 |            1145328 |         48 | memory/ioDataAllocator
              65280 |              62400 |        240 | memory/ioAllocator
              97760 |              25568 |        752 | memory/netVCAllocator
             114400 |                  0 |        880 | memory/sslNetVCAllocator
              67712 |              33856 |      33856 | memory/dnsBufAllocator
             163840 |                  0 |       1280 | memory/dnsEntryAllocator
               4096 |                 16 |         16 | memory/expiryQueueEntry
               8192 |                 64 |         64 | memory/refCountCacheHashingValueAllocator
             296960 |               2320 |       2320 | memory/hostDBContAllocator
         1162870784 |         1162797056 |       2048 | memory/hdrStrHeap
         1162870784 |         1162862592 |       2048 | memory/hdrHeap
              24576 |               6144 |        192 | memory/httpServerSessionAllocator
            1990656 |                  0 |       7776 | memory/httpSMAllocator
             114688 |              30464 |        896 | memory/http1ClientSessionAllocator
              24576 |               6624 |         96 | memory/INKContAllocator
               4096 |                224 |         32 | memory/apiHookAllocator
             262144 |                  0 |       1024 | memory/ArenaBlock
         2431268112 |         2425101056 |            | TOTAL
-----------------------------------------------------------------------------------------

Steve


--
Beware the fisherman who's casting out his line in to a dried up riverbed.
Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.
It's easier that way. - Genesis : Duke : VI 25-28