Personally I'd just avoid the interim cache all together. If you want tiered storage it is better to do it at the block level. In newer linux kernels there are a few (I'd use http://en.wikipedia.org/wiki/Bcache). If not you can compile kernel modules for other (such as flashcache). These use block-level caching-- so they'll do a better job of caching hot blocks instead of full http responses-- especially useful if clients abort downloads soon after starting. In addition this will let your SSD maintain parts of your files instead of having all or none of it.

On Mon, Jan 12, 2015 at 9:46 AM, Daniel Biazus <daniel.biazus@azion.com> wrote:
Hi guys,

    I agree with your approach, I also think that objects are being migrated too frequently between the storage and interim cache. However, I can't see any workaround regarding this issue, since the interim cache doesn't have a configuration that limits the object size on SSD storage, like we have in RAM cache, "proxy.config.cache.ram_cache_cutoff".
    I tried to configure the "proxy.config.cache.interim.migrate_threshold" variable to a higher value, something like 20, but it is not enough because when a large object (1GB) reaches this threshold, it is copied to interim cache, overloading the CPU and IO. 

perf record output:





Does anyone have an insight about this situation ?

Let me know if you guys needs some more info.

Thank You & Best Regards,


On Mon, Jan 12, 2015 at 1:49 AM, gang li <portl4t.cn@gmail.com> wrote:
I don't think it is a good idea to use interim cache if the size of the cache objects are very large. As the ssd disk is only 120GB,  the objects on the HDD will be migrated to the ssd frequently, and the ssd storage is meaningless as it will be overwritten quickly, and this will increase the consumption of cpu and io.

Can you give more infos from the perf, such as call graph.


On Mon, Jan 12, 2015 at 4:52 AM, Daniel Biazus <daniel.biazus@azion.com> wrote:
Hi Guys,

We' ve been using ATS as a reverse proxy, and a few week ago We started to use the interim cache feature in more intense way, caching objects with the average size of 200 MB and max size of 1 GB. 

We have ~ 1 TB HDD as a default storage:

cat /etc/trafficserver/storage.config

# ATS - Storage
/dev/sda6 volume=1

And also, a 120 GB SDD storage as interim cache:

LOCAL proxy.config.cache.interim.storage STRING /dev/sdc1

After 20 ~ 30 minutes in production with this configuration, We could notice a sudden CPU high usage, increasing up to 65 %, considering that our regular usage is 10 %. However the throughput still stable in 250 Mbps per box. 

We've found the following behavior using the perf top tool:

   88.18%  traffic_server      [.] _Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread
     0.32%  traffic_server      [.] _ZN10NetHandler12mainNetEventEiP5Event
     0.30%  [kernel]             [k] update_sd_lb_stats
     0.29%  [e1000e]           [k] e1000e_check_ltr_demote
     0.25%  [kernel]             [k] __ticket_spin_lock
     0.24%  traffic_server      [.] _ZN7EThread13process_eventEP5Eventi
     0.21%  [kernel]             [k] timerqueue_add
     0.17%  libc-2.12.so       [.] epoll_wait
     0.17%  libpcre.so.0.0.   [.] 0x00000000000100dd
     0.14%  [kernel]             [k] __schedule

1) This behavior is easily reproduced caching large objects with interim cache active.
2) With interim cache disabled, this behavior is not reproduced.

 As you can see, at the perf top output, the write_to_net_io function is responsible for this heavy CPU usage. We would like to hear of you guys, if anyone has faced a issue like that, or if you have any clues about this possible bug.

Thanks & Regards,

--


Daniel Biazus
infrastructure Engineering
Azion Technologies
Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
Miami, USA +1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma de utilização deste documento depende de autorização do emissor, sujeito as penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and privileged, protected by legal confidentiality. The use of this document require authorization by the issuer, subject to penalties.






--


Daniel Biazus
infrastructure Engineering
Azion Technologies
Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
Miami, USA +1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma de utilização deste documento depende de autorização do emissor, sujeito as penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and privileged, protected by legal confidentiality. The use of this document require authorization by the issuer, subject to penalties.