trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Biazus <daniel.bia...@azion.com>
Subject Re: Interim cache - High CPU usage
Date Mon, 12 Jan 2015 17:46:43 GMT
Hi guys,

    I agree with your approach, I also think that objects are being
migrated too frequently between the storage and interim cache. However, I
can't see any workaround regarding this issue, since the interim cache
doesn't have a configuration that limits the object size on SSD storage,
like we have in RAM cache, "proxy.config.cache.ram_cache_cutoff".
    I tried to configure the "proxy.config.cache.interim.migrate_threshold"
variable to a higher value, something like 20, but it is not enough because
when a large object (1GB) reaches this threshold, it is copied to interim
cache, overloading the CPU and IO.

perf record output:



​

Does anyone have an insight about this situation ?

Let me know if you guys needs some more info.

Thank You & Best Regards,


On Mon, Jan 12, 2015 at 1:49 AM, gang li <portl4t.cn@gmail.com> wrote:

> I don't think it is a good idea to use interim cache if the size of the
> cache objects are very large. As the ssd disk is only 120GB,  the objects
> on the HDD will be migrated to the ssd frequently, and the ssd storage is
> meaningless as it will be overwritten quickly, and this will increase the
> consumption of cpu and io.
>
> Can you give more infos from the perf, such as call graph.
>
>
> On Mon, Jan 12, 2015 at 4:52 AM, Daniel Biazus <daniel.biazus@azion.com>
> wrote:
>
>> Hi Guys,
>>
>> We' ve been using ATS as a reverse proxy, and a few week ago We started
>> to use the interim cache feature in more intense way, caching objects with
>> the average size of 200 MB and max size of 1 GB.
>>
>> *We have ~ 1 TB HDD as a default storage:*
>>
>> cat /etc/trafficserver/storage.config
>>
>> # ATS - Storage
>> /dev/sda6 volume=1
>>
>> *And also, a 120 GB SDD storage as interim cache:*
>>
>> LOCAL proxy.config.cache.interim.storage STRING /dev/sdc1
>>
>> After 20 ~ 30 minutes in production with this configuration, We could
>> notice a sudden CPU high usage, increasing up to 65 %, considering that our
>> regular usage is 10 %. However the throughput still stable in 250 Mbps per
>> box.
>>
>> We've found the following behavior using the perf top tool:
>>
>>    88.18%  traffic_server      [.]
>> _Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread
>>      0.32%  traffic_server      [.] _ZN10NetHandler12mainNetEventEiP5Event
>>      0.30%  [kernel]             [k] update_sd_lb_stats
>>      0.29%  [e1000e]           [k] e1000e_check_ltr_demote
>>      0.25%  [kernel]             [k] __ticket_spin_lock
>>      0.24%  traffic_server      [.] _ZN7EThread13process_eventEP5Eventi
>>      0.21%  [kernel]             [k] timerqueue_add
>>      0.17%  libc-2.12.so       [.] epoll_wait
>>      0.17%  libpcre.so.0.0.   [.] 0x00000000000100dd
>>      0.14%  [kernel]             [k] __schedule
>>
>> 1) This behavior is easily reproduced caching* large objects with
>> interim cache active*.
>> 2) With interim cache *disabled*, this behavior* is not reproduced.*
>>
>>  As you can see, at the perf top output, the write_to_net_*io *function
>> is responsible for this heavy CPU usage. We would like to hear of you guys,
>> if anyone has faced a issue like that, or if you have any clues about this
>> possible bug.
>>
>> Thanks & Regards,
>>
>> --
>>
>> Daniel Biazus
>> infrastructure Engineering
>> Azion Technologies
>> Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
>> Miami, USA +1 305 704 8816
>>
>> Quaisquer informações contidas neste e-mail e anexos podem ser
>> confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
>> de utilização deste documento depende de autorização do emissor, sujeito as
>> penalidades cabíveis.
>>
>> Any information in this e-mail and attachments may be confidential and
>> privileged, protected by legal confidentiality. The use of this document
>> require authorization by the issuer, subject to penalties.
>>
>>
>


-- 

Daniel Biazus
infrastructure Engineering
Azion Technologies
Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
Miami, USA +1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.

Mime
View raw message