trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Malenfant <smalenf...@gmail.com>
Subject Re: ksoftirqd and traffic server load testing
Date Tue, 28 Jul 2015 12:28:02 GMT
So I should have sent an update already. Irqbalance wasn't
running/installed. This was pinning all network IO on a single CPU causing
this problem. After installing/starting irqbalance the performance issues
went away.

Although I'm aware this cause latency issues by spreading requests across
all CPUs. More testing will be required.


On Monday, July 27, 2015, Leif Hedstrom <zwoop@apache.org> wrote:

>
> On Jul 24, 2015, at 9:15 AM, Steve Malenfant <smalenfant@gmail.com
> <javascript:_e(%7B%7D,'cvml','smalenfant@gmail.com');>> wrote:
>
> During HLS load testing, I'm seeing high ksoftirqd usage (99%) and traffic
> starts to slow down. Load on traffic server seems OK.
>
> Has anybody noticed that before and is there an irqbalance/numa change
> that needs to be used by ixgbe adapter to work around it?
>
>    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>
>
>
>      9 root      20   0     0    0    0 R 98.0  0.0  26:07.07 ksoftirqd/1
>
>
> There’s not a whole lot you can do on the ATS side of things. You can
> fiddle with the NUMA / CPU affinity settings, and perhaps disable accept
> threads as well (that allows the conn to be accepted and processed on a
> single net-thread).
>
> Now, I know for a fact that Linux is notoriously annoying when it comes to
> SIRQs. Our team at work spent a significant amount of time figuring out how
> to balance the IRQs best. I’ll see if maybe John or Ask can contribute
> something here, explaining all the magic they had to do. Ask / John, either
> of you reading this? :-).
>
> — leif
>
> CONFIG proxy.config.accept_threads INT 0
> CONFIG proxy.config.exec_thread.affinity INT 1
>
>

Mime
View raw message