cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio <lapostadiser...@gmail.com>
Subject Re: Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine
Date Wed, 30 Oct 2019 21:39:11 GMT
Hi Reid,

I don't have anymore this loading problem.
I solved by changing the Cassandra Driver Configuration.
Now my cluster is pretty stable and I don't have machines with crazy CPU
Load.
The only thing not urgent but I need to investigate is the number of
ESTABLISHED TCP connections. I see just one node having 7K TCP connections
ESTABLISHED while the others are having around 4-6K connection opened. So
the newest nodes added into the cluster have a higher number of ESTABLISHED
TCP connections.

default['cassandra']['sysctl'] = {
'net.ipv4.tcp_keepalive_time' => 60,
'net.ipv4.tcp_keepalive_probes' => 3,
'net.ipv4.tcp_keepalive_intvl' => 10,
'net.core.rmem_max' => 16777216,
'net.core.wmem_max' => 16777216,
'net.core.rmem_default' => 16777216,
'net.core.wmem_default' => 16777216,
'net.core.optmem_max' => 40960,
'net.ipv4.tcp_rmem' => '4096 87380 16777216',
'net.ipv4.tcp_wmem' => '4096 65536 16777216',
'net.ipv4.ip_local_port_range' => '10000 65535',
'net.ipv4.tcp_window_scaling' => 1,
  'net.core.netdev_max_backlog' => 2500,
  'net.core.somaxconn' => 65000,
'vm.max_map_count' => 1048575,
'vm.swappiness' => 0
}

These are my tweaked value and I used the values recommended from datastax.

Do you have something different?

Best,
Sergio

Il giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback <
rpinchback@tripadvisor.com> ha scritto:

> Oh nvm, didn't see the later msg about just posting what your fix was.
>
> R
>
>
> ´╗┐On 10/30/19, 4:24 PM, "Reid Pinchback" <rpinchback@tripadvisor.com>
> wrote:
>
>      Message from External Sender
>
>     Hi Sergio,
>
>     Assuming nobody is actually mounting a SYN flood attack, then this
> sounds like you're either being hammered with connection requests in very
> short periods of time, or your TCP backlog tuning is off.   At least,
> that's where I'd start looking.  If you take that log message and google it
> (Possible SYN flooding... Sending cookies") you'll find explanations.  Or
> just googling "TCP backlog tuning".
>
>     R
>
>
>     On 10/30/19, 3:29 PM, "Sergio Bilello" <lapostadisergio@gmail.com>
> wrote:
>
>         >
>         >Oct 17 00:23:03 prod-personalization-live-data-cassandra-08
> kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending
> cookies. Check SNMP counters.
>
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>     For additional commands, e-mail: user-help@cassandra.apache.org
>
>
>

Mime
View raw message