From user-return-64703-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed Oct 30 21:56:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 47832180654 for ; Wed, 30 Oct 2019 22:56:04 +0100 (CET) Received: (qmail 64806 invoked by uid 500); 30 Oct 2019 21:56:00 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 64795 invoked by uid 99); 30 Oct 2019 21:56:00 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Oct 2019 21:56:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 669C31A42CB for ; Wed, 30 Oct 2019 21:55:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id UhhT8yr9n-Kl for ; Wed, 30 Oct 2019 21:55:54 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::244; helo=mail-lj1-x244.google.com; envelope-from=lapostadisergio@gmail.com; receiver= Received: from mail-lj1-x244.google.com (mail-lj1-x244.google.com [IPv6:2a00:1450:4864:20::244]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 4761F7DE13 for ; Wed, 30 Oct 2019 21:39:24 +0000 (UTC) Received: by mail-lj1-x244.google.com with SMTP id a21so4312171ljh.9 for ; Wed, 30 Oct 2019 14:39:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=SkikogVonptjaF71Nr/ZWr0jhEHCd/jn2hdCdxaHRS0=; b=M3KvnY31NmQAj3YYMOstdWFV1eJyGoq0vvoPEDD22GL/86y0CHZI0WTIOYReNSS0Bq 9isQizXaPUinqJM9SIQ8Pxp4RJtRFwRMc75JlzKZHxStYa8nPrfiKaET69/x+X3y0ZT1 xhCMQMva3nMovPJe1q1rWlCJkWBOt5RK3o3lJM27ULNBtIX++tYxCzurnhhc8S3p+qnW +QHyTJ1ovW6VUbbb+M8iByZ/FKcomT7ED2vBISNj8OqfpNozNNGkBWA4ZMFkAT3zVjAF yyiSIP1LLYW+L5lfDXgV3HJ6BTwgB1mPxaGM9q4Ky7Kwcc+gfYrGKvE3Q1JA1gKfupE2 Lw+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=SkikogVonptjaF71Nr/ZWr0jhEHCd/jn2hdCdxaHRS0=; b=WXLcgyBzSYW3EQEoQXDp9sNfeBo7bOT4NIta9c7tuJLtmpTZ9Nbw+46K/cMcNWGnQR Vow8RxwTnZIN934Qc6YwrNeJopUVAp5Tmfz0ZCcn3UXUunFg544M+m7RIMbhNSuKHT7s ML1Nl2fEQNCrWYn1efnlbYCjgEqAF3xPiubW97PgXzIBjg1FrcuBV/wavvq0C9a6b1Bx fFG2ZBNF1YpmHXMM74vBGzFz/SFsqJNkLcm/VWfcmtrSbclpbRit0kP3VJv04JY+8IYU KSlCSaNzJbEHE2kSCxEXT4wnhy09RKQUfEU7lqg2nkoVX0ektqPpQJ0h4zWIJuWlSvNa lHHA== X-Gm-Message-State: APjAAAUyuiFHnATxhWE7IsUehQ2d7lZywz2MfjGw/t/yNSnk1+Tvbp64 UMyPA6ucacQM4T1aruKmHaLM7vxu/uVXN6g92hwd0MkieYQ= X-Google-Smtp-Source: APXvYqwoWAUX+HmOm92yBl6SBNpg5VmjHxBZL2ZL0m7NlsOuw+rE0c2M39wChXKvL9RY5g2MsQPEEMF9XIJwYw/jh0w= X-Received: by 2002:a2e:b019:: with SMTP id y25mr1143578ljk.153.1572471563048; Wed, 30 Oct 2019 14:39:23 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Sergio Date: Wed, 30 Oct 2019 14:39:11 -0700 Message-ID: Subject: Re: Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="000000000000ef5ea605962790a9" --000000000000ef5ea605962790a9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Reid, I don't have anymore this loading problem. I solved by changing the Cassandra Driver Configuration. Now my cluster is pretty stable and I don't have machines with crazy CPU Load. The only thing not urgent but I need to investigate is the number of ESTABLISHED TCP connections. I see just one node having 7K TCP connections ESTABLISHED while the others are having around 4-6K connection opened. So the newest nodes added into the cluster have a higher number of ESTABLISHED TCP connections. default['cassandra']['sysctl'] =3D { 'net.ipv4.tcp_keepalive_time' =3D> 60, 'net.ipv4.tcp_keepalive_probes' =3D> 3, 'net.ipv4.tcp_keepalive_intvl' =3D> 10, 'net.core.rmem_max' =3D> 16777216, 'net.core.wmem_max' =3D> 16777216, 'net.core.rmem_default' =3D> 16777216, 'net.core.wmem_default' =3D> 16777216, 'net.core.optmem_max' =3D> 40960, 'net.ipv4.tcp_rmem' =3D> '4096 87380 16777216', 'net.ipv4.tcp_wmem' =3D> '4096 65536 16777216', 'net.ipv4.ip_local_port_range' =3D> '10000 65535', 'net.ipv4.tcp_window_scaling' =3D> 1, 'net.core.netdev_max_backlog' =3D> 2500, 'net.core.somaxconn' =3D> 65000, 'vm.max_map_count' =3D> 1048575, 'vm.swappiness' =3D> 0 } These are my tweaked value and I used the values recommended from datastax. Do you have something different? Best, Sergio Il giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback < rpinchback@tripadvisor.com> ha scritto: > Oh nvm, didn't see the later msg about just posting what your fix was. > > R > > > =EF=BB=BFOn 10/30/19, 4:24 PM, "Reid Pinchback" > wrote: > > Message from External Sender > > Hi Sergio, > > Assuming nobody is actually mounting a SYN flood attack, then this > sounds like you're either being hammered with connection requests in very > short periods of time, or your TCP backlog tuning is off. At least, > that's where I'd start looking. If you take that log message and google = it > (Possible SYN flooding... Sending cookies") you'll find explanations. Or > just googling "TCP backlog tuning". > > R > > > On 10/30/19, 3:29 PM, "Sergio Bilello" > wrote: > > > > >Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 > kernel: TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sendin= g > cookies. Check SNMP counters. > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org > For additional commands, e-mail: user-help@cassandra.apache.org > > > --000000000000ef5ea605962790a9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Reid,

I don't have anymore this loading prob= lem.
I solved by changing the Cassandra Driver Configuration.
Now my= cluster is pretty stable and I don't have machines with crazy CPU Load= .
The only thing not urgent but I need to investigate is the number of E= STABLISHED TCP connections. I see just one node having 7K TCP connections E= STABLISHED while the others are having around 4-6K connection opened. So th= e newest nodes added into the cluster have a higher number of ESTABLISHED T= CP connections.

default['cassandra']['sysctl'] =3D {=
'net.ipv4.tcp_keepalive_time' =3D> 60,
'net.ipv4.t= cp_keepalive_probes' =3D> 3,
'net.ipv4.tcp_keepalive_intvl&= #39; =3D> 10,
'net.core.rmem_max' =3D> 16777216,
'= ;net.core.wmem_max' =3D> 16777216,
'net.core.rmem_default= 9; =3D> 16777216,
'net.core.wmem_default' =3D> 16777216,<= br> 'net.core.optmem_max' =3D> 40960,
'net.ipv4.tcp_rmem= ' =3D> '4096 87380 16777216',
'net.ipv4.tcp_wmem'= ; =3D> '4096 65536 16777216',
'net.ipv4.ip_local_port_ra= nge' =3D> '10000 65535',
'net.ipv4.tcp_window_scalin= g' =3D> 1,
=C2=A0 'net.core.netdev_max_backlog' =3D> 2= 500,
=C2=A0 'net.core.somaxconn' =3D> 65000,
'vm.max_= map_count' =3D> 1048575,
'vm.swappiness' =3D> 0
}<= br>
These are my tweaked value and I used the values recommended from da= tastax.

Do you have something different?

Best,
Sergio
<= /div>
I= l giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback <rpinchback@tripadvisor.com> ha scritto= :
Oh nvm, didn&#= 39;t see the later msg about just posting what your fix was.

R


=EF=BB=BFOn 10/30/19, 4:24 PM, "Reid Pinchback" <rpinchback@tripadvisor.c= om> wrote:

=C2=A0 =C2=A0 =C2=A0Message from External Sender

=C2=A0 =C2=A0 Hi Sergio,

=C2=A0 =C2=A0 Assuming nobody is actually mounting a SYN flood attack, then= this sounds like you're either being hammered with connection requests= in very short periods of time, or your TCP backlog tuning is off.=C2=A0 = =C2=A0At least, that's where I'd start looking.=C2=A0 If you take t= hat log message and google it (Possible SYN flooding... Sending cookies&quo= t;) you'll find explanations.=C2=A0 Or just googling "TCP backlog = tuning".

=C2=A0 =C2=A0 R


=C2=A0 =C2=A0 On 10/30/19, 3:29 PM, "Sergio Bilello" <lapostadisergio@gma= il.com> wrote:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 >
=C2=A0 =C2=A0 =C2=A0 =C2=A0 >Oct 17 00:23:03 prod-personalization-live-d= ata-cassandra-08 kernel: TCP: request_sock_TCP: Possible SYN flooding on po= rt 9042. Sending cookies. Check SNMP counters.




=C2=A0 =C2=A0 -------------------------------------------------------------= --------
=C2=A0 =C2=A0 To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org=
=C2=A0 =C2=A0 For additional commands, e-mail: user-help@cassandra.apache.org<= br>

--000000000000ef5ea605962790a9--