cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reid Pinchback <rpinchb...@tripadvisor.com>
Subject Re: Cassandra Recommended System Settings
Date Mon, 21 Oct 2019 21:03:59 GMT
Sergio, if you do some online searching about ‘bufferbloat’ in networking, you’ll find
the background to help explain what motivates networking changes.  Actual investigation of
network performance can get a bit gnarly.  The TL;DR summary is that big buffers function
like big queues, and thus attempts to speed up throughput can cause things stuck in a queue
to have higher latency.  With very fast networks, there isn’t as much need to have big buffers.
 Imagine having a coordinator node waiting to respond to a query but can’t because a bunch
of merkel trees are sitting in the tcp buffer waiting to be sent out.  Sometimes total latency
doesn’t fairly measure actual effort to do the work, some of that can be time spent sitting
waiting in the buffer to be shipped out back to the client.

From: Sergio <lapostadisergio@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Monday, October 21, 2019 at 4:54 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Cassandra Recommended System Settings

Message from External Sender
Thanks Elliott!

How do you know if there is too much RAM used for those settings?

Which metrics do you keep track of?

What would you recommend instead?

Best,

Sergio

On Mon, Oct 21, 2019, 1:41 PM Elliott Sims <elliott@backblaze.com<mailto:elliott@backblaze.com>>
wrote:
Based on my experiences, if you have a new enough kernel I'd strongly suggest switching the
TCP scheduler algorithm to BBR.  I've found the rest tend to be extremely sensitive to even
small amounts of packet loss among cluster members where BBR holds up well.
High ulimits for basically everything are probably a good idea, although "unlimited" may not
be purely optimal for all cases.
The TCP keepalive settings are probably only necessary for traffic buggy/misconfigured firewalls,
but shouldn't really do any harm on a modern fast network.
The TCP memory settings are pretty aggressive and probably result in unnecessary RAM usage.
The net.core.rmem_default/net.core.wmem_default settings are overridden by the TCP-specific
settings as far as I know, so they're not really relevant/helpful for Cassandra
The net.ipv4.tcp_rmem/net.ipv4.tcp_wmem max settings are pretty aggressive.  That works out
to something like 1Gbps with 130ms latency per TCP connection, but on a local LAN with latencies
<1ms it's enough buffer for over 100Gbps per TCP session.  A much smaller value will probably
make more sense for most setups.


On Mon, Oct 21, 2019 at 10:21 AM Sergio <lapostadisergio@gmail.com<mailto:lapostadisergio@gmail.com>>
wrote:

Hello!

This is the kernel that I am using
Linux  4.16.13-1.el7.elrepo.x86_64 #1 SMP Wed May 30 14:31:51 EDT 2018 x86_64 x86_64 x86_64
GNU/Linux

Best,

Sergio

Il giorno lun 21 ott 2019 alle ore 07:30 Reid Pinchback <rpinchback@tripadvisor.com<mailto:rpinchback@tripadvisor.com>>
ha scritto:
I don't know which distro and version you are using, but watch out for surprises in what vm.swappiness=0
means.  In older kernels it means "only use swap when desperate".  I believe that newer kernels
changed to have 1 mean that, and 0 means to always use the oomkiller.  Neither situation is
strictly good or bad, what matters is what you intend the system behavior to be in comparison
with whatever monitoring/alerting you have put in place.

R


On 10/18/19, 9:04 PM, "Sergio Bilello" <lapostadisergio@gmail.com<mailto:lapostadisergio@gmail.com>>
wrote:

     Message from External Sender

    Hello everyone!



    Do you have any setting that you would change or tweak from the below list?



    sudo cat /proc/4379/limits

    Limit                     Soft Limit           Hard Limit           Units

    Max cpu time              unlimited            unlimited            seconds

    Max file size             unlimited            unlimited            bytes

    Max data size             unlimited            unlimited            bytes

    Max stack size            unlimited            unlimited            bytes

    Max core file size        unlimited            unlimited            bytes

    Max resident set          unlimited            unlimited            bytes

    Max processes             32768                32768                processes

    Max open files            1048576              1048576              files

    Max locked memory         unlimited            unlimited            bytes

    Max address space         unlimited            unlimited            bytes

    Max file locks            unlimited            unlimited            locks

    Max pending signals       unlimited            unlimited            signals

    Max msgqueue size         unlimited            unlimited            bytes

    Max nice priority         0                    0

    Max realtime priority     0                    0

    Max realtime timeout      unlimited            unlimited            us



    These are the sysctl settings

    default['cassandra']['sysctl'] = {

        'net.ipv4.tcp_keepalive_time' => 60,

        'net.ipv4.tcp_keepalive_probes' => 3,

        'net.ipv4.tcp_keepalive_intvl' => 10,

        'net.core.rmem_max' => 16777216,

        'net.core.wmem_max' => 16777216,

        'net.core.rmem_default' => 16777216,

        'net.core.wmem_default' => 16777216,

        'net.core.optmem_max' => 40960,

        'net.ipv4.tcp_rmem' => '4096 87380 16777216',

        'net.ipv4.tcp_wmem' => '4096 65536 16777216',

        'net.ipv4.ip_local_port_range' => '10000 65535',

        'net.ipv4.tcp_window_scaling' => 1,

           'net.core.netdev_max_backlog' => 2500,

           'net.core.somaxconn' => 65000,

        'vm.max_map_count' => 1048575,

        'vm.swappiness' => 0

    }



    Am I missing something else?



    Do you have any experience to configure CENTOS 7

    for

    JAVA HUGE PAGES

    https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html-23CheckJavaHugepagessettings&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=b6lGdbtv1SN9opBsIOFRT6IX6BroMW-8Tudk9qEh3bI&e=



    OPTIMIZE SSD

    https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html-23OptimizeSSDs&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=c0S3S3V_0YHVMx2I-pyOh24MiQs1D-L73JytaSw648M&e=



    https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=PZFG6SXF6dL5LRJ-aUoidHnnLGpKPbpxdKstM8M9JMk&e=



    We are using AWS i3.xlarge instances



    Thanks,



    Sergio



    ---------------------------------------------------------------------

    To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<mailto:user-unsubscribe@cassandra.apache.org>

    For additional commands, e-mail: user-help@cassandra.apache.org<mailto:user-help@cassandra.apache.org>




Mime
View raw message