cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliott Sims <elli...@backblaze.com>
Subject Re: Cassandra Recommended System Settings
Date Mon, 21 Oct 2019 20:31:06 GMT
Based on my experiences, if you have a new enough kernel I'd strongly
suggest switching the TCP scheduler algorithm to BBR.  I've found the rest
tend to be extremely sensitive to even small amounts of packet loss among
cluster members where BBR holds up well.

High ulimits for basically everything are probably a good idea, although
"unlimited" may not be purely optimal for all cases.
The TCP keepalive settings are probably only necessary for traffic
buggy/misconfigured firewalls, but shouldn't really do any harm on a modern
fast network.

The TCP memory settings are pretty aggressive and probably result in
unnecessary RAM usage.
The net.core.rmem_default/net.core.wmem_default settings are overridden by
the TCP-specific settings as far as I know, so they're not really
relevant/helpful for Cassandra
The net.ipv4.tcp_rmem/net.ipv4.tcp_wmem max settings are pretty
aggressive.  That works out to something like 1Gbps with 130ms latency per
TCP connection, but on a local LAN with latencies <1ms it's enough buffer
for over 100Gbps per TCP session.  A much smaller value will probably make
more sense for most setups.


On Mon, Oct 21, 2019 at 10:21 AM Sergio <lapostadisergio@gmail.com> wrote:

>
> Hello!
>
> This is the kernel that I am using
> Linux  4.16.13-1.el7.elrepo.x86_64 #1 SMP Wed May 30 14:31:51 EDT 2018
> x86_64 x86_64 x86_64 GNU/Linux
>
> Best,
>
> Sergio
>
> Il giorno lun 21 ott 2019 alle ore 07:30 Reid Pinchback <
> rpinchback@tripadvisor.com> ha scritto:
>
>> I don't know which distro and version you are using, but watch out for
>> surprises in what vm.swappiness=0 means.  In older kernels it means "only
>> use swap when desperate".  I believe that newer kernels changed to have 1
>> mean that, and 0 means to always use the oomkiller.  Neither situation is
>> strictly good or bad, what matters is what you intend the system behavior
>> to be in comparison with whatever monitoring/alerting you have put in place.
>>
>> R
>>
>>
>> ´╗┐On 10/18/19, 9:04 PM, "Sergio Bilello" <lapostadisergio@gmail.com>
>> wrote:
>>
>>      Message from External Sender
>>
>>     Hello everyone!
>>
>>
>>
>>     Do you have any setting that you would change or tweak from the below
>> list?
>>
>>
>>
>>     sudo cat /proc/4379/limits
>>
>>     Limit                     Soft Limit           Hard Limit
>>  Units
>>
>>     Max cpu time              unlimited            unlimited
>> seconds
>>
>>     Max file size             unlimited            unlimited
>> bytes
>>
>>     Max data size             unlimited            unlimited
>> bytes
>>
>>     Max stack size            unlimited            unlimited
>> bytes
>>
>>     Max core file size        unlimited            unlimited
>> bytes
>>
>>     Max resident set          unlimited            unlimited
>> bytes
>>
>>     Max processes             32768                32768
>> processes
>>
>>     Max open files            1048576              1048576
>> files
>>
>>     Max locked memory         unlimited            unlimited
>> bytes
>>
>>     Max address space         unlimited            unlimited
>> bytes
>>
>>     Max file locks            unlimited            unlimited
>> locks
>>
>>     Max pending signals       unlimited            unlimited
>> signals
>>
>>     Max msgqueue size         unlimited            unlimited
>> bytes
>>
>>     Max nice priority         0                    0
>>
>>     Max realtime priority     0                    0
>>
>>     Max realtime timeout      unlimited            unlimited            us
>>
>>
>>
>>     These are the sysctl settings
>>
>>     default['cassandra']['sysctl'] = {
>>
>>         'net.ipv4.tcp_keepalive_time' => 60,
>>
>>         'net.ipv4.tcp_keepalive_probes' => 3,
>>
>>         'net.ipv4.tcp_keepalive_intvl' => 10,
>>
>>         'net.core.rmem_max' => 16777216,
>>
>>         'net.core.wmem_max' => 16777216,
>>
>>         'net.core.rmem_default' => 16777216,
>>
>>         'net.core.wmem_default' => 16777216,
>>
>>         'net.core.optmem_max' => 40960,
>>
>>         'net.ipv4.tcp_rmem' => '4096 87380 16777216',
>>
>>         'net.ipv4.tcp_wmem' => '4096 65536 16777216',
>>
>>         'net.ipv4.ip_local_port_range' => '10000 65535',
>>
>>         'net.ipv4.tcp_window_scaling' => 1,
>>
>>            'net.core.netdev_max_backlog' => 2500,
>>
>>            'net.core.somaxconn' => 65000,
>>
>>         'vm.max_map_count' => 1048575,
>>
>>         'vm.swappiness' => 0
>>
>>     }
>>
>>
>>
>>     Am I missing something else?
>>
>>
>>
>>     Do you have any experience to configure CENTOS 7
>>
>>     for
>>
>>     JAVA HUGE PAGES
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html-23CheckJavaHugepagessettings&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=b6lGdbtv1SN9opBsIOFRT6IX6BroMW-8Tudk9qEh3bI&e=
>>
>>
>>
>>     OPTIMIZE SSD
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html-23OptimizeSSDs&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=c0S3S3V_0YHVMx2I-pyOh24MiQs1D-L73JytaSw648M&e=
>>
>>
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_dse_5.1_dse-2Dadmin_datastax-5Fenterprise_config_configRecommendedSettings.html&d=DwIBaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=zke-WpkD1c6Qt1cz8mJG0ZQ37h8kezqknMSnerQhXuU&s=PZFG6SXF6dL5LRJ-aUoidHnnLGpKPbpxdKstM8M9JMk&e=
>>
>>
>>
>>     We are using AWS i3.xlarge instances
>>
>>
>>
>>     Thanks,
>>
>>
>>
>>     Sergio
>>
>>
>>
>>     ---------------------------------------------------------------------
>>
>>     To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>
>>     For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>>
>>
>>
>>

Mime
View raw message