incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Config changes to leverage new hardware
Date Thu, 07 Nov 2013 04:33:43 GMT
> Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
You will make more use of the extra memory moving to 1.2 as it moves bloom filters and compression
data off heap. 

Also grab the TLAB setting from cassandra-env.sh in v1.2

> As of now, our performance tests (our application specific as well as cassandra-stress)
are not showing any significant difference in the hardwares, which is a little disheartening,
since the new hardware has a lot more RAM and CPU.
For reads or writes or both ? 

Writes tend to scale with cores as long as the commit log can keep up. 
Reads improve with disk IO and page cache size when the hot set is in memory. 

> Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog
and 3 disks RAID 0 for data)
> New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks (
1 disk used for commitlog and 7 disks RAID 0 for data)
Is the disk IO on the commit log volume keeping up ?
You cranked up the concurrent writers and the commit log may not keep up. You could put the
commit log on the same RAID volume to see if that improves writes. 

> The config we tried modifying so far was concurrent_reads to (16 * number of drives)
and concurrent_writes to (8 * number of cores) as per 
256 write threads is a lot. Make sure the commit log can keep up, I would put it back to 32,
maybe try 64. Not sure the concurrent list for the commit log will work well with that many
threads. 

May want to put the reads down as well. 

It’s easier to tune the system if you can provide some info on the workload. 

Cheers

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 7/11/2013, at 12:35 pm, Arindam Barua <abarua@247-inc.com> wrote:

>  
> We want to upgrade our Cassandra cluster to have newer hardware, and were wondering if
anyone has suggestions on Cassandra or linux config changes that will prove to be beneficial.
> As of now, our performance tests (our application specific as well as cassandra-stress)
are not showing any significant difference in the hardwares, which is a little disheartening,
since the new hardware has a lot more RAM and CPU.
>  
> Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for commitlog
and 3 disks RAID 0 for data)
> New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, eight 1-TB disks (
1 disk used for commitlog and 7 disks RAID 0 for data)
>  
> Most of the cassandra config currently is the default, and we are using LeveledCompaction
strategy. Default key cache, row cache turned off.
> The config we tried modifying so far was concurrent_reads to (16 * number of drives)
and concurrent_writes to (8 * number of cores) as per recommendation in cassandra.yaml, but
that didn’t make much difference.
> We were hoping that at least the extra RAM in the new hardware will be used for Linux
file caching and hence an improvement in performance will be observed.
>  
> Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 soon.
>  
> Thanks,
> Arindam


Mime
View raw message