cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Pederson <eric...@gmail.com>
Subject Bottleneck for small inserts?
Date Mon, 22 May 2017 21:02:29 GMT
Hi all:

I'm new to Cassandra and I'm doing some performance testing.  One of things
that I'm testing is ingestion throughput.   My server setup is:

   - 3 node cluster
   - SSD data (both commit log and sstables are on the same disk)
   - 64 GB RAM per server
   - 48 cores per server
   - Cassandra 3.0.11
   - 48 Gb heap using G1GC
   - 1 Gbps NICs

Since I'm using SSD I've tried tuning the following (one at a time) but
none seemed to make a lot of difference:

   - concurrent_writes=384
   - memtable_flush_writers=8
   - concurrent_compactors=8

I am currently doing ingestion tests sending data from 3 clients on the
same subnet.  I am using cassandra-stress to do some ingestion testing.
The tests are using CL=ONE and RF=2.

Using cassandra-stress (3.10) I am able to saturate the disk using a large
enough column size and the standard five column cassandra-stress schema.
For example, -col size=fixed(400) will saturate the disk and compactions
will start falling behind.

One of our main tables has a row size that approximately 200 bytes, across
64 columns.  When ingesting this table I don't see any resource
saturation.  Disk utilization is around 10-15% per iostat.  Incoming
network traffic on the servers is around 100-300 Mbps.  CPU utilization is
around 20-70%.  nodetool tpstats shows mostly zeros with occasional spikes
around 500 in MutationStage.

The stress run does 10,000,000 inserts per client, each with a separate
range of partition IDs.  The run with 200 byte rows takes about 4 minutes,
with mean Latency 4.5ms, Total GC time of 21 secs, Avg GC time 173 ms.

The overall performance is good - around 120k rows/sec ingested.  But I'm
curious to know where the bottleneck is.  There's no resource saturation and
nodetool tpstats shows only occasional brief queueing.  Is the rest just
expected latency inside of Cassandra?

Thanks,

-- Eric

Mime
View raw message