Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 12A22200C7E for ; Tue, 23 May 2017 10:23:50 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 11348160BC3; Tue, 23 May 2017 08:23:50 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 30867160BB6 for ; Tue, 23 May 2017 10:23:49 +0200 (CEST) Received: (qmail 56670 invoked by uid 500); 23 May 2017 08:23:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 56651 invoked by uid 99); 23 May 2017 08:23:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 May 2017 08:23:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 2BBBC18FC18 for ; Tue, 23 May 2017 08:23:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.203 X-Spam-Level: X-Spam-Status: No, score=0.203 tagged_above=-999 required=6.31 tests=[FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.796, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id GyYT7tLbDQq7 for ; Tue, 23 May 2017 08:23:45 +0000 (UTC) Received: from sapo.pt (relay3.ptmail.sapo.pt [212.55.154.23]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 748695F20C for ; Tue, 23 May 2017 08:23:44 +0000 (UTC) Received: (qmail 13767 invoked from network); 23 May 2017 08:23:36 -0000 Received: (qmail 31343 invoked from network); 23 May 2017 08:23:36 -0000 Received: from unknown (HELO [10.0.15.42]) (cogumelosmaravilha@sapo.pt@[193.126.81.194]) (envelope-sender ) by ptmail-mta-auth02 (qmail-ptmail-1.0.0) with ESMTPA for ; 23 May 2017 08:23:36 -0000 X-PTMail-RemoteIP: 193.126.81.194 X-PTMail-AllowedSender-Action: X-PTMail-Service: default Subject: Re: Bottleneck for small inserts? To: user@cassandra.apache.org References: From: Cogumelos Maravilha Message-ID: <488d76cf-c833-1be5-609c-97901ac54165@sapo.pt> Date: Tue, 23 May 2017 09:23:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------A258C555A9B258E159F6B3AA" Content-Language: en-US archived-at: Tue, 23 May 2017 08:23:50 -0000 --------------A258C555A9B258E159F6B3AA Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Change to *|durable_writes = false|* And please post the results. Thanks. On 05/22/2017 10:08 PM, Jonathan Haddad wrote: > How many CPUs are you using for interrupts? > http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux > > Have you tried making a flame graph to see where Cassandra is spending > its > time? http://www.brendangregg.com/blog/2014-06-12/java-flame-graphs.html > > Are you tracking GC pauses? > > Jon > > On Mon, May 22, 2017 at 2:03 PM Eric Pederson > wrote: > > Hi all: > > I'm new to Cassandra and I'm doing some performance testing. One > of things that I'm testing is ingestion throughput. My server > setup is: > > * 3 node cluster > * SSD data (both commit log and sstables are on the same disk) > * 64 GB RAM per server > * 48 cores per server > * Cassandra 3.0.11 > * 48 Gb heap using G1GC > * 1 Gbps NICs > > Since I'm using SSD I've tried tuning the following (one at a > time) but none seemed to make a lot of difference: > > * concurrent_writes=384 > * memtable_flush_writers=8 > * concurrent_compactors=8 > > I am currently doing ingestion tests sending data from 3 clients > on the same subnet. I am using cassandra-stress to do some > ingestion testing. The tests are using CL=ONE and RF=2. > > Using cassandra-stress (3.10) I am able to saturate the disk using > a large enough column size and the standard five column > cassandra-stress schema. For example, -col size=fixed(400) will > saturate the disk and compactions will start falling behind. > > One of our main tables has a row size that approximately 200 > bytes, across 64 columns. When ingesting this table I don't see > any resource saturation. Disk utilization is around 10-15% per > iostat. Incoming network traffic on the servers is around 100-300 > Mbps. CPU utilization is around 20-70%. nodetool tpstats shows > mostly zeros with occasional spikes around 500 in MutationStage. > > The stress run does 10,000,000 inserts per client, each with a > separate range of partition IDs. The run with 200 byte rows takes > about 4 minutes, with mean Latency 4.5ms, Total GC time of 21 > secs, Avg GC time 173 ms. > > The overall performance is good - around 120k rows/sec ingested. > But I'm curious to know where the bottleneck is. There's no > resource saturation andnodetool tpstats shows only occasional > brief queueing. Is the rest just expected latency inside of > Cassandra? > > Thanks, > > -- Eric > --------------A258C555A9B258E159F6B3AA Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

Hi,

Change to durable_writes = false

And please post the results.

Thanks.

On 05/22/2017 10:08 PM, Jonathan Haddad wrote:

Have you tried making a flame graph to see where Cassandra is spending its time? http://www.brendangregg.com/blog/2014-06-12/java-flame-graphs.html

Are you tracking GC pauses?

Jon

On Mon, May 22, 2017 at 2:03 PM Eric Pederson <ericacm@gmail.com> wrote:
Hi all:

I'm new to Cassandra and I'm doing some performance testing.  One of things that I'm testing is ingestion throughput.   My server setup is:
  • 3 node cluster
  • SSD data (both commit log and sstables are on the same disk)
  • 64 GB RAM per server
  • 48 cores per server
  • Cassandra 3.0.11
  • 48 Gb heap using G1GC
  • 1 Gbps NICs
Since I'm using SSD I've tried tuning the following (one at a time) but none seemed to make a lot of difference:
  • concurrent_writes=384
  • memtable_flush_writers=8
  • concurrent_compactors=8
I am currently doing ingestion tests sending data from 3 clients on the same subnet.  I am using cassandra-stress to do some ingestion testing.  The tests are using CL=ONE and RF=2.

Using cassandra-stress (3.10) I am able to saturate the disk using a large enough column size and the standard five column cassandra-stress schema.  For example, -col size=fixed(400) will saturate the disk and compactions will start falling behind. 

One of our main tables has a row size that approximately 200 bytes, across 64 columns.  When ingesting this table I don't see any resource saturation.  Disk utilization is around 10-15% per iostat.  Incoming network traffic on the servers is around 100-300 Mbps.  CPU utilization is around 20-70%.  nodetool tpstats shows mostly zeros with occasional spikes around 500 in MutationStage.  

The stress run does 10,000,000 inserts per client, each with a separate range of partition IDs.  The run with 200 byte rows takes about 4 minutes, with mean Latency 4.5ms, Total GC time of 21 secs, Avg GC time 173 ms.   

The overall performance is good - around 120k rows/sec ingested.  But I'm curious to know where the bottleneck is.  There's no resource saturation and nodetool tpstats shows only occasional brief queueing.  Is the rest just expected latency inside of Cassandra?

Thanks,

-- Eric

--------------A258C555A9B258E159F6B3AA--