Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1A9F3200C34 for ; Mon, 27 Feb 2017 19:31:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 190E6160B60; Mon, 27 Feb 2017 18:31:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1478D160B5B for ; Mon, 27 Feb 2017 19:31:05 +0100 (CET) Received: (qmail 38452 invoked by uid 500); 27 Feb 2017 18:31:04 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 38442 invoked by uid 99); 27 Feb 2017 18:31:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Feb 2017 18:31:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 251E4C0ABA for ; Mon, 27 Feb 2017 18:31:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.898 X-Spam-Level: * X-Spam-Status: No, score=1.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.fr Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id IJb3j7vrYrd2 for ; Mon, 27 Feb 2017 18:31:02 +0000 (UTC) Received: from nm5-vm2.bullet.mail.ir2.yahoo.com (nm5-vm2.bullet.mail.ir2.yahoo.com [212.82.96.109]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 38E8B5FE16 for ; Mon, 27 Feb 2017 18:31:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.fr; s=s2048; t=1488220255; bh=sHdXR9OJnFMwxtyLMaSu9b/fvuLHXBsz0en/x6XO8EU=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=R6jtWc0KAnQywF60ftv1Eb2sioOIj0ooXwLj+E2PwpcMmDUhii9GGgaXN929mWj8jIpckWVS6vW7wthSTaORRWSMZbqGcmuR3EQ5VkFQa5fH47hJQamAztoAcpn6IB0rD6TO3XDvuXW+GVl1CogyHH+vM3lULwu/L6Kj1/It96CEQa2nwR9vLAtTvdcJ0wwXwguXMaWV1tnVwRHDNsi+A96ANsiOgDhyVOHR0KHv8qlwJN2qEUZAev4rvlmGvkQxjIm0SZj0JjXZpSMYshGH4kdnMfALj78peAaAsXE3VZzTZvihgBfoF65oU7QiqmY5SKoLkZGgwJAUuFlRlved7Q== Received: from [212.82.98.49] by nm5.bullet.mail.ir2.yahoo.com with NNFMP; 27 Feb 2017 18:30:55 -0000 Received: from [212.82.98.76] by tm2.bullet.mail.ir2.yahoo.com with NNFMP; 27 Feb 2017 18:30:55 -0000 Received: from [127.0.0.1] by omp1013.mail.ir2.yahoo.com with NNFMP; 27 Feb 2017 18:30:55 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 141447.51482.bm@omp1013.mail.ir2.yahoo.com X-YMail-OSG: .5nZ2hkVM1mDiccFtA6ILNLvgg7UuXmPc5F5Tc8UGKmvlvBx1XhJzZUgj.PXHeT zz.SmBZnZOaGFUbgCvhuJyTsmuu3ZUyIUfOJnlvM_Cm7QZE0PYm7TDxa_JIv_8Raz6vpV3q5ZqYk opTKusY4.QVOqLf9dM9DOofdhSshyXucgm5HUpdMnDvLmf3nwvDkwI0q69qAUFIVTFSrZDdA_S3i BSELTvD4tFAyLQZDITJAiVvs.v40TILth.7Wgkl5ByQDFM7wjMisAHnZTZehQda1OH_BqUbQjR_G Zh_CK5_lCD22hjf3TDo0_yhPS68JdcF8osw2GxkYXLQgAuxAVyIkjmSYrZrXUmo_u53TqaK68d4V IWO3OulNI6qMWY0.NXsfGX0a1ZFfn9mypPblBY7YqFl0gT7QkEnaTEkcXI19WNnfL.JBawtKSPRF viAwV779aGI6HAwH0siTO9G0p_R5tL_kLD5nEqS9.ZCeEissB2NzggyieQm5juH7HbYX.jqKnSlQ- Received: from jws700025.mail.ir2.yahoo.com by sendmailws165.mail.ir2.yahoo.com; Mon, 27 Feb 2017 18:30:54 +0000; 1488220254.613 Date: Mon, 27 Feb 2017 18:30:54 +0000 (UTC) From: Romain Hardouin Reply-To: Romain Hardouin To: "user@cassandra.apache.org" Message-ID: <667554380.1203401.1488220254261@mail.yahoo.com> In-Reply-To: References: Subject: Re: Attached profiled data but need help understanding it MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1203400_1105598368.1488220254258" archived-at: Mon, 27 Feb 2017 18:31:07 -0000 ------=_Part_1203400_1105598368.1488220254258 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, Regarding shared pool workers see CASSANDRA-11966. You may have to backport= it depending on your Cassandra version.=C2=A0 Did you try to lower compaction throughput to see if it helps? Be sure to k= eep an eye on pending compactions, SSTables count and=C2=A0SSTable per read= =C2=A0of course. "alloc" is the memory allocation rate. You can see that compactions are GC = intensive. You won't be able to achieve impressive writes/s with LWT. But maybe there = is room for improvement. Try GC tuning, different pool sizes for NTR, concu= rrent reads and writes, compaction executors, etc. Also check if you can im= prove network latency (e.g. VF or ENA on AWS). What LWT rate would you want to achieve? Best, Romain =20 Le Lundi 27 f=C3=A9vrier 2017 12h48, Kant Kodali a = =C3=A9crit : =20 Also Attached is a flamed graph generated from a thread dump. On Mon, Feb 27, 2017 at 2:32 AM, Kant Kodali wrote: Hi, Attached are the stats of my Cassandra node running on a 4-core CPU. I am u= sing sjk-plus tool for the first time so what are the things I should watch= ed out for in my attached screenshot? I can see the CPU is almost maxed out= but should I say that is because of compaction or shared-worker-pool threa= ds (which btw, I dont know what they are doing perhaps I need to take threa= dump)? Also what is alloc for each thread?=C2=A0 I have a insert heavy workload (almost like an ingest running against cassa= ndra cluster) and in my case all writes are LWT. The current throughput is 1500 writes/sec where each write is about 1KB. Ho= w can I tune something for a higher throughput? Any pointers or suggestions= would help. Thanks much,kant =20 ------=_Part_1203400_1105598368.1488220254258 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

<= /font>
Regardi= ng shared pool workers see CASSANDRA-11966. You may have to backport= it depending on your Cassandra version. 

Did you try to lower compaction throughput to see if it h= elps? Be sure to keep an eye on pending compactions, SSTables count and SSTable per read of course.

"alloc" is the me= mory allocation rate. You can see that compactions are GC intensive.=

You won't be able to achieve impres= sive writes/s with LWT. But maybe there is room for improvement. Try GC tun= ing, different pool sizes for NTR, concurrent reads and writes, compaction = executors, etc. Also check if you can improve network latency (e.g. VF or E= NA on AWS).

What LWT rate would you want to = achieve?

Best,

Romain
=

------=_Part_1203400_1105598368.1488220254258--