From user-return-62191-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Fri Sep 7 05:36:22 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9445F180668 for ; Fri, 7 Sep 2018 05:36:21 +0200 (CEST) Received: (qmail 75648 invoked by uid 500); 7 Sep 2018 03:36:19 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 75637 invoked by uid 99); 7 Sep 2018 03:36:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Sep 2018 03:36:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A2C28C0C3A for ; Fri, 7 Sep 2018 03:36:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.899 X-Spam-Level: * X-Spam-Status: No, score=1.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=backblaze.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id geRsQxmcxSHM for ; Fri, 7 Sep 2018 03:36:17 +0000 (UTC) Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id EC8145F4A9 for ; Fri, 7 Sep 2018 03:36:16 +0000 (UTC) Received: by mail-qk1-f178.google.com with SMTP id g13-v6so8868595qki.9 for ; Thu, 06 Sep 2018 20:36:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=backblaze.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=hbFrp15oimdUnNppZNWvIrEj6+tL/5mOw8830HJfMa8=; b=HQmBCNBvZkfd5DP3lU3W1kEYEGpWFrNzZiEEUYMizS97AycoU5fOVytCxoQFeY7wHJ z/EcVjUw4qTaOt3UWqcLRlFxKSHavox7+sA0BT+5VtWeXe0YB0s8ItXLNsW6Z2xoGBOj efay+TycadVWGUdTJZErK5HlMVeBm1c1gI4TwwTf11TZ5fFjCPbacis8dyYLf/5TEiG4 7CbXEenB09bAsyql1jpLGDZxa+4e1Wo8RcO9L83ZwodmFvwMbZCQE1ODMAaaC/iOyW0U MPT3pLpcVWtnlWhoeJ6FiNKV3KjW5plPcRl7akR5yivPMXryzkAb79ZUm/t1IBdTxJZt uTtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=hbFrp15oimdUnNppZNWvIrEj6+tL/5mOw8830HJfMa8=; b=clU+WPMCyCpfSyDHex2kcWNirXIgduQuvlpIe4McD2oIjt9uAMWz4HPZLBFBXkhCVd sIEi0zDB7XeVVM6Gn7y5d7ImXGn0qTnTG22KzimzcOETQyTNuqoYxw5FczdP7k8BZ08T QO8bs3V0hhx9kwF+VZNc96uy9A55zHbY+AMAAHUnyL/H+OgQLdSzUCEoBY0tlUG0uhiY 16pb648m0hN+ZEmcQJyH1fBJX5FvFXboJJVd8ekBOdSzeND6NSwuSZgUgQlM+Ggck37X tIJ0JBm5ZQDn9rvLE+63p2FxTNshmAbFEHZlQ5cQN0SQBgDToLI5Z7tKMmq2tM08POkL gbpg== X-Gm-Message-State: APzg51DvsAdodkFQ5vVtB9UkovURogWvend4BaZJ5JV4MSfdYfFN7E7j 6DhfSnIGBsi6h0gdoYjhZZN+lKDAtdO3hPZ0eMKdoT+o X-Google-Smtp-Source: ANB0VdbgvT0YE+fuXBrngxZuYcy78276xO3ag43D1M1tZ2pGhhwAoA0L/PMlZQ5ne8KzFpHBU9TfVoFTbEZfPOfO4do= X-Received: by 2002:a37:4453:: with SMTP id r80-v6mr4476102qka.79.1536291368174; Thu, 06 Sep 2018 20:36:08 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:ac8:3718:0:0:0:0:0 with HTTP; Thu, 6 Sep 2018 20:35:27 -0700 (PDT) In-Reply-To: <165ae4b5d7e.c117ee1237740.814102694178717573@zoho.com> References: <165ae4b5d7e.c117ee1237740.814102694178717573@zoho.com> From: Elliott Sims Date: Thu, 6 Sep 2018 23:35:27 -0400 Message-ID: Subject: Re: Cluster CPU usage limit To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="000000000000459e4905753fb5b0" --000000000000459e4905753fb5b0 Content-Type: text/plain; charset="UTF-8" It's interesting and a bit surprising that 256 write threads isn't enough. Even with a lot of cores, I'd expect you to be able to saturate CPU with that many threads. I'd make sure you don't have other bottlenecks, like GC, IOPs, network, or "microbursts" where your load is actually fluctuating between 20-100% CPU. Admittedly, I actually did get best results with 256 threads (and haven't tested higher, but lower is definitely not enough), but every advice I've seen is for a lower write thread count being optimal for most cases. On Thu, Sep 6, 2018 at 5:51 AM, onmstester onmstester wrote: > IMHO, Cassandra write is more of a CPU bound task, so while determining > cluster write throughput, what CPU usage percent (avg among all cluster > nodes) should be determined as limit? > Rephrase: what's the normal CPU usage in Cassandra cluster (while no > compaction, streaming or heavy-read running) ? > For a cluster with 10 nodes, i got 700K write per seconds for my data > model, average cpu load is about 40%, i'm going to increase number of > native threads (now is 256) and native queue (1024) to increase throughput > (and CPU usage subsequently). > > Sent using Zoho Mail > > > --000000000000459e4905753fb5b0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
It's interesting and a bit surprising that 256 wr= ite threads isn't enough.=C2=A0 Even with a lot of cores, I'd expec= t you to be able to saturate CPU with that many threads.=C2=A0 I'd make= sure you don't have other bottlenecks, like GC, IOPs, network, or &quo= t;microbursts" where your load is actually fluctuating between 20-100%= CPU.
Admittedly, I actually did get best results with 256 threads= (and haven't tested higher, but lower is definitely not enough), but e= very advice I've seen is for a lower write thread count being optimal f= or most cases.

On Thu, Sep 6, 2018 at 5:51 AM, onmstester onmstester <onmstester= @zoho.com> wrote:
<= div>
IMHO, Cassandra write is more of a CPU bound task, = so while determining cluster write throughput, what CPU usage percent (avg = among all cluster nodes) should be determined as limit?=C2=A0=C2=A0
Rephrase: what's the normal CPU usage in Cassandra cluster (whil= e no compaction, streaming or heavy-read running) ?
For a clu= ster with 10 nodes, i got 700K write per seconds for my data model, average= cpu load is about 40%, i'm going to increase number of native threads = (now is 256) and native queue (1024) to increase throughput (and CPU usage = subsequently).

Se= nt using Zoho Mail



<= /div>

--000000000000459e4905753fb5b0--