From user-return-61853-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Fri Aug 3 15:58:53 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 317E0180647 for ; Fri, 3 Aug 2018 15:58:53 +0200 (CEST) Received: (qmail 58968 invoked by uid 500); 3 Aug 2018 13:58:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 58958 invoked by uid 99); 3 Aug 2018 13:58:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Aug 2018 13:58:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 1AD94181284 for ; Fri, 3 Aug 2018 13:58:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.112 X-Spam-Level: X-Spam-Status: No, score=-0.112 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Y9bH33lNjo43 for ; Fri, 3 Aug 2018 13:58:50 +0000 (UTC) Received: from mail-pl0-f48.google.com (mail-pl0-f48.google.com [209.85.160.48]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id C43CC5F118 for ; Fri, 3 Aug 2018 13:58:49 +0000 (UTC) Received: by mail-pl0-f48.google.com with SMTP id s17-v6so2599470plp.7 for ; Fri, 03 Aug 2018 06:58:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:date:subject:message-id :references:in-reply-to:to; bh=0rWU3dGYBc3olCTceIyAZx+b4elTAQNPCpGyN3zHSkE=; b=EJMlzniFnJBpm7m6AByodw43HHsZOi8zlxMjTRIogXmR0gTIOBxq9T15v1CuqxF+HP 6kh5j304jKve3BazVDiRf84TImSpC44MsQ8Ybtr/5ZsI1Lf6dGZisMuGpvjuUMJ1x+PM R9/71VheM5Zy9AQJ1tm0VHGVw6cEIREbUUfhYT8UQb/EGnKOzS1lJ/VCqcQCDdxMgT5q bYiTkwzT0IdMcFPWTvV6UIoy5Ir2FCMmjnAuziUUfRn1BdXi3nB754JCfxdd5AuB0vRr nh0MFMsiFkT7nqsT0V8nQ6X/b3/CzpiN6X4o36F6dY9KYHvSH4CUUaqXwfKkJ47SXVGe LoYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version:date :subject:message-id:references:in-reply-to:to; bh=0rWU3dGYBc3olCTceIyAZx+b4elTAQNPCpGyN3zHSkE=; b=O0Rb+OdGxvSX65byKgJHYpW78C4B7XXGjb1N+T3m1Lp1TvKYM6LlFsq5x8XtGkkxWC aD5GAJBQDn3BOzdTiBzMCIzOvLhCVWGajLJJJinyA8ATRV+2pVTUDpwDFBAQmZSHcH8U 88JVfixWxQ/jfwlrgyaVgcJ+Kgc2OjncYK7ji2Ie1K84ULyPf0m2L3RtXHi1gRp5VN7V uxtdBC0G1IiKIkZeSnIrKDIW+n5r5KZ6c4YozfpLrjOz4P5QVyT11dHwAmuxiyBhNuUR nYWzJY/64Z6BQqKFBWEo6blaAIfwd/v3jfvAqY10D5A8hclkJY4cNAjFDjag2WKwML2R 8PZQ== X-Gm-Message-State: AOUpUlFLfLzY0l71sWNRKCIWeRzulJU8nFrzUTxuyvBD9rjtufsFX4kZ 25ZBb1TRXMH4fVeUcpeIN5oViPYz X-Google-Smtp-Source: AAOMgpfOu5Wml1WLNMJ2/0TbO4Ov7rO4aMYQoR3wzdxl5QNFMkO4nASy9m9Y5t4V3st8xGZi4ywFDA== X-Received: by 2002:a17:902:6b:: with SMTP id 98-v6mr3666758pla.68.1533304728783; Fri, 03 Aug 2018 06:58:48 -0700 (PDT) Received: from [192.168.0.49] (97-113-99-36.tukw.qwest.net. [97.113.99.36]) by smtp.gmail.com with ESMTPSA id a191-v6sm8994513pge.45.2018.08.03.06.58.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 03 Aug 2018 06:58:48 -0700 (PDT) From: Jeff Jirsa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Date: Fri, 3 Aug 2018 06:58:46 -0700 Subject: Re: Cassandra rate dropping over long term test Message-Id: References: In-Reply-To: To: user@cassandra.apache.org X-Mailer: iPhone Mail (15G77) Probably Compaction Cassandra data files are immutable The write path first appends to a commitlog, then puts data into the memtabl= e. When the memtable hits a threshold, it=E2=80=99s flushed to data files on= disk (let=E2=80=99s call the first one =E2=80=9C1=E2=80=9D, second =E2=80=9C= 2=E2=80=9D and so on) Over time we build up multiple data files on disk - when Cassandra reads, it= will merge data in those files to give you the result you expect, choosing t= he latest value for each column But it=E2=80=99s usually wasteful to lots of files around, and that merging i= s expensive, so compaction combines those data files behind the scenes in a b= ackground thread. By default they=E2=80=99re combined when 4 or more files are approximately t= he same size, so if your write rate is such that you fill and flush the memt= able every 5 minutes, compaction will likely happen at least every 20 minute= s (sometimes more). This is called size tiered compaction; there are 4 strat= egies but size tiered is default and easiest to understand. You=E2=80=99re seeing mostly writes because the reads are likely in page cac= he (the kernel doesn=E2=80=99t need to go to disk to read the files, it=E2=80= =99s got them in memory for serving normal reads). --=20 Jeff Jirsa > On Aug 3, 2018, at 12:30 AM, Mihai Stanescu wro= te: >=20 > Hi all, >=20 > I am perftesting cassandra over a longrun in a cluster of 8 nodes and i no= ticed the rate of service drops.=20 > Most of the nodes have the CPU between 40-65% however one of the nodes has= a higher CPU and also started performing a lot of read IOPS as seen in the i= mage. (green is read IOPS) >=20 > My test has a mixed rw scenario. > 1. insert row > 2. after 60 seconds read row > 3. delete row.=20 >=20 > The rate of inserts is bigger than the rate of deletes so some delete will= not happen. >=20 > I have checked the client it it does not accumulate RAM, GC is a straight l= ine so o don't understand whats going on. >=20 > Any hints? >=20 > Regards, > MIhai >=20 > >=20 >=20 --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org For additional commands, e-mail: user-help@cassandra.apache.org