Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DD453200BCA for ; Mon, 7 Nov 2016 04:42:23 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id DBD5E160B0D; Mon, 7 Nov 2016 03:42:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 09743160AFC for ; Mon, 7 Nov 2016 04:42:22 +0100 (CET) Received: (qmail 61568 invoked by uid 500); 7 Nov 2016 03:42:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 61558 invoked by uid 99); 7 Nov 2016 03:42:21 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Nov 2016 03:42:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 91BDF180058 for ; Mon, 7 Nov 2016 03:42:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.989 X-Spam-Level: * X-Spam-Status: No, score=1.989 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=instaclustr-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id NcW1zcCTPTSw for ; Mon, 7 Nov 2016 03:42:16 +0000 (UTC) Received: from mail-qk0-f177.google.com (mail-qk0-f177.google.com [209.85.220.177]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 747BC5F19D for ; Mon, 7 Nov 2016 03:42:16 +0000 (UTC) Received: by mail-qk0-f177.google.com with SMTP id x190so160323534qkb.0 for ; Sun, 06 Nov 2016 19:42:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=instaclustr-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=lxatLa5xC8JvbLb9RC8ud/FX9DC2ddYnMuWifRqED4E=; b=hObDIFz5LwdiFUAdUzrurByDT87EnykUxfKgwGOw6bJdSZc2ROyBw37AXb1LiGex3b veBjrpAYG4A38E0WwGL5t0eTKNtz8fJppuvbEKl5DKKd8oCoNDwatWDEd/FrGz+nPnOk BbkhfWNoBX6F6j62ZRnrr8zQS64UcdpFPo2eH1QHuCzTo8vOHLe1PlG8b3FREPiZiGTj efQlx0XjBtLMeQfAMI787I9nAIOyHPgg5zBD3uWLGAx3Y2GrJzYY0MY8Mja4TrK0ur1K rrCADs1ZfGRwLhcAOW5BQnNeU8iRolqTic+Xw9Xr/aLdg7Z09GZAuPRUp6Afk7uy1E06 fTtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=lxatLa5xC8JvbLb9RC8ud/FX9DC2ddYnMuWifRqED4E=; b=nCLZpktW87WqsvCkviEgsUtaUVHLdVcpo6HAqkopZWHJfye5q7iPaDLHRvyVWnk9Av sxzJU8xEaYAnI0HGpVZ1iegOra68/rB38FI2L7yLTidEw+UnoTzOwF5/2/lH7f/dtwQI rIVDkGZ6i7ToThX7x2eY/5FAKwP7/51UcUcISSTO0+G1vIjoCnG/A08pB7iPpYmxkKIO /ZvXEsKsed6+3Ua4lSNabgRN3g86a8gd1khte9U+7AMYd5j2RSz58FhOTgDda2KNIwRv bKzq0nJNnpjD8dMKZr/B1WY2H1WIobu4d7ATG45AnEEUSHd3Z2FphhGepQgrX5VFeIBZ dmYQ== X-Gm-Message-State: ABUngvc1Q3yyzqBsNvA7/PceeXj+fJY9iolfVnQ6k5MaBZ6KSkOt2LqLk5sDtpZvCyUDKtDIFDO/KrVTW4ol1U7i X-Received: by 10.55.74.1 with SMTP id x1mr5186373qka.316.1478490119577; Sun, 06 Nov 2016 19:41:59 -0800 (PST) MIME-Version: 1.0 References: <2016110711171841093835@zjqunshuo.com> In-Reply-To: <2016110711171841093835@zjqunshuo.com> From: Ben Slater Date: Mon, 07 Nov 2016 03:41:49 +0000 Message-ID: Subject: Re: Is it a memory issue? To: user Content-Type: multipart/alternative; boundary=001a114a7db861b62b0540adce78 archived-at: Mon, 07 Nov 2016 03:42:24 -0000 --001a114a7db861b62b0540adce78 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable This sounds to me like your writes go ahead of compactions trying to keep up which can eventually cause issues. Keep an eye on nodetool compactionstats if the number of compactions continually climbs then you are writing faster than Cassandra can actually process. If this is happening then you need to either add more processing capacity (nodes) to your cluster or throttle writes on the client side. It could also be related to conditions like an individual partition growing too big but I=E2=80=99d check for backed up compactions first. Cheers Ben On Mon, 7 Nov 2016 at 14:17 wxn002@zjqunshuo.com wrote: > Hi All, > We have one issue on C* testing. At first the inserting was very fast and > TPS was about 30K/s, but when the size of data rows reached 2 billion, th= e > insertion rate decreased very badly and the TPS was 20K/s. When the size = of > rows reached 2.3 billion, the TPS decreased to 0.5K/s, and writing timeou= t > come out. At last OOM issue happened in some nodes and C* deamon in some > nodes crashed. In production we have about 8 billion rows. My testing > cluster setting is as below. My question is if the memory is the main > issue. Do I need increase the memory, and what's the right setting for MA= X_HEAP_SIZE > and HEAP_NEWSIZE? > > My cluster setting: > C* cluster with 3 nodes in Aliyun Cloud > CPU: 4core > Memory: 8G > Disk: 500G > MAX_HEAP_SIZE=3D2G > HEAP_NEWSIZE=3D500M > > My table schema: > > CREATE KEYSPACE IF NOT EXISTS cargts WITH REPLICATION =3D {'class': 'Simp= leStrategy','replication_factor':2}; > use cargts; > CREATE TABLE eventdata ( > deviceId int, > date int, > event_time bigint, > lat decimal, > lon decimal, > speed int, > heading int, > PRIMARY KEY ((deviceId,date),event_time) > ) > WITH CLUSTERING ORDER BY (event_time ASC); > CREATE INDEX ON eventdata (event_time); > > Best Regards, > -Simon Wu > > --001a114a7db861b62b0540adce78 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
This sounds to me like your writes go ahead of compactions= trying to keep up which can eventually cause issues. Keep an eye on nodeto= ol compactionstats if the number of compactions continually climbs then you= are writing faster than Cassandra can actually process. If this is happeni= ng then you need to either add more processing capacity (nodes) to your clu= ster or throttle writes on the client side.

It could als= o be related to conditions like an individual partition growing too big but= I=E2=80=99d check for backed up compactions first.

Cheers
Ben

On Mon, 7 Nov 2016 at 14:17 wxn002@zjqunshuo.com <wxn0= 02@zjqunshuo.com> wrote:
Hi All,
We have one issue on C* testi= ng.=C2=A0At first the inserting was very fast and TPS was= about 30K/s, but when the size of data rows reached 2 billion, the inserti= on rate decreased very badly and the TPS was 20K/s. When the size of rows r= eached 2.3 billion, the TPS decreased to 0.5K/s, and writing timeout come o= ut. At last OOM issue happened in some nodes and C* deamon in some nodes cr= ashed. =C2=A0In production we have about 8 billion= rows. My testing cluster setting is as below. My question is=C2=A0<= span style=3D"background-color:window;font-size:10.5pt;line-height:1.5" cla= ss=3D"gmail_msg">if the memory is the main issue. Do I need increase the me= mory, and what's the right setting for=C2=A0MAX_HEAP_SIZE and=C2=A0HEAP_NEWS= IZE?

My cluster setting:
C* cluster with 3 nodes in Aliyun Cloud
CPU: 4core
Memory: 8G
<= div class=3D"gmail_msg">Disk: 500G
MAX_HEAP_SIZE=3D2G
HEAP_NEWSIZE=3D500M

My table schema:
= CREATE=C2=A0KEYSPACE=C2=A0IF=C2=A0NOT=C2=A0EXISTS=C2=A0cargts=C2=A0WITH=C2= =A0REPLICATION=C2=A0=3D=C2=A0{'class':=C2=A0'SimpleStrategy'= ;,'replication_factor':2};
use=C2=A0cargts;<= br class=3D"gmail_msg">CREATE=C2=A0TABLE=C2=A0eventdata=C2=A0(
deviceId=C2=A0int,
date=C2=A0int,
event_time=C2=A0bigint,
lat=C2=A0dec= imal,
lon=C2=A0decimal,
speed= =C2=A0int,
heading=C2=A0int,
= PRIMARY=C2=A0KEY=C2=A0((deviceId,date),event_time)
)=
WITH=C2=A0CLUSTERING=C2=A0ORDER=C2=A0BY=C2=A0(event= _time=C2=A0ASC);
CREATE=C2=A0INDEX=C2=A0ON=C2=A0even= tdata=C2=A0(event_time);

Best Regards,
-Simon Wu

--001a114a7db861b62b0540adce78--