Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BFAED200BCA for ; Mon, 7 Nov 2016 06:42:33 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BE3A2160B0D; Mon, 7 Nov 2016 05:42:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B7711160AFC for ; Mon, 7 Nov 2016 06:42:32 +0100 (CET) Received: (qmail 84714 invoked by uid 500); 7 Nov 2016 05:42:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84701 invoked by uid 99); 7 Nov 2016 05:42:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Nov 2016 05:42:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 59A8B18006C for ; Mon, 7 Nov 2016 05:42:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.289 X-Spam-Level: * X-Spam-Status: No, score=1.289 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=instaclustr-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id C45OkeNmYJH5 for ; Mon, 7 Nov 2016 05:42:26 +0000 (UTC) Received: from mail-qk0-f179.google.com (mail-qk0-f179.google.com [209.85.220.179]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id CC8F15F246 for ; Mon, 7 Nov 2016 05:42:25 +0000 (UTC) Received: by mail-qk0-f179.google.com with SMTP id n204so153755894qke.2 for ; Sun, 06 Nov 2016 21:42:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=instaclustr-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=+Hutp+h79uKsCnrZ1+8GsX8DwIVkxre/p/1vcIdkVag=; b=0/IKELtGvOwTWosu+RFlU1M1Wj5Q/CkvQwevQXBalYjngu6Hqm9VD0MERWeO7yhkFo IG88Wxx1dzJhgOjPd3+4T0XVmb154ZuWGvffLTWbLonqxxV71lIZk6tko0OZtpCFjRyg YiQGB9LhJuF9RtqgXpN2CHezJqi6FPmC98+U/J/theEh45X863s5Qn09s01gLHldzRNm 10y1XDkVs0Vbp/TjsOjJ1qlLLdUey8ue5ugFiroPvVJRAczO4xTkrzDqIDiNbBB8guvR jy630Y7rIqvvpDc1RvlvZd+NM6J3OOW21PwFwaO/3YV7d5KZ4iJ9Wk2aXohFCjLrzLFh 3BFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=+Hutp+h79uKsCnrZ1+8GsX8DwIVkxre/p/1vcIdkVag=; b=KxUwhpYFEhwwOv1yHV+UGPnCawaUg4nHuOQG+3TNEOE9IhrjqHJlGXMAFOLNn0FPZd dBucBd1mVFHm+YztSvaOxKJTixWFaGitEZGZMYIhp79pilUaU63mz2VhYqTEnSSbfF6Q 6BcO3T+IpEcOmqvoYNoW+ALqU6QYdYNuxKLdf0CLgJUsuQICh48Dg83SBC5w1IztcVtC lQmJut6Qyna8HPPZ5RHhPjQkzSIs7cSsc4JMXwMKMYzgYUl/Sp9TzVkoxzlvQZa2s3JN ptob1aZUrl4OWYeJ7xOODddZ5dso8Axlbnt/yM6ZlhniNzUG3bzx0Xk57wRUh28+TBfT p5LA== X-Gm-Message-State: ABUngveZHAKCR7d0AOU9UBjhMrVb2D7MIws5JbhJFi3EcOX+LSt1QQzJLhNlKVuLTzsg13dqUIjLcYe7GScrToCc X-Received: by 10.55.132.4 with SMTP id g4mr4793723qkd.2.1478497344554; Sun, 06 Nov 2016 21:42:24 -0800 (PST) MIME-Version: 1.0 References: <2016110711171841093835@zjqunshuo.com> <2016110713283920123940@zjqunshuo.com> In-Reply-To: <2016110713283920123940@zjqunshuo.com> From: Ben Slater Date: Mon, 07 Nov 2016 05:42:13 +0000 Message-ID: Subject: Re: Is it a memory issue? To: user Content-Type: multipart/alternative; boundary=94eb2c076394061b430540af7dd1 archived-at: Mon, 07 Nov 2016 05:42:33 -0000 --94eb2c076394061b430540af7dd1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Yes, it does mean you=E2=80=99re getting ahead of Cassandra=E2=80=99s abili= ty to keep up although I would have probably expected a higher number of pending compactions before you got serious issues (I=E2=80=99ve seen numbers in the thousands). I notice from the screenshot you provide that you are using secondary indexes. There are a lot of way to missuse secondary indexes (vs not very many way to use them well). I think it=E2=80=99s possible that what you are= seeing is the result of the secondary index on event time (I assume a very high cardinality column). This is a good blog on secondary indexes: http://www.wentnet.com/blog/?p=3D77 Cheers Ben On Mon, 7 Nov 2016 at 16:29 wxn002@zjqunshuo.com wrote: > Thanks Ben. I stopped inserting and checked compaction status as you > mentioned. Seems there is lots of compaction work waiting to do. Please s= ee > below. In this case is it a sign that writting faster than C* can process= ? > > One node, > [root@iZbp11zpafrqfsiys90kzoZ bin]# ./nodetool compactionstats > pending tasks: 195 > > id compaction type keyspace = table completed total unit progr= ess > > 5da60b10-a4a9-11e6-88e9-755b5673a02a Compaction cargts ev= entdata.eventdata_event_time_idx 1699866872 26536427792 bytes 6.= 41% > > Compaction system = hints 10354379 5172210360 bytes 0.= 20% > Active compaction remaining time : 0h29m48s > > Another node, > [root@iZbp1iqnrpsdhoodwii32bZ bin]# ./nodetool compactionstats > pending tasks: 84 > > id compaction type keyspace = table completed total unit prog= ress > > 28a9d010-a4a7-11e6-b985-979fea8d6099 Compaction cargts = eventdata 656141400 1424412420 bytes 46= .06% > > 7c034840-a48e-11e6-b985-979fea8d6099 Compaction cargts ev= entdata.eventdata_event_time_idx 32098562606 42616107664 bytes 75= .32% > Active compaction remaining time : 0h11m12s > > > *From:* Ben Slater > *Date:* 2016-11-07 11:41 > *To:* user > *Subject:* Re: Is it a memory issue? > > This sounds to me like your writes go ahead of compactions trying to keep > up which can eventually cause issues. Keep an eye on nodetool > compactionstats if the number of compactions continually climbs then you > are writing faster than Cassandra can actually process. If this is > happening then you need to either add more processing capacity (nodes) to > your cluster or throttle writes on the client side. > > It could also be related to conditions like an individual partition > growing too big but I=E2=80=99d check for backed up compactions first. > > Cheers > Ben > > On Mon, 7 Nov 2016 at 14:17 wxn002@zjqunshuo.com > wrote: > > Hi All, > We have one issue on C* testing. At first the inserting was very fast and > TPS was about 30K/s, but when the size of data rows reached 2 billion, th= e > insertion rate decreased very badly and the TPS was 20K/s. When the size = of > rows reached 2.3 billion, the TPS decreased to 0.5K/s, and writing timeou= t > come out. At last OOM issue happened in some nodes and C* deamon in some > nodes crashed. In production we have about 8 billion rows. My testing > cluster setting is as below. My question is if the memory is the main > issue. Do I need increase the memory, and what's the right setting for MA= X_HEAP_SIZE > and HEAP_NEWSIZE? > > My cluster setting: > C* cluster with 3 nodes in Aliyun Cloud > CPU: 4core > Memory: 8G > Disk: 500G > MAX_HEAP_SIZE=3D2G > HEAP_NEWSIZE=3D500M > > My table schema: > > CREATE KEYSPACE IF NOT EXISTS cargts WITH REPLICATION =3D {'class': 'Simp= leStrategy','replication_factor':2}; > use cargts; > CREATE TABLE eventdata ( > deviceId int, > date int, > event_time bigint, > lat decimal, > lon decimal, > speed int, > heading int, > PRIMARY KEY ((deviceId,date),event_time) > ) > WITH CLUSTERING ORDER BY (event_time ASC); > CREATE INDEX ON eventdata (event_time); > > Best Regards, > -Simon Wu > > --94eb2c076394061b430540af7dd1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yes, it does mean you=E2=80=99re getting ahead of Cassandr= a=E2=80=99s ability to keep up although I would have probably expected a hi= gher number of pending compactions before you got serious issues (I=E2=80= =99ve seen numbers in the thousands).

I notice from the = screenshot you provide that you are using secondary indexes. There are a lo= t of way to missuse secondary indexes (vs not very many way to use them wel= l). I think it=E2=80=99s possible that what you are seeing is the result of= the secondary index on event time (I assume a very high cardinality column= ). This is a good blog on secondary indexes:=C2=A0http://www.wentnet.com/blog/?p=3D77
Cheers
Ben=C2=A0

On Mon, 7 Nov 2016 at 16:29 wxn002@zjqunshuo.com <wxn002@zjqunshuo.com> wrote:
Thanks Ben. I sto= pped inserting and checked compaction status as you mentioned. Seems there = is lots of compaction work waiting to do. Please see below. In this case is= it a sign that writting faster than C* can process?

One node,
[root@iZbp11zpafrqfsi= ys90kzoZ=C2=A0bin]#=C2=A0./nodetool=C2=A0compactionstats
pending=C2=A0tasks:=C2=A0195
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0id=C2=A0=C2=A0=C2=A0c= ompaction=C2=A0type=C2=A0=C2=A0=C2=A0keyspace=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0table=C2=A0=C2=A0=C2=A0=C2=A0completed=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0total=C2=A0=C2=A0=C2=A0=C2=A0unit=C2=A0=C2=A0= =C2=A0progress
=C2=A0=C2=A0=C2=A05da60b10-a4a9-11e6-= 88e9-755b5673a02a=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Compaction= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0cargts=C2=A0=C2=A0=C2=A0eventdata.eventdata_e= vent_time_idx=C2=A0=C2=A0=C2=A01699866872=C2=A0=C2=A0=C2=A026536427792=C2= =A0=C2=A0=C2=A0bytes=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A06.41%
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Compac= tion=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0system=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0hints=C2=A0=C2=A0=C2=A0=C2=A0=C2=A010354379=C2=A0=C2=A0=C2=A0=C2= =A05172210360=C2=A0=C2=A0=C2=A0bytes=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00.2= 0%
Active=C2=A0compaction=C2=A0remaining=C2=A0time= =C2=A0:=C2=A0=C2=A0=C2=A00h29m48s

Another node,
[root@iZbp1iqnrpsdhoodwii32bZ=C2=A0bin]#=C2= =A0./nodetool=C2=A0compactionstats
pending=C2=A0task= s:=C2=A084
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0id=C2=A0=C2=A0=C2=A0compaction=C2=A0type= =C2=A0=C2=A0=C2=A0keyspace=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0table= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0completed=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0total=C2=A0=C2=A0=C2=A0=C2=A0unit=C2=A0=C2=A0=C2=A0progre= ss
=C2=A0=C2=A0=C2=A028a9d010-a4a7-11e6-b985-979fea8= d6099=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Compaction=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0cargts=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0eventdata=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0656141400=C2=A0=C2=A0=C2=A0=C2=A01424412420=C2=A0=C2=A0=C2=A0bytes=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A046.06%
=C2=A0=C2=A0=C2=A0= 7c034840-a48e-11e6-b985-979fea8d6099=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0Compaction=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0cargts=C2=A0=C2=A0=C2=A0ev= entdata.eventdata_event_time_idx=C2=A0=C2=A0=C2=A032098562606=C2=A0=C2=A0= =C2=A042616107664=C2=A0=C2=A0=C2=A0bytes=C2=A0=C2=A0=C2=A0=C2=A0=C2=A075.32= %
Active=C2=A0compaction=C2=A0remaining=C2=A0time=C2= =A0:=C2=A0=C2=A0=C2=A00h11m12s
=C2=A0
From:=C2=A0Ben Slater
Date:=C2=A02016-11-07=C2=A011:41
To:=C2=A0user
Subject:=C2=A0Re: Is it a= memory issue?
This sounds to me like your writes go ahead of compactions trying to kee= p up which can eventually cause issues. Keep an eye on nodetool compactions= tats if the number of compactions continually climbs then you are writing f= aster than Cassandra can actually process. If this is happening then you ne= ed to either add more processing capacity (nodes) to your cluster or thrott= le writes on the client side.

It could also be related to conditions li= ke an individual partition growing too big but I=E2=80=99d check for backed= up compactions first.

Cheers
Ben

On Mon, 7 Nov 2016 at 14:17 wxn002@zjqu= nshuo.com <wxn002@zjqunshuo.com> wrote:
Hi All,
We have one issue on C* testi= ng.=C2=A0At first the inserting was very fast and TPS was= about 30K/s, but when the size of data rows reached 2 billion, the inserti= on rate decreased very badly and the TPS was 20K/s. When the size of rows r= eached 2.3 billion, the TPS decreased to 0.5K/s, and writing timeout come o= ut. At last OOM issue happened in some nodes and C* deamon in some nodes cr= ashed. =C2=A0In production we have about 8 billion= rows. My testing cluster setting is as below. My question is=C2=A0<= span style=3D"background-color:window;font-size:10.5pt;line-height:1.5" cla= ss=3D"gmail_msg">if the memory is the main issue. Do I need increase the me= mory, and what's the right setting for=C2=A0MAX_HEAP_SIZE and=C2=A0HEAP_NEWS= IZE?

My cluster setting:
C* cluster with 3 nodes in Aliyun Cloud
CPU: 4core
Memory: 8G
<= div class=3D"gmail_msg">Disk: 500G
MAX_HEAP_SIZE=3D2G
HEAP_NEWSIZE=3D500M

My table schema:
= CREATE=C2=A0KEYSPACE=C2=A0IF=C2=A0NOT=C2=A0EXISTS=C2=A0cargts=C2=A0WITH=C2= =A0REPLICATION=C2=A0=3D=C2=A0{'class':=C2=A0'SimpleStrategy'= ;,'replication_factor':2};
use=C2=A0cargts;<= br class=3D"gmail_msg">CREATE=C2=A0TABLE=C2=A0eventdata=C2=A0(
deviceId=C2=A0int,
date=C2=A0int,
event_time=C2=A0bigint,
lat=C2=A0dec= imal,
lon=C2=A0decimal,
speed= =C2=A0int,
heading=C2=A0int,
= PRIMARY=C2=A0KEY=C2=A0((deviceId,date),event_time)
)=
WITH=C2=A0CLUSTERING=C2=A0ORDER=C2=A0BY=C2=A0(event= _time=C2=A0ASC);
CREATE=C2=A0INDEX=C2=A0ON=C2=A0even= tdata=C2=A0(event_time);

Best Regards,
-Simon Wu

--94eb2c076394061b430540af7dd1--