Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
From: Graham Sanderson <graham@vast.com>
Content-Type: multipart/alternative;
	boundary=Apple-Mail-FBAE5B64-0CB9-4519-BAA4-17B5DE723F16
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (1.0)
Subject: Re: Cassandra Tuning Issue
Message-Id: <E07CBA0A-55C0-4489-AC84-6E9ECB7FF82A@vast.com>
Date: Sun, 6 Dec 2015 10:40:06 -0600
References: <977A042F-D7BB-4C89-8D85-B075BF4EF29B@126.com>
 <CAOxAL61MAdhoY-fqf03He3evrYwhbMV0uckBRPOp7EiaDg29HA@mail.gmail.com>
In-Reply-To: 
 <CAOxAL61MAdhoY-fqf03He3evrYwhbMV0uckBRPOp7EiaDg29HA@mail.gmail.com>
To: user@cassandra.apache.org


--Apple-Mail-FBAE5B64-0CB9-4519-BAA4-17B5DE723F16
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

What version of C* are you using; what JVM version - you showed a partial GC=
 config but if that is still CMS (not G1) then you are going to have insane G=
C pauses...=20

Depending on C* versions are you using on/off heap memtables and what type

Those are the sorts of issues related to fat nodes; I'd be worried about - w=
e run very nicely at 20G total heap and 8G new - the rest of our 128G memory=
 is disk cache/mmap and all of the off heap stuff so it doesn't go to waste

That said I think Jack is probably on the right path with overloaded coordin=
ators- though you'd still expect to see CPU usage unless your timeouts are t=
oo low for the load, In which case the coordinator would be getting no respo=
nses in time and quite possibly the other nodes are just dropping the mutati=
ons (since they don't get to them before they know the coordinator would hav=
e timed out) - I forget the command to check dropped mutations off the top o=
f my head but you can see it in opcenter

If you have GC problems you certainly
Expect to see GC cpu usage but depending on how long you run your tests it m=
ight take you a little while to run thru 40G

I'm personally not a fan off >32G (ish) heaps as you can't do compressed oop=
s and also it is unrealistic for CMS ... The word is that G1 is now working o=
k with C* especially on newer C* and JDK versions, but that said it takes qu=
ite a lot of thru-put to require insane quantities of young gen... We are gu=
essing that when we remove all our legacy thrift batch inserts we will need l=
ess - and as for 20G total we actually don't need that much (we dropped from=
 24 when we moved memtables off heap, and believe we can drop further)

Sent from my iPhone

> On Dec 6, 2015, at 9:07 AM, Jack Krupansky <jack.krupansky@gmail.com> wrot=
e:
>=20
> What replication factor are you using? Even if your writes use CL.ONE, Cas=
sandra will be attempting writes to the replica nodes in the background.
>=20
> Are your writes "token aware"? If not, the receiving node has the overhead=
 of forwarding the request to the node that owns the token for the primary k=
ey.
>=20
> For the record, Cassandra is not designed and optimized for so-called "fat=
 nodes". The design focus is "commodity hardware" and "distributed cluster" (=
typically a dozen or more nodes.)
>=20
> That said, it would be good if we had a rule of thumb for how many simulta=
neous requests a node can handle, both external requests and inter-node traf=
fic. I think there is an open Jira to enforce a limit on inflight requests s=
o that nodes don't overloaded and start failing in the middle of writes as y=
ou seem to be seeing.
>=20
> -- Jack Krupansky
>=20
>> On Sun, Dec 6, 2015 at 9:29 AM, jerry <xutom2006@126.com> wrote:
>> Dear All,
>>=20
>>     Now I have a 4 nodes Cassandra cluster, and I want to know the highes=
t performance of my Cassandra cluster. I write a JAVA client to batch insert=
 datas into ALL 4 nodes Cassandra, when I start less than 30 subthreads in m=
y client applications to insert datas into cassandra, it will be ok for ever=
ything, but when I start more than 80 or 100 subthreads in my client applica=
tions, there will be too much timeout Exceptions (Such as: Cassandra timeout=
 during write query at consistency ONE (1 replica were required but only 0 a=
cknowledged the write)). And no matter how many subthreads or even I start m=
ultiple clients with multiple subthreads on different computers, I can get t=
he highest performance for about 60000 - 80000 TPS. By the way, each row I i=
nsert into cassandra is about 130 Bytes.
>>     My 4 nodes of Cassandra is :
>>         CPU: 4*15
>>         Memory: 512G
>>         Disk: flash card (only one disk but better than SSD)
>>     My cassandra configurations are:
>>         MAX_HEAP_SIZE: 60G
>>         NEW_HEAP_SIZE: 40G
>>=20
>>     When I insert datas into my cassandra cluster, each nodes has NOT rea=
ched bottleneck such as CPU or Memory or Disk. Each of the three main hardwa=
res is idle=E3=80=82So I think maybe there is something wrong about my confi=
guration of cassandra cluster. Can somebody please help me to My Cassandra T=
uning? Thanks in advances!
>=20

--Apple-Mail-FBAE5B64-0CB9-4519-BAA4-17B5DE723F16
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D=
utf-8"></head><body dir=3D"auto"><div>What version of C* are you using; what=
 JVM version - you showed a partial GC config but if that is still CMS (not G=
1) then you are going to have insane GC pauses...&nbsp;</div><div id=3D"Appl=
eMailSignature"><br></div><div id=3D"AppleMailSignature">Depending on C* ver=
sions are you using on/off heap memtables and what type</div><div id=3D"Appl=
eMailSignature"><br></div><div id=3D"AppleMailSignature">Those are the sorts=
 of issues related to fat nodes; I'd be worried about - we run very nicely a=
t 20G total heap and 8G new - the rest of our 128G memory is disk cache/mmap=
 and all of the off heap stuff so it doesn't go to waste</div><div id=3D"App=
leMailSignature"><br></div><div id=3D"AppleMailSignature">That said I think J=
ack is probably on the right path with overloaded coordinators- though you'd=
 still expect to see CPU usage unless your timeouts are too low for the load=
, In which case the coordinator would be getting no responses in time and qu=
ite possibly the other nodes are just dropping the mutations (since they don=
't get to them before they know the coordinator would have timed out) - I fo=
rget the command to check dropped mutations off the top of my head but you c=
an see it in opcenter</div><div id=3D"AppleMailSignature"><br></div><div id=3D=
"AppleMailSignature">If you have GC problems you certainly</div><div id=3D"A=
ppleMailSignature">Expect to see GC cpu usage but depending on how long you r=
un your tests it might take you a little while to run thru 40G</div><div id=3D=
"AppleMailSignature"><br></div><div id=3D"AppleMailSignature">I'm personally=
 not a fan off &gt;32G (ish) heaps as you can't do compressed oops and also i=
t is unrealistic for CMS ... The word is that G1 is now working ok with C* e=
specially on newer C* and JDK versions, but that said it takes quite a lot o=
f thru-put to require insane quantities of young gen... We are guessing that=
 when we remove all our legacy thrift batch inserts we will need less - and a=
s for 20G total we actually don't need that much (we dropped from 24 when we=
 moved memtables off heap, and believe we can drop further)</div><div id=3D"=
AppleMailSignature"><br>Sent from my iPhone</div><div><br>On Dec 6, 2015, at=
 9:07 AM, Jack Krupansky &lt;<a href=3D"mailto:jack.krupansky@gmail.com">jac=
k.krupansky@gmail.com</a>&gt; wrote:<br><br></div><blockquote type=3D"cite">=
<div><div dir=3D"ltr">What replication factor are you using? Even if your wr=
ites use CL.ONE, Cassandra will be attempting writes to the replica nodes in=
 the background.<div><br></div><div>Are your writes "token aware"? If not, t=
he receiving node has the overhead of forwarding the request to the node tha=
t owns the token for the primary key.</div><div><br></div><div>For the recor=
d, Cassandra is not designed and optimized for so-called "fat nodes". The de=
sign focus is "commodity hardware" and "distributed cluster" (typically a do=
zen or more nodes.)</div><div><br></div><div>That said, it would be good if w=
e had a rule of thumb for how many simultaneous requests a node can handle, b=
oth external requests and inter-node traffic. I think there is an open Jira t=
o enforce a limit on inflight requests so that nodes don't overloaded and st=
art failing in the middle of writes as you seem to be seeing.</div></div><di=
v class=3D"gmail_extra"><br clear=3D"all"><div><div class=3D"gmail_signature=
"><div dir=3D"ltr">-- Jack Krupansky</div></div></div>
<br><div class=3D"gmail_quote">On Sun, Dec 6, 2015 at 9:29 AM, jerry <span d=
ir=3D"ltr">&lt;<a href=3D"mailto:xutom2006@126.com" target=3D"_blank">xutom2=
006@126.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear All,=
<br>
<br>
&nbsp; &nbsp; Now I have a 4 nodes Cassandra cluster, and I want to know the=
 highest performance of my Cassandra cluster. I write a JAVA client to batch=
 insert datas into ALL 4 nodes Cassandra, when I start less than 30 subthrea=
ds in my client applications to insert datas into cassandra, it will be ok f=
or everything, but when I start more than 80 or 100 subthreads in my client a=
pplications, there will be too much timeout Exceptions (Such as: Cassandra t=
imeout during write query at consistency ONE (1 replica were required but on=
ly 0 acknowledged the write)). And no matter how many subthreads or even I s=
tart multiple clients with multiple subthreads on different computers, I can=
 get the highest performance for about 60000 - 80000 TPS. By the way, each r=
ow I insert into cassandra is about 130 Bytes.<br>
&nbsp; &nbsp; My 4 nodes of Cassandra is :<br>
&nbsp; &nbsp; &nbsp; &nbsp; CPU: 4*15<br>
&nbsp; &nbsp; &nbsp; &nbsp; Memory: 512G<br>
&nbsp; &nbsp; &nbsp; &nbsp; Disk: flash card (only one disk but better than S=
SD)<br>
&nbsp; &nbsp; My cassandra configurations are:<br>
&nbsp; &nbsp; &nbsp; &nbsp; MAX_HEAP_SIZE: 60G<br>
&nbsp; &nbsp; &nbsp; &nbsp; NEW_HEAP_SIZE: 40G<br>
<br>
&nbsp; &nbsp; When I insert datas into my cassandra cluster, each nodes has N=
OT reached bottleneck such as CPU or Memory or Disk. Each of the three main h=
ardwares is idle=E3=80=82So I think maybe there is something wrong about my c=
onfiguration of cassandra cluster. Can somebody please help me to My Cassand=
ra Tuning? Thanks in advances!<br>
</blockquote></div><br></div>
</div></blockquote></body></html>=

--Apple-Mail-FBAE5B64-0CB9-4519-BAA4-17B5DE723F16--