Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C484A1153C for ; Tue, 23 Sep 2014 20:47:05 +0000 (UTC) Received: (qmail 57258 invoked by uid 500); 23 Sep 2014 20:47:02 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 57219 invoked by uid 500); 23 Sep 2014 20:47:02 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 57209 invoked by uid 99); 23 Sep 2014 20:47:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Sep 2014 20:47:02 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of clohfink@blackbirdit.com designates 209.85.223.169 as permitted sender) Received: from [209.85.223.169] (HELO mail-ie0-f169.google.com) (209.85.223.169) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Sep 2014 20:46:57 +0000 Received: by mail-ie0-f169.google.com with SMTP id rp18so8268063iec.14 for ; Tue, 23 Sep 2014 13:46:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=KZN5wP3X6CYzNp2QuRV/QqrbR+qLIDhD7ZufqgcjPUg=; b=QaGA7kXIgC30lUJUIYIH4hopgWan1P9X1uN5AOGom4fxJokGSUH3snUqmB4Cw5iebL 2qzWElzcX8186RFaOhzAFJfbN1iO2RtU1oAks4zewZ4bRKhD3ObKxze0KA8WPIH6jzkI HPXHmH8qfW10E7N1hSosUg+IDXoJ4a52YSOwpp26hCBPg+CbyNOa7g1VVncZ8KinCtoD cS/sDe3040+08Zf+ZDfsY79QG7MvLfoCQ6taskcdugLWuG9aan+zvZ2G5T4e5YZDjZlt iGIxaLPnaENtC88bCsw6Ljx89VGfqiMrbVPCzzRw8M6r9vmq/Q/s17Ok3xWriWXRjm6S RHvA== X-Gm-Message-State: ALoCoQnzEPfgaB5OuRMNYVuqI7tjAu/n6lZXV9+SHmlcukSp6QGGvDxrBDzAMsBraxvmhyl9OkIm X-Received: by 10.50.73.130 with SMTP id l2mr6847141igv.9.1411505196865; Tue, 23 Sep 2014 13:46:36 -0700 (PDT) Received: from [10.10.10.101] (97-86-246-164.dhcp.roch.mn.charter.com. [97.86.246.164]) by mx.google.com with ESMTPSA id uf4sm2570945igc.0.2014.09.23.13.46.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 23 Sep 2014 13:46:35 -0700 (PDT) From: Chris Lohfink Content-Type: multipart/alternative; boundary="Apple-Mail=_C761A09E-B7DF-4E9B-8A03-250569BB25A6" Message-Id: <69965B00-134C-41FE-B4E4-5FB5D05733DE@blackbirdit.com> Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: CPU consumption of Cassandra Date: Tue, 23 Sep 2014 15:46:35 -0500 References: <186D431703062245B153D0E5CEBAF3E7A21FB3DE2E@FRSPX100.fr01.awl.atosorigin.net> <52F52A90-3CFD-4BCC-9E30-FA6B0462779F@blackbirdit.com> <186D431703062245B153D0E5CEBAF3E7A21FC312A9@FRSPX100.fr01.awl.atosorigin.net>,<860D8501-FE43-4DB3-8BAF-42B05FF78F3E@blackbirdit.com> <186D431703062245B153D0E5CEBAF3E7A21FC47DBE@FRSPX100.fr01.awl.atosorigin.net> To: user@cassandra.apache.org In-Reply-To: <186D431703062245B153D0E5CEBAF3E7A21FC47DBE@FRSPX100.fr01.awl.atosorigin.net> X-Mailer: Apple Mail (2.1874) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_C761A09E-B7DF-4E9B-8A03-250569BB25A6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 CPU consumption may be affected from the cassandra-stress tool in 2nd = example as well. Running on a separate system eliminates it as a = possible cause. There is a little extra work but not anything that I = think would be that obvious. tracing (can enable with nodetool) or = profiling (ie with yourkit) can give more exposure to the bottleneck. = Id run test from separate system first. --- Chris Lohfink=20 On Sep 23, 2014, at 12:48 PM, Leleu Eric = wrote: > First of all, Thanks for your help ! :) >=20 > Here is some details : >=20 >> With RF=3DN=3D2 your essentially testing a single machine locally = which isnt the best indicator long term > I will test with more nodes, (4 with RF =3D 2) but for now I'm = limited to 2 nodes for non technical reason ... >=20 >> Well, first off you shouldn't run stress tool on the node your = testing. Give it its own box. > I performed the test in a new Keyspace in order to have a clear = dataset. >=20 >> the 2nd query since its returning 10x the data and there will be more = to go through within the partition > I configured cassandra-stress in a way of each user has only one = bucket so the amount of data is the same in the both case. ("select * = from buckets where name =3D ? and tenantid =3D ? limit 1" and "select * = from owner_to_buckets where owner =3D ? and tenantid =3D ? limit 10"). > Does cassandra perform extra read when the limit is bigger than the = available data (even if the partition key contains only one single value = in the clustering column) ? > If the amount of data is the same, how can we explain the difference = of CPU consumption? >=20 >=20 > Regards, > Eric >=20 > ________________________________________ > De : Chris Lohfink [clohfink@blackbirdit.com] > Date d'envoi : mardi 23 septembre 2014 19:23 > =C3=80 : user@cassandra.apache.org > Objet : Re: CPU consumption of Cassandra >=20 > Well, first off you shouldn't run stress tool on the node your = testing. Give it its own box. >=20 > With RF=3DN=3D2 your essentially testing a single machine locally = which isnt the best indicator long term (optimizations available when = reading data thats local to the node). 80k/sec on a system is pretty = good though, your probably seeing slower on the 2nd query since its = returning 10x the data and there will be more to go through within the = partition. 42k/sec is still acceptable imho since these are smaller = boxes. You are probably seeing high CPU because the system is doing a = lot :) >=20 > If you want to get more out of these systems can do some tuning = probably, enable trace to see whats actually the bottleneck. >=20 > Collections will very likely hurt more then help. >=20 > --- > Chris Lohfink >=20 > On Sep 23, 2014, at 9:39 AM, Leleu Eric = > wrote: >=20 > I tried to run =E2=80=9Ccassandra-stress=E2=80=9D on some of my table = as proposed by Jake Luciani. >=20 > For a simple table, this tool is able to perform 80000 read op/s with = a few CPU consumption if I request the table by the PK(name, tenanted) >=20 > Ex : > TABLE : >=20 > CREATE TABLE IF NOT EXISTS buckets (tenantid varchar, > name varchar, > owner varchar, > location varchar, > description varchar, > codeQuota varchar, > creationDate timestamp, > updateDate timestamp, > PRIMARY KEY (name, tenantid)); >=20 > QUERY : select * from buckets where name =3D ? and tenantid =3D ? = limit 1; >=20 > TOP output for 900 threads on cassandra-stress : > top - 13:17:09 up 173 days, 21:54, 4 users, load average: 11.88, = 4.30, 2.76 > Tasks: 272 total, 1 running, 270 sleeping, 0 stopped, 1 zombie > Cpu(s): 71.4%us, 14.0%sy, 0.0%ni, 13.1%id, 0.0%wa, 0.0%hi, 1.5%si, = 0.0%st > Mem: 98894704k total, 96367436k used, 2527268k free, 15440k = buffers > Swap: 0k total, 0k used, 0k free, 88194556k = cached >=20 > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 25857 root 20 0 29.7g 1.5g 12m S 693.0 1.6 38:45.58 java = <=3D=3D Cassandra-stress > 29160 cassandr 20 0 16.3g 4.8g 10m S 1.3 5.0 44:46.89 java <=3D=3D= Cassandra >=20 >=20 >=20 > Now, If I run another query on a table that provides a list of buckets = according to the owner, the number of op/s is divided by 2 (42000 = op/s) and CPU consumption grow UP. >=20 > Ex : > TABLE : >=20 > CREATE TABLE IF NOT EXISTS owner_to_buckets (tenantid varchar, > name varchar, > owner varchar, > location varchar, > description varchar, > codeQuota varchar, > creationDate timestamp, > updateDate timestamp, > PRIMARY KEY ((owner, tenantid), name)); >=20 > QUERY : select * from owner_to_buckets where owner =3D ? and tenantid = =3D ? limit 10; >=20 > TOP output for 4 threads on cassandra-stress: >=20 > top - 13:49:16 up 173 days, 22:26, 4 users, load average: 1.76, = 1.48, 1.17 > Tasks: 273 total, 1 running, 271 sleeping, 0 stopped, 1 zombie > Cpu(s): 26.3%us, 8.0%sy, 0.0%ni, 64.7%id, 0.0%wa, 0.0%hi, 1.0%si, = 0.0%st > Mem: 98894704k total, 97512156k used, 1382548k free, 14580k = buffers > Swap: 0k total, 0k used, 0k free, 90413772k = cached >=20 > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 29160 cassandr 20 0 13.6g 4.8g 37m S 186.7 5.1 62:26.77 java <=3D=3D= Cassandra > 50622 root 20 0 28.8g 469m 12m S 102.5 0.5 0:45.84 java <=3D=3D= Cassandra-stress >=20 > TOP output for 271 threads on cassandra-stress: >=20 >=20 > top - 13:57:03 up 173 days, 22:34, 4 users, load average: 4.67, = 1.76, 1.25 > Tasks: 272 total, 1 running, 270 sleeping, 0 stopped, 1 zombie > Cpu(s): 81.5%us, 14.0%sy, 0.0%ni, 3.1%id, 0.0%wa, 0.0%hi, 1.3%si, = 0.0%st > Mem: 98894704k total, 94955936k used, 3938768k free, 15892k = buffers > Swap: 0k total, 0k used, 0k free, 85993676k = cached >=20 > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 29160 cassandr 20 0 13.6g 4.8g 38m S 430.0 5.1 82:31.80 java <=3D=3D= Cassandra > 50622 root 20 0 29.1g 2.3g 12m S 343.4 2.4 17:51.22 java <=3D=3D= Cassandra-stress >=20 >=20 > I have 4 tables with a composed PRIMARY KEY (two of them has 4 = entries : 2 for the partition key, one for cluster column and one for = sort column) > Two of these tables are frequently read with the partition key because = we want to list data of a given user, this should explain my CPU load = according to the simple test done with Cassandra-stress =E2=80=A6 >=20 > How can I avoid this? > Collections could be an option but the number of data per user is not = limited and can easily exceed 200 entries. According to the Cassandra = documentation, collections have a size limited to 64KB. So it is = probably not a solution in my case. =E2=98=B9 >=20 >=20 > Regards, > Eric >=20 > De : Chris Lohfink [mailto:clohfink@blackbirdit.com] > Envoy=C3=A9 : lundi 22 septembre 2014 22:03 > =C3=80 : user@cassandra.apache.org > Objet : Re: CPU consumption of Cassandra >=20 > Its going to depend a lot on your data model but 5-6k is on the low = end of what I would expect. N=3DRF=3D2 is not really something I would = recommend. That said 93GB is not much data so the bottleneck may exist = more in your data model, queries, or client. >=20 > What profiler are you using? The cpu on the select/read is marked as = RUNNABLE but its really more of a wait state that may throw some = profilers off, it may be a red haring. >=20 > --- > Chris Lohfink >=20 > On Sep 22, 2014, at 11:39 AM, Leleu Eric = > wrote: >=20 >=20 > Hi, >=20 >=20 > I=E2=80=99m currently testing Cassandra 2.0.9 (and since the last = week 2.1) under some read heavy load=E2=80=A6 >=20 > I have 2 cassandra nodes (RF : 2) running under CentOS 6 with 16GB of = RAM and 8 Cores. > I have around 93GB of data per node (one Disk of 300GB with SAS = interface and a Rotational Speed of 10500) >=20 > I have 300 active client threads and they request the C* nodes with a = Consitency level set to ONE (I=E2=80=99m using the CQL datastax driver). >=20 > During my tests I saw a lot of CPU consumption (70% user / 6%sys / 4% = iowait / 20%idle). > C* nodes respond to around 5000 op/s (sometime up to 6000op/s) >=20 > I try to profile a node and at the first look, 60% of the CPU is = passed in the =E2=80=9Csun.nio.ch=E2=80=9D package. = (SelectorImpl.select or Channel.read) >=20 > I know that Benchmark results are highly dependent of the Dataset and = use cases, but according to my point of view this CPU consumption is = normal according to the load. > Someone can confirm that point ? > According to my Hardware configuration, can I expect to have more than = 6000 read op/s ? >=20 >=20 > Regards, > Eric >=20 >=20 >=20 >=20 >=20 > ________________________________ >=20 > Ce message et les pi=C3=A8ces jointes sont confidentiels et r=C3=A9serv=C3= =A9s =C3=A0 l'usage exclusif de ses destinataires. Il peut =C3=A9galement = =C3=AAtre prot=C3=A9g=C3=A9 par le secret professionnel. Si vous recevez = ce message par erreur, merci d'en avertir imm=C3=A9diatement = l'exp=C3=A9diteur et de le d=C3=A9truire. L'int=C3=A9grit=C3=A9 du = message ne pouvant =C3=AAtre assur=C3=A9e sur Internet, la = responsabilit=C3=A9 de Worldline ne pourra =C3=AAtre recherch=C3=A9e = quant au contenu de ce message. Bien que les meilleurs efforts soient = faits pour maintenir cette transmission exempte de tout virus, = l'exp=C3=A9diteur ne donne aucune garantie =C3=A0 cet =C3=A9gard et sa = responsabilit=C3=A9 ne saurait =C3=AAtre recherch=C3=A9e pour tout = dommage r=C3=A9sultant d'un virus transmis. >=20 > This e-mail and the documents attached are confidential and intended = solely for the addressee; it may also be privileged. If you receive this = e-mail in error, please notify the sender immediately and destroy it. As = its integrity cannot be secured on the Internet, the Worldline liability = cannot be triggered for the message content. Although the sender = endeavours to maintain a computer virus-free network, the sender does = not warrant that this transmission is virus-free and will not be liable = for any damages resulting from any virus transmitted. >=20 >=20 > ________________________________ >=20 > Ce message et les pi=C3=A8ces jointes sont confidentiels et r=C3=A9serv=C3= =A9s =C3=A0 l'usage exclusif de ses destinataires. Il peut =C3=A9galement = =C3=AAtre prot=C3=A9g=C3=A9 par le secret professionnel. Si vous recevez = ce message par erreur, merci d'en avertir imm=C3=A9diatement = l'exp=C3=A9diteur et de le d=C3=A9truire. L'int=C3=A9grit=C3=A9 du = message ne pouvant =C3=AAtre assur=C3=A9e sur Internet, la = responsabilit=C3=A9 de Worldline ne pourra =C3=AAtre recherch=C3=A9e = quant au contenu de ce message. Bien que les meilleurs efforts soient = faits pour maintenir cette transmission exempte de tout virus, = l'exp=C3=A9diteur ne donne aucune garantie =C3=A0 cet =C3=A9gard et sa = responsabilit=C3=A9 ne saurait =C3=AAtre recherch=C3=A9e pour tout = dommage r=C3=A9sultant d'un virus transmis. >=20 > This e-mail and the documents attached are confidential and intended = solely for the addressee; it may also be privileged. If you receive this = e-mail in error, please notify the sender immediately and destroy it. As = its integrity cannot be secured on the Internet, the Worldline liability = cannot be triggered for the message content. Although the sender = endeavours to maintain a computer virus-free network, the sender does = not warrant that this transmission is virus-free and will not be liable = for any damages resulting from any virus transmitted. >=20 >=20 >=20 > Ce message et les pi=C3=A8ces jointes sont confidentiels et r=C3=A9serv=C3= =A9s =C3=A0 l'usage exclusif de ses destinataires. Il peut =C3=A9galement = =C3=AAtre prot=C3=A9g=C3=A9 par le secret professionnel. Si vous recevez = ce message par erreur, merci d'en avertir imm=C3=A9diatement = l'exp=C3=A9diteur et de le d=C3=A9truire. L'int=C3=A9grit=C3=A9 du = message ne pouvant =C3=AAtre assur=C3=A9e sur Internet, la = responsabilit=C3=A9 de Worldline ne pourra =C3=AAtre recherch=C3=A9e = quant au contenu de ce message. Bien que les meilleurs efforts soient = faits pour maintenir cette transmission exempte de tout virus, = l'exp=C3=A9diteur ne donne aucune garantie =C3=A0 cet =C3=A9gard et sa = responsabilit=C3=A9 ne saurait =C3=AAtre recherch=C3=A9e pour tout = dommage r=C3=A9sultant d'un virus transmis. >=20 > This e-mail and the documents attached are confidential and intended = solely for the addressee; it may also be privileged. If you receive this = e-mail in error, please notify the sender immediately and destroy it. As = its integrity cannot be secured on the Internet, the Worldline liability = cannot be triggered for the message content. Although the sender = endeavours to maintain a computer virus-free network, the sender does = not warrant that this transmission is virus-free and will not be liable = for any damages resulting from any virus transmitted. --Apple-Mail=_C761A09E-B7DF-4E9B-8A03-250569BB25A6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
CPU consumption may be affected from the = cassandra-stress tool in 2nd example as well.  Running on a = separate system eliminates it as a possible cause.  There is a = little extra work but not anything that I think would be that obvious. =  tracing (can enable with nodetool) or profiling (ie with yourkit) = can give more exposure to the bottleneck.  Id run test from = separate system first.

---
Chris = Lohfink 


On Sep 23, 2014, at = 12:48 PM, Leleu Eric <Eric.Leleu@worldline.com> = wrote:

First of all, Thanks for your help = ! :)

Here is some details :

With = RF=3DN=3D2 your essentially testing a single machine locally which isnt = the best indicator long term
I will  test with more = nodes, (4 with RF =3D 2) but for now I'm limited to 2 nodes for non = technical reason ...

Well, first off = you shouldn't run stress tool on the node your testing.  Give it = its own box.
I performed the test in a new Keyspace in = order to have a clear dataset.

the 2nd = query since its returning 10x the data and there will be more to go = through within the partition
I configured = cassandra-stress in a way of each user has only one bucket so the amount = of data is the same in the both case. ("select * from buckets where name = =3D ? and tenantid =3D ? limit 1" and "select * from owner_to_buckets =  where owner =3D ? and tenantid =3D ? limit 10").
Does cassandra = perform extra read when the limit is bigger than the available data = (even if the partition key contains only one single value in the = clustering column) ?
If the amount of data is the same, how can we = explain the difference of CPU = consumption?


Regards,
Eric

__________________________= ______________
De : Chris Lohfink [clohfink@blackbirdit.com]
= Date d'envoi : mardi 23 septembre 2014 19:23
=C3=80 : user@cassandra.apache.orgObjet : Re: CPU consumption of Cassandra

Well, first off you = shouldn't run stress tool on the node your testing.  Give it its = own box.

With RF=3DN=3D2 your essentially testing a single = machine locally which isnt the best indicator long term (optimizations = available when reading data thats local to the node).  80k/sec on a = system is pretty good though, your probably seeing slower on the 2nd = query since its returning 10x the data and there will be more to go = through within the partition. 42k/sec is still acceptable imho since = these are smaller boxes.  You are probably seeing high CPU because = the system is doing a lot :)

If you want to get more out of these = systems can do some tuning probably, enable trace to see whats actually = the bottleneck.

Collections will very likely hurt more then = help.

---
Chris Lohfink

On Sep 23, 2014, at 9:39 AM, = Leleu Eric <Eric.Leleu@worldline.com<<= a = href=3D"mailto:Eric.Leleu@worldline.com">mailto:Eric.Leleu@worldline.com>> wrote:

I tried to run =E2=80=9Ccassandra-stress=E2=80=9D= on some of my table as proposed by Jake Luciani.

For a simple = table, this tool is able to perform 80000 read op/s with a few CPU = consumption if I request the table by the PK(name, tenanted)

Ex = :
TABLE :

CREATE TABLE IF NOT EXISTS buckets (tenantid = varchar,
name varchar,
owner varchar,
location = varchar,
description varchar,
codeQuota varchar,
creationDate = timestamp,
updateDate timestamp,
PRIMARY KEY (name, = tenantid));

QUERY : select * from buckets where name =3D ? and = tenantid =3D ? limit 1;

TOP output for 900 threads on = cassandra-stress :
top - 13:17:09 up 173 days, 21:54,  4 users, =  load average: 11.88, 4.30, 2.76
Tasks: 272 total,   1 = running, 270 sleeping,   0 stopped,   1 = zombie
Cpu(s): 71.4%us, 14.0%sy,  0.0%ni, 13.1%id,  0.0%wa, =  0.0%hi,  1.5%si,  0.0%st
Mem:  98894704k total, = 96367436k used,  2527268k free,    15440k = buffers
Swap:        0k total, =        0k used, =        0k free, 88194556k = cached

 PID USER      PR  NI =  VIRT  RES  SHR S %CPU %MEM    TIME+ =  COMMAND
25857 root      20 =   0 29.7g 1.5g  12m S 693.0  1.6  38:45.58 java =  <=3D=3D Cassandra-stress
29160 cassandr  20 =   0 16.3g 4.8g  10m S  1.3  5.0  44:46.89 = java  <=3D=3D Cassandra



Now, If I run another = query on a table that provides a list of buckets according to the =  owner, the number of op/s is divided by 2  (42000 op/s) and = CPU consumption grow UP.

Ex :
TABLE :

CREATE TABLE IF = NOT EXISTS owner_to_buckets (tenantid varchar,
name varchar,
owner = varchar,
location varchar,
description varchar,
codeQuota = varchar,
creationDate timestamp,
updateDate timestamp,
PRIMARY = KEY ((owner, tenantid), name));

QUERY : select * from = owner_to_buckets  where owner =3D ? and tenantid =3D ? limit = 10;

TOP output for 4  threads on = cassandra-stress:

top - 13:49:16 up 173 days, 22:26,  4 = users,  load average: 1.76, 1.48, 1.17
Tasks: 273 total, =   1 running, 271 sleeping,   0 stopped, =   1 zombie
Cpu(s): 26.3%us,  8.0%sy,  0.0%ni, = 64.7%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
Mem: =  98894704k total, 97512156k used,  1382548k free, =    14580k buffers
Swap: =        0k total, =        0k used, =        0k free, 90413772k = cached

 PID USER      PR  NI =  VIRT  RES  SHR S %CPU %MEM    TIME+ =  COMMAND
29160 cassandr  20   0 13.6g 4.8g =  37m S 186.7  5.1  62:26.77 java <=3D=3D = Cassandra
50622 root      20   0 = 28.8g 469m  12m S 102.5  0.5   0:45.84 java <=3D=3D= Cassandra-stress

TOP output for 271  threads on = cassandra-stress:


top - 13:57:03 up 173 days, 22:34,  4 = users,  load average: 4.67, 1.76, 1.25
Tasks: 272 total, =   1 running, 270 sleeping,   0 stopped, =   1 zombie
Cpu(s): 81.5%us, 14.0%sy,  0.0%ni, =  3.1%id,  0.0%wa,  0.0%hi,  1.3%si, =  0.0%st
Mem:  98894704k total, 94955936k used, =  3938768k free,    15892k buffers
Swap: =        0k total, =        0k used, =        0k free, 85993676k = cached

 PID USER      PR  NI =  VIRT  RES  SHR S %CPU %MEM    TIME+ =  COMMAND
29160 cassandr  20   0 13.6g 4.8g =  38m S 430.0  5.1  82:31.80 java <=3D=3D = Cassandra
50622 root      20   0 = 29.1g 2.3g  12m S 343.4  2.4  17:51.22 java <=3D=3D = Cassandra-stress


I have 4 tables with  a composed = PRIMARY KEY (two of them has 4 entries : 2 for the partition key, one = for cluster column and one for sort column)
Two of these tables are = frequently read with the partition key because we want to list data of a = given user, this should explain my CPU load according to the simple test = done with Cassandra-stress =E2=80=A6

How can I avoid = this?
Collections could be an option but the number of data per user = is not limited and can easily exceed 200 entries. According to the = Cassandra documentation, collections have a size limited to 64KB. So it = is probably not a solution in my case. = =E2=98=B9


Regards,
Eric

De : Chris Lohfink [mailto:clohfink@blackbirdit.com]
Envoy=C3=A9 : lundi 22 septembre 2014 22:03
=C3=80 : user@cassandra.apache.org<= ;mailto:user@cassandra.apache.org= >
Objet : Re: CPU consumption of Cassandra

Its going to = depend a lot on your data model but 5-6k is on the low end of what I = would expect.  N=3DRF=3D2 is not really something I would = recommend.  That said 93GB is not much data so the bottleneck may = exist more in your data model, queries, or client.

What profiler = are you using?  The cpu on the select/read is marked as RUNNABLE = but its really more of a wait state that may throw some profilers off, = it may be a red haring.

---
Chris Lohfink

On Sep 22, = 2014, at 11:39 AM, Leleu Eric <Eric.Leleu@worldline.com<<= a = href=3D"mailto:Eric.Leleu@worldline.com">mailto:Eric.Leleu@worldline.com>> wrote:


Hi,


I=E2=80=99m currently = testing Cassandra 2.0.9  (and since the last week 2.1) under some = read heavy load=E2=80=A6

I have 2 cassandra nodes (RF : 2) = running under CentOS 6 with 16GB of RAM and 8 Cores.
I have around = 93GB of data per node (one Disk of 300GB with SAS interface and a = Rotational Speed of 10500)

I have 300 active client threads and = they request the C* nodes with a Consitency level set to ONE (I=E2=80=99m = using the CQL datastax driver).

During my tests I saw  a lot = of CPU consumption (70% user / 6%sys / 4% iowait / 20%idle).
C* nodes = respond to around 5000 op/s (sometime up to 6000op/s)

I try to = profile a node and at the first look, 60% of the CPU is passed in the = =E2=80=9Csun.nio.ch<http://sun.nio.ch/>=E2=80=9D package. = (SelectorImpl.select or Channel.read)

I know that Benchmark = results are highly dependent of the Dataset and use cases, but according = to my point of view this CPU consumption is normal according to the = load.
Someone can confirm that point ?
According to my Hardware = configuration, can I expect to have more than 6000 read op/s = ?


Regards,
Eric





_____________________= ___________

Ce message et les pi=C3=A8ces jointes sont = confidentiels et r=C3=A9serv=C3=A9s =C3=A0 l'usage exclusif de ses = destinataires. Il peut =C3=A9galement =C3=AAtre prot=C3=A9g=C3=A9 par le = secret professionnel. Si vous recevez ce message par erreur, merci d'en = avertir imm=C3=A9diatement l'exp=C3=A9diteur et de le d=C3=A9truire. = L'int=C3=A9grit=C3=A9 du message ne pouvant =C3=AAtre assur=C3=A9e sur = Internet, la responsabilit=C3=A9 de Worldline ne pourra =C3=AAtre = recherch=C3=A9e quant au contenu de ce message. Bien que les meilleurs = efforts soient faits pour maintenir cette transmission exempte de tout = virus, l'exp=C3=A9diteur ne donne aucune garantie =C3=A0 cet =C3=A9gard = et sa responsabilit=C3=A9 ne saurait =C3=AAtre recherch=C3=A9e pour tout = dommage r=C3=A9sultant d'un virus transmis.

This e-mail and the = documents attached are confidential and intended solely for the = addressee; it may also be privileged. If you receive this e-mail in = error, please notify the sender immediately and destroy it. As its = integrity cannot be secured on the Internet, the Worldline liability = cannot be triggered for the message content. Although the sender = endeavours to maintain a computer virus-free network, the sender does = not warrant that this transmission is virus-free and will not be liable = for any damages resulting from any virus = transmitted.


________________________________

Ce = message et les pi=C3=A8ces jointes sont confidentiels et r=C3=A9serv=C3=A9= s =C3=A0 l'usage exclusif de ses destinataires. Il peut =C3=A9galement = =C3=AAtre prot=C3=A9g=C3=A9 par le secret professionnel. Si vous recevez = ce message par erreur, merci d'en avertir imm=C3=A9diatement = l'exp=C3=A9diteur et de le d=C3=A9truire. L'int=C3=A9grit=C3=A9 du = message ne pouvant =C3=AAtre assur=C3=A9e sur Internet, la = responsabilit=C3=A9 de Worldline ne pourra =C3=AAtre recherch=C3=A9e = quant au contenu de ce message. Bien que les meilleurs efforts soient = faits pour maintenir cette transmission exempte de tout virus, = l'exp=C3=A9diteur ne donne aucune garantie =C3=A0 cet =C3=A9gard et sa = responsabilit=C3=A9 ne saurait =C3=AAtre recherch=C3=A9e pour tout = dommage r=C3=A9sultant d'un virus transmis.

This e-mail and the = documents attached are confidential and intended solely for the = addressee; it may also be privileged. If you receive this e-mail in = error, please notify the sender immediately and destroy it. As its = integrity cannot be secured on the Internet, the Worldline liability = cannot be triggered for the message content. Although the sender = endeavours to maintain a computer virus-free network, the sender does = not warrant that this transmission is virus-free and will not be liable = for any damages resulting from any virus transmitted.



Ce = message et les pi=C3=A8ces jointes sont confidentiels et r=C3=A9serv=C3=A9= s =C3=A0 l'usage exclusif de ses destinataires. Il peut =C3=A9galement = =C3=AAtre prot=C3=A9g=C3=A9 par le secret professionnel. Si vous recevez = ce message par erreur, merci d'en avertir imm=C3=A9diatement = l'exp=C3=A9diteur et de le d=C3=A9truire. L'int=C3=A9grit=C3=A9 du = message ne pouvant =C3=AAtre assur=C3=A9e sur Internet, la = responsabilit=C3=A9 de Worldline ne pourra =C3=AAtre recherch=C3=A9e = quant au contenu de ce message. Bien que les meilleurs efforts soient = faits pour maintenir cette transmission exempte de tout virus, = l'exp=C3=A9diteur ne donne aucune garantie =C3=A0 cet =C3=A9gard et sa = responsabilit=C3=A9 ne saurait =C3=AAtre recherch=C3=A9e pour tout = dommage r=C3=A9sultant d'un virus transmis.

This e-mail and the = documents attached are confidential and intended solely for the = addressee; it may also be privileged. If you receive this e-mail in = error, please notify the sender immediately and destroy it. As its = integrity cannot be secured on the Internet, the Worldline liability = cannot be triggered for the message content. Although the sender = endeavours to maintain a computer virus-free network, the sender does = not warrant that this transmission is virus-free and will not be liable = for any damages resulting from any virus = transmitted.

= --Apple-Mail=_C761A09E-B7DF-4E9B-8A03-250569BB25A6--