Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D7F4C10C75 for ; Fri, 14 Mar 2014 12:51:15 +0000 (UTC) Received: (qmail 33062 invoked by uid 500); 14 Mar 2014 12:51:14 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 31812 invoked by uid 500); 14 Mar 2014 12:51:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 31803 invoked by uid 99); 14 Mar 2014 12:51:07 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Mar 2014 12:51:07 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [98.136.216.182] (HELO nm29-vm7.bullet.mail.gq1.yahoo.com) (98.136.216.182) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Mar 2014 12:51:00 +0000 Received: from [98.137.12.59] by nm29.bullet.mail.gq1.yahoo.com with NNFMP; 14 Mar 2014 12:50:39 -0000 Received: from [98.137.12.236] by tm4.bullet.mail.gq1.yahoo.com with NNFMP; 14 Mar 2014 12:50:39 -0000 Received: from [127.0.0.1] by omp1044.mail.gq1.yahoo.com with NNFMP; 14 Mar 2014 12:50:39 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 450489.96516.bm@omp1044.mail.gq1.yahoo.com Received: (qmail 58736 invoked by uid 60001); 14 Mar 2014 12:50:39 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1394801439; bh=8SPYYGoJ6RQMpNA2Me4ZhOOON6sEUSS7a29Irltelrs=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=xPUF2/Y/lPWfRl3YNxEd/cnik8spzPDezAYryGvbH6jiqseU+u+gZ6i0A9TEmnrtMn1/2oIJbjY9AklbEI1DwgfcuAYJk+tsTJYYlg1nkHlX2DyaDFrTXwExd3Su3R7EdhbGcoC9TWgJNG03p51vKXhJgC3vKFWOSZkrzI4CLfA= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=lWVTFI1R2pohg1j4wTyhQwqdqmCri4tyiT4d1FVKTrPz+aLcCli0hf0A058++p2BCuAM7xFEFbK2Y00heawQfLMWm3pRT/ZeGUMZLbq3fURuHtfAIBJl+lDRS5MRs0VtH6hAJzd5gjOciH3k0sQhgWY232idHCljvYKBTXBoOsw=; X-YMail-OSG: SeqHF3IVM1mL_DTp2NB2z1hmnq0Mjenl5XJrlu4QZeFZxGP cSUeffCr6.Ya0wlUKjg392e0VIfXU0G1OwV8XXk.yfTtS6.gAM8sGr2HVLiG MbXRVoRvpuBXkDGiKErEUp58KpDmbowJgMO.JkwH_ee_IPb7qUc.wWy0.Cod wsedBLXVXM7Vd09eOMjGvJXJByr8hIbscnA1hUfxCguduTQCvlETuDXi00Kt rIKwvA0TJKdNNe9AjOlct9vtpDFaQElE1TmqBNkoJ.Xta4mnjkwon27qzHFH M2vXZR.Vc3yyc.3pziu_vj8I0uU4KyOeFsgczbj7KfD.lhkjE_.lqlKOJsWs bAyv5rWD35JbGLT2o62QKIG1A_iUOG0odmTSfoe_eL2kOOCXGgo9koA_MOAo QN6gwga0g2GTbE6.hZiG6gLK6JQwkH2nn.QtrrkOftvg554DuIvL_U8MhadR tjB34U4QioUmh0GQEx5JHnHHR.cRZC1f8E21hdTLtteJ4UnpIZDlt.MA.9jQ qFOfKmcSguA2QFKksPDblH4PLcmkg3ncX_2GxBliPOUqwF9g0YY0tj7PqqO6 5PA0t Received: from [81.196.95.134] by web164502.mail.gq1.yahoo.com via HTTP; Fri, 14 Mar 2014 05:50:39 PDT X-Rocket-MIMEInfo: 002.001,SGVsbG8gYWxsLAoKSGVyZSBpcyB0aGUgZW52aXJvbm1lbnQ6CgpJIGhhdmUgYSA2IG5vZGUgQ2Fzc2FuZHJhIGNsdXN0ZXIuIE9uIGVhY2ggbm9kZSBJIGhhdmU6Ci0gMzIgRyBSQU0KLSAyNCBHIFJBTSBmb3IgY2Fzc2EKLSB.MTUwIC0gMjAwIE1CL3MgZGlzayBzcGVlZAotIHRvbWNhdCA2IHdpdGggYXhpczIgd2Vic2VydmljZSB0aGF0IHVzZXMgdGhlIGRhdGFzdGF4IGphdmEgZHJpdmVyIHRvIG1ha2UKYXN5bmNoIHJlYWRzIC8gd3JpdGVzwqAKLSByZXBsaWNhdGlvbiBmYWN0b3IgZm9yIHRoZSBrZXlzcGEBMAEBAQE- X-Mailer: YahooMailWebService/0.8.178.641 Message-ID: <1394801439.6654.YahooMailNeo@web164502.mail.gq1.yahoo.com> Date: Fri, 14 Mar 2014 05:50:39 -0700 (PDT) From: Batranut Bogdan Reply-To: Batranut Bogdan Subject: Cassandra slow on some reads To: "user@cassandra.apache.org" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-2109934955-1525087398-1394801439=:6654" X-Virus-Checked: Checked by ClamAV on apache.org ---2109934955-1525087398-1394801439=:6654 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hello all,=0A=0AHere is the environment:=0A=0AI have a 6 node Cassandra clu= ster. On each node I have:=0A- 32 G RAM=0A- 24 G RAM for cassa=0A- ~150 - 2= 00 MB/s disk speed=0A- tomcat 6 with axis2 webservice that uses the datasta= x java driver to make=0Aasynch reads / writes=A0=0A- replication factor for= the keyspace is 3=0A=0A(I know that there is a lot of heap but I also have= write heavy tasks and I want them to get into mem fast) .=0A=0AAll nodes i= n the same data center=A0=0AThe clients that read / write are in the same d= atacenter so network is=A0Gigabit.=0A=0A=0AThe table structure is like this= : PK(key String , timestam int, column1 string, col2 string) , list1 , list= 2, list 3 .=0AThere are about 300 milions individual keys.=0AThere are abo= ut 100 timestamps for each key now, so the rows will get wider as time pass= es.=0A=0AI am using datastax java driver to query the cluster.=0A=0AI have = ~450 queries that are like this: SELECT * FROM table where key =3D 'some st= ring' and ts =3D some value; some value is close to present time.=0A=0AThe = problem:=0A=0AAbout 10 - 20 % of these queries take more than 5 seconds to = execute, in fact, the majority of those take around 10 seconds.=0AWhen inve= stigating I saw that if I have a slow response and I redo the query it will= finish in 8 - 10 MILIseconds like the rest of the queries that I have.=A0= =0AI could not see using JConsole any spikes in CPU / memory when executing= the queries. The rise in resource consumtion is very small on all nodes on= the cluster. I expect such delays to be generated by a BIG increase in res= ource consumption.=0A=0AAny comments will be appreciated.=0A=0AThank you. ---2109934955-1525087398-1394801439=:6654 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Hello all,

Here is the environment:

I have a 6 n= ode Cassandra cluster. On each node I have:
- 32 G RAM
- 24 G RA= M for cassa
- ~150 - 200 MB/s di= sk speed
- tomcat 6 with axis2 w= ebservice that uses the datastax java driver to make
asynch reads / writes 
- replication factor for the keyspace is 3

(I know that there is a l= ot of heap but I also have write heavy tasks and I want them to get into mem fast) .
All nodes in the same data center 
The clients that read / write are= in the same datacenter so network is Gigabit.

The table structure is like this: PK(key String , timestam int, colum= n1 string, col2 string) , list1 , list 2, list 3 .
There are about 300 milions individual keys= .
There are about= 100 timestamps for each key now, so the rows will get wider as time passes= .

I am using datastax java driver to query the cluster.

=
I have ~450 quer= ies that are like this: SELECT * FROM table where key =3D 'some string' and= ts =3D some value; some value is close to present time.

The problem:

About 10 - 20 % of these queries take more than 5 seconds to execute, = in fact, the majority of those take around 10 seconds.
When investigating I saw that if I ha= ve a slow response and I redo the query it will finish in 8 - 10 MILIsecond= s like the rest of the queries that I have. 
I could not see using JConsole any spikes i= n CPU / memory when executing the queries. The rise in resource consumtion = is very small on all nodes on the cluster. I expect such delays to be gener= ated by a BIG increase in resource consumption.

Any comments will be appreciated.

Thank you.

 


---2109934955-1525087398-1394801439=:6654--