Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 31BF4D23C for ; Tue, 21 May 2013 08:35:22 +0000 (UTC) Received: (qmail 69016 invoked by uid 500); 21 May 2013 08:35:20 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 68659 invoked by uid 500); 21 May 2013 08:35:19 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 68607 invoked by uid 99); 21 May 2013 08:35:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 May 2013 08:35:16 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a47.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 May 2013 08:35:13 +0000 Received: from homiemail-a47.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTP id 9955228405B for ; Tue, 21 May 2013 01:34:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=mMMPOODLS7U1MHSxOhV6YbeFEG k=; b=hI2fFs/meZ1HwsG/d24rizqxLfbUCwffjsaB5x6hu4FIB/8wrXSRfBJufb p0Dy4t7iDs9UEQwHb/cAMw7BD3ny++QWAVysjmTURlo7nDbQOcKLEyaGhUioTJVD 89OphzCMneejWlnb1ANd6SWK47ZUpeIqZg1s9gbn4iLf4VGOE= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTPSA id D8F51284058 for ; Tue, 21 May 2013 01:34:51 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_DC38D8B7-2D8B-45EB-A8F0-4701C0BDFE00" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Logging Cassandra queries Date: Tue, 21 May 2013 20:34:50 +1200 References: <1ECC2E1A-D2F4-418A-BCDF-6EE0C8C7EE5A@thelastpickle.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1503) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_DC38D8B7-2D8B-45EB-A8F0-4701C0BDFE00 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > (but still the list of classes would be helpful :) ) The source code is the most up to date list of the classes. This is a talk I did on Cassandra Internals at Apache Con in Feb 2013, = unfortunately it looks like the videos will never be released owing to = technical snafu's = http://www.slideshare.net/aaronmorton/apachecon-nafeb2013 Cass 1.2 also has probabilistic logging of queries.=20 I'm doing a cut down version at the Cassandra SF conference in June.=20 Cheers =20 ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 21/05/2013, at 10:28 AM, Ilya Kirnos wrote: > sure, i think it'd be a useful feature >=20 >=20 > On Sat, May 18, 2013 at 4:13 PM, Tom=E0s N=FAnez = wrote: > If you're looking for logging like "get keyX with CL quorum and slice = Y took n millis"=20 >=20 > That would be even better! Maybe should I file a ticket in Cassandra = Jira for this feature? Do you think it would be helpful? >=20 > BTW, just "get keyX" or "set keyX" would work for me. I'll check = org.apache.cassandra.thrift.CassandraServer as Aaron suggested (but = still the list of classes would be helpful :) ) >=20 > Thanks! >=20 > 2013/5/19 Ilya Kirnos > If you're looking for logging like "get keyX with CL quorum and slice = Y took n millis" there's nothing like that from what I could find. We = had to modify c* source (CassandraServer.java) to add this query logging = to the thrift codepath. >=20 > On May 18, 2013 3:20 PM, "Tom=E0s N=FAnez" = wrote: > Yes, I read how to do that here, as well: > http://www.datastax.com/docs/1.1/configuration/logging_options >=20 > But I didn't know what classes to enable logging for the queries... Is = there any document with the list of classes with a bit explanation for = each of them? I can't find any, and I don't understand java enough to = dive through the code.... >=20 > Thanks! >=20 >=20 > 2013/5/17 aaron morton >> And... could I be more precise when enabling logging? Because right = now, with log4j.rootLogger=3DDEBUG,stdout,R I'm getting a lot of = information I won't use ever, and I'd like to enable just what I need to = see gets and seds=85. >=20 > see the example at the bottom of this file about setting the log level = for a single class = https://github.com/apache/cassandra/blob/trunk/conf/log4j-server.propertie= s >=20 > You probably want to set it for the = org.apache.cassandra.thrift.CassandraServer class. But I cannot remember = what the logging is like in 0.8.=20 >=20 > Cassandra gets faster in the later versions, which normally means = doing less work. Upgrading to 1.1 would be the first step I would take = in improving performance. =20 >=20 > Cheers >=20 > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand >=20 > @aaronmorton > http://www.thelastpickle.com >=20 > On 18/05/2013, at 4:00 AM, Tom=E0s N=FAnez = wrote: >=20 >> Hi! >>=20 >> For quite time I've been having some unexpected loadavg in the = cassandra servers. I suspect there are lots of uncontrolled queries to = the cassandra servers causing this load, but the developers say that = there are none, and the load is due to cassandra internal processes.=20 >>=20 >> Trying to get to the bottom, I've been looking into completed = ReadStage and MutationStage through JMX, and the numbers seem to confirm = my theory, but I'd like to go one step forward and, if possible, list = all the queries from the webservers to the cassandra cluster (just one = node would be enough).=20 >>=20 >> I've been playing with cassandra loglevels, and I can see when a Read = or a Write is done, but it would be better if I could knew the CF of the = query. For my tests I've put the in the log4j.server " = log4j.rootLogger=3DDEBUG,stdout,R", writing and reading a test CF, and I = can't see the name of it anywhere. >>=20 >> For the test I'm using Cassandra 0.8.4 (yes, still), as my production = servers, and also 1.0.11. Maybe this changes in 1.1? Maybe I'm doing = something wrong? Any hint? >>=20 >> And... could I be more precise when enabling logging? Because right = now, with log4j.rootLogger=3DDEBUG,stdout,R I'm getting a lot of = information I won't use ever, and I'd like to enable just what I need to = see gets and seds.... >>=20 >> Thanks in advance,=20 >> Tom=E0s >>=20 >=20 >=20 >=20 >=20 >=20 > --=20 > -ilya --Apple-Mail=_DC38D8B7-2D8B-45EB-A8F0-4701C0BDFE00 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
 (but still the list of = classes would be helpful :) = )
The source code is the most = up to date list of the classes.

This is a talk I did = on Cassandra Internals at Apache Con in Feb 2013, unfortunately it looks = like the videos will never be released owing to technical = snafu's http://= www.slideshare.net/aaronmorton/apachecon-nafeb2013

Cass 1.2 also has probabilistic logging of = queries. 

I'm doing a cut down version at = the Cassandra SF conference in = June. 

Cheers
 
http://www.thelastpickle.com

On 21/05/2013, at 10:28 AM, Ilya Kirnos <ilya@cardspring.com> = wrote:

sure, i think it'd be a useful = feature


On Sat, May 18, 2013 at 4:13 PM, Tom=E0s N=FAnez = <tomas.nunez@groupalia.com> wrote:
If you're looking for logging like "get keyX with CL quorum and slice Y = took n millis" 

That would be even better! Maybe should I file = a ticket in Cassandra Jira for this feature? Do you think it would be = helpful?

BTW, just "get keyX" or "set keyX" = would work for me. I'll = check org.apache.cassandra.thrift.CassandraServer as Aaron = suggested (but still the list of classes would be helpful :) )

Thanks!

2013/5/19 Ilya = Kirnos <ilya@cardspring.com>

If you're looking for = logging like "get keyX with CL quorum and slice Y took n millis" there's = nothing like that from what I could find.  We had to modify c* = source (CassandraServer.java) to add this query logging to the thrift = codepath.

On May 18, 2013 3:20 PM, "Tom=E0s N=FAnez" = <tomas.nunez@groupalia.com> wrote:
Yes, I read how to do that here, as well:

But I didn't know what classes to enable logging for the queries... = Is there any document with the list of classes with a bit explanation = for each of them? I can't find any, and I don't understand java enough = to dive through the code....

Thanks!


2013/5/17 aaron morton <aaron@thelastpickle.com>
And... could I be more precise when enabling logging? = Because right now, = with log4j.rootLogger=3DDEBUG,stdout,R I'm getting a = lot of information I won't use ever, and I'd like to enable just what I = need to see gets and seds=85.

see the example at the bottom of this file about = setting the log level for a single class https://github.com/apache/cassandra/blob/trunk/conf/log4= j-server.properties

You probably want to set it for the = org.apache.cassandra.thrift.CassandraServer class. But I cannot remember = what the logging is like in = 0.8. 

Cassandra gets faster in the later = versions, which normally means doing less work. Upgrading to 1.1 would = be the first step I would take in improving performance.  

Cheers

-----------------
Aaron Morton
Freelance = Cassandra Consultant
New = Zealand

@aaronmorton

On 18/05/2013, at 4:00 AM, Tom=E0s N=FAnez <tomas.nunez@groupalia.com> = wrote:

Hi!

For quite time I've been having some unexpected loadavg in the = cassandra servers. I suspect there are lots of uncontrolled queries to = the cassandra servers causing this load, but the developers say that = there are none, and the load is due to cassandra internal = processes. 

Trying to get to the bottom, I've been looking into = completed ReadStage and MutationStage through JMX, and the numbers seem = to confirm my theory, but I'd like to go one step forward and, if = possible, list all the queries from the webservers to the cassandra = cluster (just one node would be enough). 

I've been playing with cassandra loglevels, and I = can see when a Read or a Write is done, but it would be better if I = could knew the CF of the query. For my tests I've put the in the = log4j.server " log4j.rootLogger=3DDEBUG,stdout,R", writing and = reading a test CF, and I can't see the name of it anywhere.

For the test I'm using Cassandra 0.8.4 (yes, still), = as my production servers, and also 1.0.11. Maybe this changes in 1.1? = Maybe I'm doing something wrong? Any hint?

And... could I be more precise when enabling logging? Because right now, = with log4j.rootLogger=3DDEBUG,stdout,R I'm getting a lot of = information I won't use ever, and I'd like to enable just what I need to = see gets and seds....

Thanks in = advance, 
Tom=E0s

=





--
-ilya

= --Apple-Mail=_DC38D8B7-2D8B-45EB-A8F0-4701C0BDFE00--