Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50DC611F0A for ; Thu, 24 Jul 2014 19:23:36 +0000 (UTC) Received: (qmail 88684 invoked by uid 500); 24 Jul 2014 19:23:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 88649 invoked by uid 500); 24 Jul 2014 19:23:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 88638 invoked by uid 99); 24 Jul 2014 19:23:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2014 19:23:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kwright@nanigans.com designates 216.82.243.203 as permitted sender) Received: from [216.82.243.203] (HELO mail1.bemta8.messagelabs.com) (216.82.243.203) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2014 19:23:26 +0000 Received: from [216.82.242.115:33865] by server-11.bemta-8.messagelabs.com id 18/58-06420-51D51D35; Thu, 24 Jul 2014 19:23:01 +0000 X-Env-Sender: kwright@nanigans.com X-Msg-Ref: server-4.tower-132.messagelabs.com!1406229780!16151914!1 X-Originating-IP: [216.166.12.7] X-StarScan-Received: X-StarScan-Version: 6.11.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 20682 invoked from network); 24 Jul 2014 19:23:00 -0000 Received: from nat.collaborationhost.net (HELO out001.collaborationhost.net) (216.166.12.7) by server-4.tower-132.messagelabs.com with RC4-SHA encrypted SMTP; 24 Jul 2014 19:23:00 -0000 Received: from AUSP01VMBX28.collaborationhost.net ([192.168.20.73]) by AUSP01MHUB52.collaborationhost.net ([10.2.69.71]) with mapi; Thu, 24 Jul 2014 14:22:33 -0500 From: Keith Wright To: "user@cassandra.apache.org" Date: Thu, 24 Jul 2014 14:22:55 -0500 Subject: Re: Hot, large row Thread-Topic: Hot, large row Thread-Index: Ac+ndJ5/3Wa5s7FJT8qzRSb8c9j3VA== Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.3.120616 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CFF6D4A833759kwrightnaniganscom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CFF6D4A833759kwrightnaniganscom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable I can see from cfhistograms that I do have some wide rows (see below). I s= et trace probability as you suggested but the output doesn=92t appear to te= ll me what row was actually read unless I missed something. I just see exe= cuting prepared statement. Any ideas how I can find the row in question? I am considering reducing read_request_timeout_in_ms: 5000 in cassandra.yam= l so that it reduces the impact when this occurs. Any help in identifying my issue would be GREATLY appreciated Cell Count per Partition 1 cells: 50449950 2 cells: 14281828 3 cells: 8093366 4 cells: 5029200 5 cells: 3103023 6 cells: 3059903 7 cells: 1903018 8 cells: 1509297 10 cells: 2420359 12 cells: 1624895 14 cells: 1171678 17 cells: 1289391 20 cells: 909777 24 cells: 852081 29 cells: 722925 35 cells: 587067 42 cells: 459473 50 cells: 358744 60 cells: 304146 72 cells: 244682 86 cells: 191045 103 cells: 155337 124 cells: 127061 149 cells: 98913 179 cells: 77454 215 cells: 59849 258 cells: 46117 310 cells: 35321 372 cells: 26319 446 cells: 19379 535 cells: 13783 642 cells: 9993 770 cells: 6973 924 cells: 4713 1109 cells: 3229 1331 cells: 2062 1597 cells: 1338 1916 cells: 773 2299 cells: 495 2759 cells: 268 3311 cells: 150 3973 cells: 100 4768 cells: 42 5722 cells: 24 6866 cells: 12 8239 cells: 9 9887 cells: 3 11864 cells: 0 14237 cells: 5 17084 cells: 1 20501 cells: 0 24601 cells: 2 29521 cells: 0 35425 cells: 0 42510 cells: 0 51012 cells: 0 61214 cells: 2 From: DuyHai Doan > Reply-To: "user@cassandra.apache.org" > Date: Thursday, July 24, 2014 at 3:01 PM To: "user@cassandra.apache.org" > Subject: Re: Hot, large row "How can I detect wide rows?" --> nodetool cfhistograms Look at column "Column count" (last column) and identify a line in this col= umn with very high value of "Offset". In a well designed application you sh= ould have a gaussian distribution where 80% of your row have a similar numb= er of columns. "Anyone know what debug level I can set so that I can see what reads the ho= t node is handling? " --> "nodetool settraceprobability ", where value is a small number (0.0= 01) on the node where you encounter the issue. Activate the tracing for a w= hile (5 mins) and deactivate it (value =3D 0). Then look into system_traces= tables "events" & "sessions". It may help or not since the tracing is done= once every 1000. "Any way to get the server to blacklist these wide rows automatically?" -->= No On Thu, Jul 24, 2014 at 8:48 PM, Keith Wright > wrote: Hi all, We are seeing an issue where basically daily one of our nodes spikes in = load and is churning in CMS heap pressure. It appears that reads are backi= ng up and my guess is that our application is reading a large row repeatedl= y. Our write structure can lead itself to wide rows very infrequently (<0.= 001%) and we do our best to detect and delete them but obviously we=92re mi= ssing a case. Hoping for assistance on the following questions: * How can I detect wide rows? * Anyone know what debug level I can set so that I can see what reads th= e hot node is handling? I=92m hoping to see the =93bad=94 row * Any way to get the server to blacklist these wide rows automatically? We=92re using C* 2.0.6 with Vnodes. Thanks --_000_CFF6D4A833759kwrightnaniganscom_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
I can see from cfhistograms that= I do have some wide rows (see below).  I set trace probability as you= suggested but the output doesn=92t appear to tell me what row was actually= read unless I missed something.  I just see executing prepared statem= ent.   Any ideas how I can find the row in question?

I am considering reducing read_request_timeout_in_ms: 5000 in = cassandra.yaml so that it reduces the impact when this occurs.
Any help in identifying my issue would be GREATLY appreciated<= /div>

Cell Count per Partition

   = ; 1 cells: 50449950

   = ; 2 cells: 14281828

   = ; 3 cells: 8093366

   = ; 4 cells: 5029200

   = ; 5 cells: 3103023

   = ; 6 cells: 3059903

   = ; 7 cells: 1903018

   = ; 8 cells: 1509297

  = 10 cells: 2420359

  = 12 cells: 1624895

  = 14 cells: 1171678

  = 17 cells: 1289391

  = 20 cells: 909777

  = 24 cells: 852081

  = 29 cells: 722925

  = 35 cells: 587067

  = 42 cells: 459473

  = 50 cells: 358744

  = 60 cells: 304146

  = 72 cells: 244682

  = 86 cells: 191045

  103 c= ells: 155337

  124 c= ells: 127061

  149 c= ells: 98913

  179 c= ells: 77454

  215 c= ells: 59849

  258 c= ells: 46117

  310 c= ells: 35321

  372 c= ells: 26319

  446 c= ells: 19379

  535 c= ells: 13783

  642 c= ells: 9993

  770 c= ells: 6973

  924 c= ells: 4713

 1109 c= ells: 3229

 1331 c= ells: 2062

 1597 c= ells: 1338

 1916 c= ells: 773

 2299 c= ells: 495

 2759 c= ells: 268

 3311 c= ells: 150

 3973 c= ells: 100

 4768 c= ells: 42

 5722 c= ells: 24

 6866 c= ells: 12

 8239 c= ells: 9

 9887 c= ells: 3

11864 cells:= 0

14237 cells:= 5

17084 cells:= 1

20501 cells:= 0

24601 cells:= 2

29521 cells:= 0

35425 cells:= 0

42510 cells:= 0

51012 cells:= 0

61214 cells:= 2


From: DuyHai Doan <doanduyhai@gmai= l.com>
Reply-To: "<= a href=3D"mailto:user@cassandra.apache.org">user@cassandra.apache.org&q= uot; <user@cassandra.apache= .org>
Date: Thursday, Ju= ly 24, 2014 at 3:01 PM
To: &quo= t;user@cassandra.apache.org" <user@cassandra.apa= che.org>
Subject: Re: Ho= t, large row

"How c= an I detect wide rows?" -->

nodetool cfhistograms <= keyspace> <suspected column family>

Look at column &q= uot;Column count" (last column) and identify a line in this column wit= h very high value of "Offset". In a well designed application you= should have a gaussian distribution where 80% of your row have a similar n= umber of columns.

"Anyone know what debug level I can set = so that I can see what reads the hot node is handling?  " --><= br>
"nodetool settraceprobability <value>",&n= bsp; where value is a small number (0.001) on the node where you encounter = the issue. Activate the tracing for a while (5 mins) and deactivate it (val= ue =3D 0). Then look into system_traces tables "events" & &qu= ot;sessions". It may help or not since the tracing is done once every = 1000.

"Any way to get the server to blacklist these= wide rows automatically?" --> No


On Thu, Jul 24, 2014 at 8:48 PM= , Keith Wright <kwright@nanigans.com> wrote:
Hi all,

   We are seeing an issue where basically daily one of o= ur nodes spikes in load and is churning in CMS heap pressure.  It appe= ars that reads are backing up and my guess is that our application is readi= ng a large row repeatedly.  Our write structure can lead itself to wid= e rows very infrequently (<0.001%) and we do our best to detect and dele= te them but obviously we=92re missing a case.  Hoping for assistance o= n the following questions:
  • How can I detect wide rows?
    • Anyone know what debug level I can set so that I can see what reads the ho= t node is handling?  I=92m hoping to see the =93bad=94 row
  • Any= way to get the server to blacklist these wide rows automatically?
    • We=92re using C* 2.0.6 with Vnodes.

    Thanks

--_000_CFF6D4A833759kwrightnaniganscom_--