cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DuyHai Doan <doanduy...@gmail.com>
Subject Re: Hot, large row
Date Thu, 24 Jul 2014 19:01:51 GMT
"How can I detect wide rows?" -->

nodetool cfhistograms <keyspace> <suspected column family>

Look at column "Column count" (last column) and identify a line in this
column with very high value of "Offset". In a well designed application you
should have a gaussian distribution where 80% of your row have a similar
number of columns.

"Anyone know what debug level I can set so that I can see what reads the
hot node is handling?  " -->

"nodetool settraceprobability <value>",  where value is a small number
(0.001) on the node where you encounter the issue. Activate the tracing for
a while (5 mins) and deactivate it (value = 0). Then look into
system_traces tables "events" & "sessions". It may help or not since the
tracing is done once every 1000.

"Any way to get the server to blacklist these wide rows automatically?" -->
No


On Thu, Jul 24, 2014 at 8:48 PM, Keith Wright <kwright@nanigans.com> wrote:

> Hi all,
>
>    We are seeing an issue where basically daily one of our nodes spikes in
> load and is churning in CMS heap pressure.  It appears that reads are
> backing up and my guess is that our application is reading a large row
> repeatedly.  Our write structure can lead itself to wide rows very
> infrequently (<0.001%) and we do our best to detect and delete them but
> obviously we’re missing a case.  Hoping for assistance on the following
> questions:
>
>    - How can I detect wide rows?
>    - Anyone know what debug level I can set so that I can see what reads
>    the hot node is handling?  I’m hoping to see the “bad” row
>    - Any way to get the server to blacklist these wide rows automatically?
>
> We’re using C* 2.0.6 with Vnodes.
>
> Thanks
>

Mime
View raw message