cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wright <>
Subject Re: Hot, large row
Date Mon, 28 Jul 2014 13:48:58 GMT
I don’t know but my guess is it would be without tombstones.  I did more research this weekend
(note that my Sunday was largely interrupted by again seeing a node go to high load/high CMS
for ~3 hours) and came across this presentation:

I definitely suggestion you give this a look, very informative.  The important take away is
that they ran into the same issue as I due to using the same model where I am updating to
the same row over time with a TTL causing that row to fragment across SSTables and once across
4+ tables, compaction can never actually remove tombstones.  As I see it, I have the following
options and was hoping to get some advice:

1.  Modify my write structure to include time within the key.  Currently we want to get all
of a row but I can likely add month to the time and it would be ok for the application to
do two reads to get the most recent data (to deal with month boundaries).  This will contain
the fragmentation to one month.

2.  Following off of item #1, it appears that according to CASSANDRA-5514 that if I include
time within my query it will not bother going through older SSTables and thus reduce the impact
of the row fragmentation.  Problem here is that likely my data space will still continue to
grow over time as tombstones will never be removed.

3.  Move from LCS to STCS and run full compactions periodically to cleanup tombstones

I appreciate the help!

From: Jack Krupansky <<>>
Reply-To: "<>" <<>>
Date: Friday, July 25, 2014 at 11:15 AM
To: "<>" <<>>
Subject: Re: Hot, large row

Is it the accumulated tombstones on a row that make it act as if “wide”? Does cfhistograms
count the tombstones or subtract them when reporting on cell-count for rows? (I don’t know.)

-- Jack Krupansky

From: Keith Wright<>
Sent: Friday, July 25, 2014 10:24 AM
Cc: Don Jackson<>
Subject: Re: Hot, large row

Ha, check out who filed that ticket!   Yes I’m aware of it.  My hope is that it was mostly
addressed in CASSANDRA-6563 so I may upgrade from 2.0.6 to 2.0.9.  I’m really just surprised
that others are not doing similar actions as I and thus experiencing similar issues.

To answer DuyHai’s questions:

How many nodes do you have ? And how many distinct user_id roughtly is there ?
- 14 nodes with approximately 250 million distinct user_ids

For GC activity, in general we see low GC pressure in both Par New and CMS (we see the occasional
CMS spike but its usually under 100 ms).  When we see a node locked up in CMS GC, its not
that anyone GC takes a long time, its just that the consistent nature of them causes the read
latency to spike from the usual 3-5 ms up to 35 ms which causes issues for our application.

Also Jack Krupansky question is interesting. Even though you limit a request to 5000, if each
cell is a big blob or block of text, it mays add up a lot into JVM heap …
- The columns values are actually timestamps and thus not variable in length and we cap the
length of other columns used in the primary key so I find if VERY unlikely that this is a

I will look into the paging option with that native client but from the docs it appears that
its enabled by default, right?

I greatly appreciate all the help!

From: Ken Hancock <<>>
Reply-To: "<>" <<>>
Date: Friday, July 25, 2014 at 10:06 AM
To: "<>" <<>>
Cc: Don Jackson <<>>
Subject: Re: Hot, large row

View raw message