incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Kjellman <mkjell...@barracuda.com>
Subject Re: persistent compaction issue (1.1.4 and 1.1.5)
Date Tue, 18 Sep 2012 18:02:22 GMT
There are a large number of members in generation 0, which I'm assuming refers to L0 according
to a few of the .json files I checked in my largest column families.

This particular node I'm checking I have already tried a scrub and repair. What steps should
I take to move these SSTables to the next level? Looking like these compactions are indeed
legit then…

Thank you.

From: Ben Coverston <ben.coverston@datastax.com<mailto:ben.coverston@datastax.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: persistent compaction issue (1.1.4 and 1.1.5)

In your data directory there should be a .json file for each column family that holds the
manifest.

Do any of those indicate that you have a large number of SSTables in L0?

This number is also indicated in JMX by the UnLeveledSSTables count for each column family.

If not it's possible that the number is coming from SSTable fragments that are the result
of a repair/decommission.



On Tue, Sep 18, 2012 at 8:57 AM, Michael Kjellman <mkjellman@barracuda.com<mailto:mkjellman@barracuda.com>>
wrote:
Leveled. nothing in the logs. Normal compactions seem to be occurring...these ones just won't
go away.

I've tried a rolling restart and literally tries killing our entire cluster and bringing up
one node at a time in case gossip was causing this. Same result.

The compactions are there immediately after Thrift starts listening for clients.

Thanks Aaron

On Sep 18, 2012, at 3:54 AM, "aaron morton" <aaron@thelastpickle.com<mailto:aaron@thelastpickle.com><mailto:aaron@thelastpickle.com<mailto:aaron@thelastpickle.com>>>
wrote:

What Compaction Strategy are you using ?
Are there any errors in the logs ?
If you restart a node how long does it take for the numbers to start to rise ?

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 18/09/2012, at 7:39 AM, Michael Kjellman <mkjellman@barracuda.com<mailto:mkjellman@barracuda.com><mailto:mkjellman@barracuda.com<mailto:mkjellman@barracuda.com>>>
wrote:

Hi All,

I have an issue where each one of my nodes (currently all running at 1.1.5) is reporting around
30,000 pending compactions. I understand that a pending compaction doesn't necessarily mean
it is a scheduled task however I'm confused why this behavior is occurring. It is the same
on all nodes, occasionally goes down 5k pending compaction tasks, and then returns to 25,000-35,000
compaction tasks pending.

I have tried a repair operation/scrub operation on two of the nodes and while compactions
initially happen the number of pending compactions does not decrease.

Any ideas? Thanks for your time.

Best,
michael


'Like' us on Facebook for exclusive content and other resources on all Barracuda Networks
solutions.
Visit http://barracudanetworks.com/facebook




'Like' us on Facebook for exclusive content and other resources on all Barracuda Networks
solutions.
Visit http://barracudanetworks.com/facebook





--
Ben Coverston
DataStax -- The Apache Cassandra Company


'Like' us on Facebook for exclusive content and other resources on all Barracuda Networks
solutions.
Visit http://barracudanetworks.com/facebook



Mime
View raw message