Every time a repair creates a new SSTable segment you'll get another index build compaction for the keys in that segment.

On Tue, Apr 17, 2012 at 8:32 AM, Maxim Potekhin <potekhin@bnl.gov> wrote:
The "offending" CF only has one. The other one, that seems to behave well,
has nine.

Maxim



On 4/17/2012 10:20 AM, Jake Luciani wrote:
How many indexes are there?

On Tue, Apr 17, 2012 at 10:16 AM, Maxim Potekhin <potekhin@bnl.gov> wrote:
Yes. Sorry I didn't mention this, but of course I'm checking on indexes once in a while.
So yes, they are marked as built.

All of this started happening after a few days of continuous loading process. Since
the nodes have good hardware (24 cores + SSD), the apparent load on each node
was nothing remarkable, even at 20kHz insertion rate. But maybe I'm being overoptimistic.

Maxim



On 4/17/2012 10:12 AM, Jake Luciani wrote:
Hmm that does sound fishy. 

When you run show keyspaces from cassandra-cli it shows which indexes are built.  Are they marked built in your column family?

-Jake

On Tue, Apr 17, 2012 at 10:09 AM, Maxim Potekhin <potekhin@bnl.gov> wrote:
I understand that indexes are CFs. But the compaction stats says it's building the
index, not compacting the corresponding CF. Either that's an ambiguous diagnostic,
or indeed something is not right with my rig as of late.

Maxim




On 4/17/2012 10:05 AM, Jake Luciani wrote:
Well, the since the secondary indexes are themselves column families they too are compacted along with everything else.

On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin <potekhin@bnl.gov> wrote:
Thanks Jake. Then I am definitely seeing weirdness, as there are tons of
"pending tasks" in compaction stats, and tons of index files created in the
data directory. Plus it does tell me that it is building the secondary index,
and that seems to be happening at an amazingly glacial pace.

I have 2 CFs there, with multiple secondary indexes. I'll try
to compact the CF one by one, reboot and see if that helps.

Maxim



On 4/17/2012 9:53 AM, Jake Luciani wrote:
No, the indexes are not rebuilt every compaction.  Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild.

On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin <potekhin@bnl.gov> wrote:
Thanks Aaaron. Just to be clear, every time I do a compaction,
I rebuild all indexes from scratch. Right?

Maxim



On 4/17/2012 6:16 AM, aaron morton wrote:
Yes secondary index builds are done via the compaction manager. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote:

I noticed that "nodetool compactionstats" shows the building of the secondary index while
I initiate compaction. Is this to be expected? Cassandra version 0.8.8.

Thank you

Maxim






--
http://twitter.com/tjake




--
http://twitter.com/tjake




--
http://twitter.com/tjake




--
http://twitter.com/tjake




--
Ben Coverston
DataStax -- The Apache Cassandra Company