Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 56034964B for ; Tue, 17 Apr 2012 21:46:41 +0000 (UTC) Received: (qmail 6696 invoked by uid 500); 17 Apr 2012 21:46:39 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 6624 invoked by uid 500); 17 Apr 2012 21:46:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 6616 invoked by uid 99); 17 Apr 2012 21:46:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Apr 2012 21:46:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ben.coverston@datastax.com designates 209.85.212.178 as permitted sender) Received: from [209.85.212.178] (HELO mail-wi0-f178.google.com) (209.85.212.178) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Apr 2012 21:46:30 +0000 Received: by wibhq7 with SMTP id hq7so6395wib.7 for ; Tue, 17 Apr 2012 14:46:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=RreruUMJnHxTyK3Nq/sdh/ssQb0Uk0ZRv6fbaleJQ9E=; b=DryKNwVfdIv28eQTFer1sHqCYtir35rIPJataHDnZ72MTOC8GNkoys+oK5MdDfReYY zVp88GZ3RilKvS4smvkvja5VnsM7VwbS+hhXLL/nYtAHeAraE4yWejACGlzb6DYQQg3t dWfFfnSADD0/fEqHdamvzufW4BMwlR2I/s+9VZEDvfJfMPqcxLa8DaGXoUVJrTZkPKpc KaFc7hcGJZ+zLqdVd6GvgsoeBZkvD9qacjTB6ejN56agA/KGR8PpDx+fGgBcqgiBs+LV IUmc8gv+Lals8SFw0ADPyp8UW6ojnj4IOqs9uwhrF4Hhe8i/HyvK4Ba/rzIzY6zPYd1p Mb4A== Received: by 10.216.144.149 with SMTP id n21mr9529072wej.42.1334699170290; Tue, 17 Apr 2012 14:46:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.183.19 with HTTP; Tue, 17 Apr 2012 14:45:50 -0700 (PDT) In-Reply-To: <4F8D7EF6.3000901@bnl.gov> References: <4F8CC20F.2010505@bnl.gov> <3293FEFB-B409-46C8-A197-9D5CD96F6B6E@thelastpickle.com> <4F8D745E.3060004@bnl.gov> <4F8D7808.8060900@bnl.gov> <4F8D797C.9020808@bnl.gov> <4F8D7B5A.4020501@bnl.gov> <4F8D7EF6.3000901@bnl.gov> From: Ben Coverston Date: Tue, 17 Apr 2012 15:45:50 -0600 Message-ID: Subject: Re: Is the secondary index re-built under compaction? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e6dee843ed5c3704bde6dded X-Gm-Message-State: ALoCoQmLRNyGQ/UGf0C3m/qxvTnNPrhpq92IbSAH2gEho9lBMruggOnle5aiK00Q7jBIFmq2aVef --0016e6dee843ed5c3704bde6dded Content-Type: text/plain; charset=ISO-8859-1 Every time a repair creates a new SSTable segment you'll get another index build compaction for the keys in that segment. On Tue, Apr 17, 2012 at 8:32 AM, Maxim Potekhin wrote: > The "offending" CF only has one. The other one, that seems to behave well, > has nine. > > Maxim > > > > On 4/17/2012 10:20 AM, Jake Luciani wrote: > > How many indexes are there? > > On Tue, Apr 17, 2012 at 10:16 AM, Maxim Potekhin wrote: > >> Yes. Sorry I didn't mention this, but of course I'm checking on indexes >> once in a while. >> So yes, they are marked as built. >> >> All of this started happening after a few days of continuous loading >> process. Since >> the nodes have good hardware (24 cores + SSD), the apparent load on each >> node >> was nothing remarkable, even at 20kHz insertion rate. But maybe I'm being >> overoptimistic. >> >> Maxim >> >> >> >> On 4/17/2012 10:12 AM, Jake Luciani wrote: >> >> Hmm that does sound fishy. >> >> When you run show keyspaces from cassandra-cli it shows which indexes >> are built. Are they marked built in your column family? >> >> -Jake >> >> On Tue, Apr 17, 2012 at 10:09 AM, Maxim Potekhin wrote: >> >>> I understand that indexes are CFs. But the compaction stats says it's >>> building the >>> index, not compacting the corresponding CF. Either that's an ambiguous >>> diagnostic, >>> or indeed something is not right with my rig as of late. >>> >>> Maxim >>> >>> >>> >>> >>> On 4/17/2012 10:05 AM, Jake Luciani wrote: >>> >>> Well, the since the secondary indexes are themselves >>> column families they too are compacted along with everything else. >>> >>> On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin wrote: >>> >>>> Thanks Jake. Then I am definitely seeing weirdness, as there are tons >>>> of >>>> "pending tasks" in compaction stats, and tons of index files created in >>>> the >>>> data directory. Plus it does tell me that it is building the secondary >>>> index, >>>> and that seems to be happening at an amazingly glacial pace. >>>> >>>> I have 2 CFs there, with multiple secondary indexes. I'll try >>>> to compact the CF one by one, reboot and see if that helps. >>>> >>>> Maxim >>>> >>>> >>>> >>>> On 4/17/2012 9:53 AM, Jake Luciani wrote: >>>> >>>> No, the indexes are not rebuilt every compaction. Only if you manually >>>> rebuild or bootstrap a new node does it use compaction manager to rebuild. >>>> >>>> On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin wrote: >>>> >>>>> Thanks Aaaron. Just to be clear, every time I do a compaction, >>>>> I rebuild all indexes from scratch. Right? >>>>> >>>>> Maxim >>>>> >>>>> >>>>> >>>>> On 4/17/2012 6:16 AM, aaron morton wrote: >>>>> >>>>> Yes secondary index builds are done via the compaction manager. >>>>> >>>>> Cheers >>>>> >>>>> ----------------- >>>>> Aaron Morton >>>>> Freelance Developer >>>>> @aaronmorton >>>>> http://www.thelastpickle.com >>>>> >>>>> On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote: >>>>> >>>>> I noticed that "nodetool compactionstats" shows the building of the >>>>> secondary index while >>>>> I initiate compaction. Is this to be expected? Cassandra version 0.8.8. >>>>> >>>>> Thank you >>>>> >>>>> Maxim >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> http://twitter.com/tjake >>>> >>>> >>>> >>> >>> >>> -- >>> http://twitter.com/tjake >>> >>> >>> >> >> >> -- >> http://twitter.com/tjake >> >> >> > > > -- > http://twitter.com/tjake > > > -- Ben Coverston DataStax -- The Apache Cassandra Company --0016e6dee843ed5c3704bde6dded Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Every time a repair creates a new SSTable segment you'll get another in= dex build compaction for the keys in that segment.

On Tue, Apr 17, 2012 at 8:32 AM, Maxim Potekhin <potekhin@bnl.gov> wr= ote:
=20 =20 =20
The "offending" CF only has one. The other one, that seems to= behave well,
has nine.

Maxim



On 4/17/2012 10:20 AM, Jake Luciani wrote:
How many indexes are there?

On Tue, Apr 17, 2012 at 10:16 AM, Maxim Potekhin <potekhin@bnl.gov> wrote:
Yes. Sorry I didn'= t mention this, but of course I'm checking on indexes once in a while.
So yes, they are marked as built.

All of this started happening after a few days of continuous loading process. Since
the nodes have good hardware (24 cores + SSD), the apparent load on each node
was nothing remarkable, even at 20kHz insertion rate. But maybe I'm being overoptimistic.

Maxim



On 4/17/2012 10:12 AM, Jake Luciani wrote:
Hmm that does sound fishy.=A0

When you run show keyspaces from cassandra-cli it shows which indexes are built. =A0Are they marked built in your column family?

-Jake

On Tue, Apr 17, 2012 at 10:09 AM, Maxim Potekhin <potekhin@bnl.gov> wrote:
I understand that indexes are CFs. But the compaction stats says it's building the
index, not compacting the corresponding CF. Either that's an ambiguous diagnostic,
or indeed something is not right with my rig as of late.

Maxim




On 4/17/2012 10:05 AM, Jake Luciani wrote:
Well, the since the secondary indexes are themselves column=A0families=A0they too are compacted along with everything else.

On Tue, Apr 17, 2012 at 10:02 AM, Maxim Potekhin <potekhi= n@bnl.gov> wrote:
Thanks Jake. Then I am definitely seeing weirdness, as there are tons of
"pending tasks" in compacti= on stats, and tons of index files created in the
data directory. Plus it does tell me that it is building the secondary index,
and that seems to be happening at an amazingly glacial pace.

I have 2 CFs there, with multiple secondary indexes. I'll try
to compact the CF one by one, reboot and see if that helps.

Maxim



On 4/17/2012 9:53 AM, Jake Luciani wrote:
No, the indexes are not rebuilt every compaction. =A0Only if you manually rebuild or bootstrap a new node does it use compaction manager to rebuild.

On Tue, Apr 17, 2012 at 9:47 AM, Maxim Potekhin <potekhin= @bnl.gov> wrote:
Thanks Aaaron. Just to be clear, every time I do a compaction,
I rebuild all indexes from scratch. Right?

Maxim



On 4/17/2012 6:16 AM, aaron morton wrote:
Yes secondary index builds are done via the compaction manager.=A0

Cheers

------------= -----
Aaron Morton
Freelance Developer
@aaronmorton=

On 17/04/2012, at 1:06 PM, Maxim Potekhin wrote:

I noticed that "nodeto= ol compactionstats&q= uot; shows the building of the secondary index while
I initiate compaction. Is this to be expected? Cassandra version 0.8.8.
Thank you

Maxim






--
http://twitter.com/tjake




--
http://twitter.com/tjake




--
= http://twitter.com/tjake




--
http://twitter= .com/tjake




--
Ben Coversto= n
DataStax -- The Apache Cassandra Company

--0016e6dee843ed5c3704bde6dded--