Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0EB82DAAE for ; Thu, 20 Sep 2012 05:35:38 +0000 (UTC) Received: (qmail 29901 invoked by uid 500); 20 Sep 2012 05:35:35 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 29755 invoked by uid 500); 20 Sep 2012 05:35:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 29722 invoked by uid 99); 20 Sep 2012 05:35:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Sep 2012 05:35:34 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tivv00@gmail.com designates 209.85.210.172 as permitted sender) Received: from [209.85.210.172] (HELO mail-iy0-f172.google.com) (209.85.210.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Sep 2012 05:35:29 +0000 Received: by iabz21 with SMTP id z21so1558372iab.31 for ; Wed, 19 Sep 2012 22:35:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=PdRQCEkIR7klklu2QxZ6tGDkFZtfQqkX68A4NaD/hy4=; b=BpthMsZ+PEub39C52cV4RX2O0WK35T8ZvNrrEE2lXisJuj109pwXdo5T8Dv5RCRblM YcMojaIiKPviBKknIUsv/hRi02gSZqUopy0nyTVqrZVLnYZXOdN8TAQ5JINptPOSQdt7 PgUZM0rXwzw75X6wxX2eDndOD5wsubWNdrjG/jWzlTQmKdyff/u3WuQb/2YtkpLQvz+N B8PyKMVBTvd7q+r84m9W/x8haX8AnZR5VC/VXura02RjEjasbA2aLeQmxML0AaffSoAQ zYUbQRleybzRh8Y4tvhlBUKNMs250ZNJUDZa1jNCmY2m98kVxlktAPoSnegK0ygRCLaP YE6Q== MIME-Version: 1.0 Received: by 10.50.188.129 with SMTP id ga1mr1474954igc.6.1348119309115; Wed, 19 Sep 2012 22:35:09 -0700 (PDT) Received: by 10.64.26.234 with HTTP; Wed, 19 Sep 2012 22:35:09 -0700 (PDT) In-Reply-To: References: Date: Thu, 20 Sep 2012 08:35:09 +0300 Message-ID: Subject: Re: persistent compaction issue (1.1.4 and 1.1.5) From: =?KOI8-U?B?96bUwcymyiD0yc3eydvJzg==?= To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=14dae93407f788e22e04ca1b7c8b X-Virus-Checked: Checked by ClamAV on apache.org --14dae93407f788e22e04ca1b7c8b Content-Type: text/plain; charset=KOI8-U Content-Transfer-Encoding: quoted-printable I did see problems with schema agreement on 1.1.4, but they did go away after rolling restart (BTW: it would be still good to check describe schema for unreachable). Same rolling restart helped to force compactions after moving to Leveled compaction. If your compactions still don't go, you can try removing *.json files from the data directory of the stopped node to force moving all SSTables to level0. Best regards, Vitalii Tymchyshyn 2012/9/19 Michael Kjellman > Potentially the pending compactions are a symptom and not the root > cause/problem. > > When updating a 3rd column family with a larger sstable_size_in_mb it > looks like the schema may not be in a good state > > [default@xxxx] UPDATE COLUMN FAMILY screenshots WITH > compaction_strategy=3DLeveledCompactionStrategy AND > compaction_strategy_options=3D{sstable_size_in_mb: 200}; > 290cf619-57b0-3ad1-9ae3-e313290de9c9 > Waiting for schema agreement... > Warning: unreachable nodes 10.8.30.102The schema has not settled in 10 > seconds; further migrations are ill-advised until it does. > Versions are UNREACHABLE:[10.8.30.102], > 290cf619-57b0-3ad1-9ae3-e313290de9c9:[10.8.30.15, 10.8.30.14, 10.8.30.13, > 10.8.30.103, 10.8.30.104, 10.8.30.105, 10.8.30.106], > f1de54f5-8830-31a6-9cdd-aaa6220cccd1:[10.8.30.101] > > > However, tpstats looks good. And the schema changes eventually do get > applied on *all* the nodes (even the ones that seem to have different > schema versions). There are no communications issues between the nodes an= d > they are all in the same rack > > root@xxxx:~# nodetool tpstats > Pool Name Active Pending Completed Blocked > All time blocked > ReadStage 0 0 1254592 0 > 0 > RequestResponseStage 0 0 9480827 0 > 0 > MutationStage 0 0 8662263 0 > 0 > ReadRepairStage 0 0 339158 0 > 0 > ReplicateOnWriteStage 0 0 0 0 > 0 > GossipStage 0 0 1469197 0 > 0 > AntiEntropyStage 0 0 0 0 > 0 > MigrationStage 0 0 1808 0 > 0 > MemtablePostFlusher 0 0 248 0 > 0 > StreamStage 0 0 0 0 > 0 > FlushWriter 0 0 248 0 > 4 > MiscStage 0 0 0 0 > 0 > commitlog_archiver 0 0 0 0 > 0 > InternalResponseStage 0 0 5286 0 > 0 > HintedHandoff 0 0 21 0 > 0 > > Message type Dropped > RANGE_SLICE 0 > READ_REPAIR 0 > BINARY 0 > READ 0 > MUTATION 0 > REQUEST_RESPONSE 0 > > So I'm guessing maybe the different schema versions may be potentially > stopping compactions? Will compactions still happen if there are differen= t > versions of the schema? > > > > > > On 9/18/12 11:38 AM, "Michael Kjellman" wrote: > > >Thanks, I just modified the schema on the worse offending column family > >(as determined by the .json) from 10MB to 200MB. > > > >Should I kick off a compaction on this cf now/repair?/scrub? > > > >Thanks > > > >-michael > > > >From: =F7=A6=D4=C1=CC=A6=CA =F4=C9=CD=DE=C9=DB=C9=CE > > >Reply-To: "user@cassandra.apache.org" > >> > >To: "user@cassandra.apache.org" > >> > >Subject: Re: persistent compaction issue (1.1.4 and 1.1.5) > > > >I've started to use LeveledCompaction some time ago and from my > >experience this indicates some SST on lower levels than they should be. > >The compaction is going, moving them up level by level, but total count > >does not change as new data goes in. > >The numbers are pretty high as for me. Such numbers mean a lot of files > >(over 100K in single directory) and a lot of thinking for compaction > >executor to decide what to compact next. I can see numbers like 5K-10K > >and still thing this is high number. If I were you, I'd increase > >sstable_size_in_mb 10-20 times it is now. > > > >2012/9/17 Michael Kjellman > >> > >Hi All, > > > >I have an issue where each one of my nodes (currently all running at > >1.1.5) is reporting around 30,000 pending compactions. I understand that > >a pending compaction doesn't necessarily mean it is a scheduled task > >however I'm confused why this behavior is occurring. It is the same on > >all nodes, occasionally goes down 5k pending compaction tasks, and then > >returns to 25,000-35,000 compaction tasks pending. > > > >I have tried a repair operation/scrub operation on two of the nodes and > >while compactions initially happen the number of pending compactions doe= s > >not decrease. > > > >Any ideas? Thanks for your time. > > > >Best, > >michael > > > > > >'Like' us on Facebook for exclusive content and other resources on all > >Barracuda Networks solutions. > > > >Visit http://barracudanetworks.com/facebook > > > > > > > > > > > > > > > >-- > >Best regards, > > Vitalii Tymchyshyn > > > >'Like' us on Facebook for exclusive content and other resources on all > >Barracuda Networks solutions. > > > >Visit http://barracudanetworks.com/facebook > > > > > > > > > > > 'Like' us on Facebook for exclusive content and other resources on all > Barracuda Networks solutions. > > Visit http://barracudanetworks.com/facebook > > > > > --=20 Best regards, Vitalii Tymchyshyn --14dae93407f788e22e04ca1b7c8b Content-Type: text/html; charset=KOI8-U Content-Transfer-Encoding: quoted-printable I did see problems with schema agreement on 1.1.4, but they did go away aft= er rolling restart (BTW: it would be still good to check describe schema fo= r unreachable). Same rolling restart helped to force compactions after movi= ng to Leveled compaction. If your compactions still don't go, you can t= ry removing *.json files from the data directory of the stopped node to for= ce moving all SSTables to level0.

Best regards, Vitalii Tymchyshyn

2012/9/19 Michael Kjellman <mkjellman@barracuda.com>
Potentially the pending compactions are a sy= mptom and not the root
cause/problem.

When updating a 3rd column family with a larger sstable_size_in_mb it
looks like the schema may not be in a good state

[default@xxxx] UPDATE COLUMN FAMILY screenshots WITH
compaction_strategy=3DLeveledCompactionStrategy AND
compaction_strategy_options=3D{sstable_size_in_mb: 200};
290cf619-57b0-3ad1-9ae3-e313290de9c9
Waiting for schema agreement...
Warning: unreachable nodes 10.8.30.102The schema has not settled in 10
seconds; further migrations are ill-advised until it does.
Versions are UNREACHABLE:[10.8.30.102],
290cf619-57b0-3ad1-9ae3-e313290de9c9:[10.8.30.15, 10.8.30.14, 10.8.30.13, 10.8.30.103, 10.8.30.104, 10.8.30.105, 10.8.30.106],
f1de54f5-8830-31a6-9cdd-aaa6220cccd1:[10.8.30.101]


However, tpstats looks good. And the schema changes eventually do get
applied on *all* the nodes (even the ones that seem to have different
schema versions). There are no communications issues between the nodes and<= br> they are all in the same rack

root@xxxx:~# nodetool tpstats
Pool Name =9A =9A =9A =9A =9A =9A =9A =9A =9A =9AActive =9A Pending =9A =9A= =9ACompleted =9A Blocked
All time blocked
ReadStage =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A= 0 =9A =9A =9A =9A1254592 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
RequestResponseStage =9A =9A =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0 =9A =9A= =9A =9A9480827 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
MutationStage =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0 = =9A =9A =9A =9A8662263 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
ReadRepairStage =9A =9A =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0 =9A= =9A =9A =9A 339158 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
ReplicateOnWriteStage =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0 =9A =9A = =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
GossipStage =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0= =9A =9A =9A =9A1469197 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
AntiEntropyStage =9A =9A =9A =9A =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0 =9A= =9A =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
MigrationStage =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0 = =9A =9A =9A =9A =9A 1808 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
MemtablePostFlusher =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0 =9A =9A= =9A =9A =9A =9A248 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
StreamStage =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0= =9A =9A =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
FlushWriter =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0= =9A =9A =9A =9A =9A =9A248 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 4
MiscStage =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A= 0 =9A =9A =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
commitlog_archiver =9A =9A =9A =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0 =9A = =9A =9A =9A =9A =9A =9A0 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
InternalResponseStage =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0 =9A =9A = =9A =9A =9A 5286 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0
HintedHandoff =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0 =9A =9A =9A =9A 0 = =9A =9A =9A =9A =9A =9A 21 =9A =9A =9A =9A 0
=9A =9A =9A =9A =9A =9A 0

Message type =9A =9A =9A =9A =9A Dropped
RANGE_SLICE =9A =9A =9A =9A =9A =9A =9A =9A =9A0
READ_REPAIR =9A =9A =9A =9A =9A =9A =9A =9A =9A0
BINARY =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0
READ =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0
MUTATION =9A =9A =9A =9A =9A =9A =9A =9A =9A =9A 0
REQUEST_RESPONSE =9A =9A =9A =9A =9A =9A 0

So I'm guessing maybe the different schema versions may be potentially<= br> stopping compactions? Will compactions still happen if there are different<= br> versions of the schema?





On 9/18/12 11:38 AM, "Michael Kjellman" <mkjellman@barracuda.com> wrote:

>Thanks, I just modified the schema on the worse offending column family=
>(as determined by the .json) from 10MB to 200MB.
>
>Should I kick off a compaction on this cf now/repair?/scrub?
>
>Thanks
>
>-michael
>
>From: =F7=A6=D4=C1=CC=A6=CA =F4=C9=CD=DE=C9=DB=C9=CE <tivv00@gmail.com<mailto:tivv00@gmail.com>>
>Reply-To: "user@cassa= ndra.apache.org<mailto:= user@cassandra.apache.org>"
><user@cassandra.apache.= org<mailto:user@cassand= ra.apache.org>>
>To: "user@cassandra.a= pache.org<mailto:user@c= assandra.apache.org>"
><user@cassandra.apache.= org<mailto:user@cassand= ra.apache.org>>
>Subject: Re: persistent compaction issue (1.1.4 and 1.1.5)
>
>I've started to use LeveledCompaction some time ago and from my
>experience this indicates some SST on lower levels than they should be.=
>The compaction is going, moving them up level by level, but total count=
>does not change as new data goes in.
>The numbers are pretty high as for me. Such numbers mean a lot of files=
>(over 100K in single directory) and a lot of thinking for compaction >executor to decide what to compact next. I can see numbers like 5K-10K<= br> >and still thing this is high number. If I were you, I'd increase >sstable_size_in_mb 10-20 times it is now.
>
>2012/9/17 Michael Kjellman
><mkjellman@barracuda.com<= /a><mailto:mkjellman@barracud= a.com>>
>Hi All,
>
>I have an issue where each one of my nodes (currently all running at >1.1.5) is reporting around 30,000 pending compactions. I understand tha= t
>a pending compaction doesn't necessarily mean it is a scheduled tas= k
>however I'm confused why this behavior is occurring. It is the same= on
>all nodes, occasionally goes down 5k pending compaction tasks, and then=
>returns to 25,000-35,000 compaction tasks pending.
>
>I have tried a repair operation/scrub operation on two of the nodes and=
>while compactions initially happen the number of pending compactions do= es
>not decrease.
>
>Any ideas? Thanks for your time.
>
>Best,
>michael
>
>
>'Like' us on Facebook for exclusive content and other resources= on all
>Barracuda Networks solutions.
>
>Visit http://barracudanetworks.com/facebook
>
>
>
>
>
>
>
>--
>Best regards,
> Vitalii Tymchyshyn
>
>'Like' us on Facebook for exclusive content and other resources= on all
>Barracuda Networks solutions.
>
>Visit http://barracudanetworks.com/facebook
>
>
>
>


'Like' us on Facebook for exclusive content and other resources on = all Barracuda Networks solutions.

Visit h= ttp://barracudanetworks.com/facebook







--
= Best regards,
=9AVitalii Tymchyshyn
--14dae93407f788e22e04ca1b7c8b--