Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3A124107E6 for ; Fri, 21 Jun 2013 11:03:59 +0000 (UTC) Received: (qmail 46953 invoked by uid 500); 21 Jun 2013 11:03:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 46615 invoked by uid 500); 21 Jun 2013 11:03:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 46604 invoked by uid 99); 21 Jun 2013 11:03:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Jun 2013 11:03:55 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [74.125.149.244] (HELO na3sys009aog118.obsmtp.com) (74.125.149.244) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 21 Jun 2013 11:03:51 +0000 Received: from mail-ve0-f169.google.com ([209.85.128.169]) (using TLSv1) by na3sys009aob118.postini.com ([74.125.148.12]) with SMTP ID DSNKUcQy7dKzzrcWoONjoqniP/rvEHO6zMcK@postini.com; Fri, 21 Jun 2013 04:03:30 PDT Received: by mail-ve0-f169.google.com with SMTP id m1so6264591ves.0 for ; Fri, 21 Jun 2013 04:03:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=ZZ1n62eufLADX2jkb9dJbC4mYA8WQ0LyIao1QcCc9Ek=; b=abV5n8I6+kSA2mtcbJdlXIIHbvjwSVno9T0t+d6QAtL9ybAhfRHbUoDs8b4FhOb3Fn 7aFEbOhalEp/64f6/8tr3rofIw2DufBMwK/6jo0/x/Uoz7buAl6Kr4xwmxgWEElArfbO Gf7t2PJ8ugYMdmincPQJ3Gxvc780KzMQaZfXyRsQMJyAydMTMJSu6mKv7ZxthSZmrAtJ JAfH0GoyjdnMO6jN8aXQm3a3CO7Wtwewcw0aHxWJHN91seiQjlo+4UJr2v9ZOchEkFbs /OgQijVbZhKnuH6RQNkUjgtP/Q80JaBpGVRCgxfUMgDTe5t5oxPle0U3mm3Dcwf+IX2O t2Og== X-Received: by 10.52.90.50 with SMTP id bt18mr4556885vdb.35.1371812589097; Fri, 21 Jun 2013 04:03:09 -0700 (PDT) X-Received: by 10.52.90.50 with SMTP id bt18mr4556872vdb.35.1371812588882; Fri, 21 Jun 2013 04:03:08 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.181.9 with HTTP; Fri, 21 Jun 2013 04:02:48 -0700 (PDT) In-Reply-To: <8AEF1470-FCA7-452E-B504-0DB32F5C0B3E@thelastpickle.com> References: <8AEF1470-FCA7-452E-B504-0DB32F5C0B3E@thelastpickle.com> From: Franc Carter Date: Fri, 21 Jun 2013 21:02:48 +1000 Message-ID: Subject: Re: Compaction not running To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec50162af0f246804dfa80236 X-Gm-Message-State: ALoCoQnwwkK6715sNM37wxlWUFx0NBffGb1WVT+QXjjaav43xxvYgsGjS9YD9aaMo1pYaTnKkmbQ4+oAHxDw/x9hNIAnuGIY/vqhPnVzNxyPhwwwOI4KSiUHfZrfreXCV05yJwLg4i+HwObgWdjXVmeTiZO2Ci2EstGhggC9VbsxyRKAE3ZoUX0= X-Virus-Checked: Checked by ClamAV on apache.org --bcaec50162af0f246804dfa80236 Content-Type: text/plain; charset=ISO-8859-1 On Fri, Jun 21, 2013 at 6:16 PM, aaron morton wrote: > Do you think it's worth posting an issue, or not enough traceable evidence > ? > > If you can reproduce it then certainly file a bug. > I'll keep my eye on it to see if it happens again and there is a pattern cheers > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 20/06/2013, at 9:41 PM, Franc Carter wrote: > > On Thu, Jun 20, 2013 at 7:27 PM, aaron morton wrote: > >> nodetool compactionstats, gives >> >> pending tasks: 13120 >> >> If there are no errors in the log, I would say this is a bug. >> > > This happened after the node ran out of file descriptors, so an edge case > wouldn't surprise me. > > I've rebuilt the node (blown the data way and am running a nodetool > rebuild). Do you think it's worth posting an issue, or not enough traceable > evidence ? > > cheers > > >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 19/06/2013, at 11:41 AM, Franc Carter >> wrote: >> >> On Wed, Jun 19, 2013 at 9:34 AM, Bryan Talbot wrote: >> >>> Manual compaction for LCS doesn't really do much. It certainly doesn't >>> compact all those little files into bigger files. What makes you think >>> that compactions are not occurring? >>> >> >> Yeah, that's what I thought, however:- >> >> nodetool compactionstats, gives >> >> pending tasks: 13120 >> Active compaction remaining time : n/a >> >> when I run nodetool compact in a loop the pending tasks goes down >> gradually. >> >> This node also has vastly higher latencies (x10) than the other nodes. I >> saw this with a previous CF than I 'manually compacted', and when the >> pending tasks reached low numbers (stuck on 9) then latencies were back to >> low milliseconds >> >> cheers >> >> >>> -Bryan >>> >>> >>> >>> On Tue, Jun 18, 2013 at 3:59 PM, Franc Carter >> > wrote: >>> >>>> On Sat, Jun 15, 2013 at 11:49 AM, Franc Carter < >>>> franc.carter@sirca.org.au> wrote: >>>> >>>>> On Sat, Jun 15, 2013 at 8:48 AM, Robert Coli wrote: >>>>> >>>>>> On Wed, Jun 12, 2013 at 3:26 PM, Franc Carter < >>>>>> franc.carter@sirca.org.au> wrote: >>>>>> > We are running a test system with Leveled compaction on >>>>>> Cassandra-1.2.4. >>>>>> > While doing an initial load of the data one of the nodes ran out of >>>>>> file >>>>>> > descriptors and since then it hasn't been automatically compacting. >>>>>> >>>>>> You have (at least) two options : >>>>>> >>>>>> 1) increase file descriptors available to Cassandra with ulimit, if >>>>>> possible >>>>>> 2) increase the size of your sstables with levelled compaction, such >>>>>> that you have fewer of them >>>>>> >>>>> >>>>> Oops, I wasn't clear enough. >>>>> >>>>> I have increased the number of file descriptors and no longer have a >>>>> file descriptor issue. However the node still doesn't compact >>>>> automatically. If I run a 'nodetool compact' it will do a small amount of >>>>> compaction and then stop. The Column Family is using LCS >>>>> >>>> >>>> Any ideas on this - compaction is still not automatically running for >>>> one of my nodes >>>> >>>> thanks >>>> >>>> >>>>> >>>>> cheers >>>>> >>>>> >>>>>> >>>>>> =Rob >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Franc Carter* | Systems architect | Sirca Ltd >>>>> >>>>> franc.carter@sirca.org.au | www.sirca.org.au >>>>> Tel: +61 2 8355 2514 >>>>> Level 4, 55 Harrington St, The Rocks NSW 2000 >>>>> PO Box H58, Australia Square, Sydney NSW 1215 >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Franc Carter* | Systems architect | Sirca Ltd >>>> >>>> franc.carter@sirca.org.au | www.sirca.org.au >>>> Tel: +61 2 8355 2514 >>>> Level 4, 55 Harrington St, The Rocks NSW 2000 >>>> PO Box H58, Australia Square, Sydney NSW 1215 >>>> >>>> >>> >> >> >> -- >> *Franc Carter* | Systems architect | Sirca Ltd >> >> franc.carter@sirca.org.au | www.sirca.org.au >> Tel: +61 2 8355 2514 >> Level 4, 55 Harrington St, The Rocks NSW 2000 >> PO Box H58, Australia Square, Sydney NSW 1215 >> >> >> > > > -- > *Franc Carter* | Systems architect | Sirca Ltd > > franc.carter@sirca.org.au | www.sirca.org.au > Tel: +61 2 8355 2514 > Level 4, 55 Harrington St, The Rocks NSW 2000 > PO Box H58, Australia Square, Sydney NSW 1215 > > > -- *Franc Carter* | Systems architect | Sirca Ltd franc.carter@sirca.org.au | www.sirca.org.au Tel: +61 2 8355 2514 Level 4, 55 Harrington St, The Rocks NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215 --bcaec50162af0f246804dfa80236 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Fri, Jun 21, 2013 at 6:16 PM, aaron morton <aaron@thelastpickle.c= om> wrote:
Do you think it's worth posting an issu= e, or not enough traceable evidence ?
If you can re= produce it then certainly file a bug.=A0

I'll keep my eye on it to see if it ha= ppens again and there is a pattern

cheers
=A0

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 20/06/2013, at 9:41 PM, Franc= Carter <= franc.carter@sirca.org.au> wrote:

On Thu, Jun 20, 2013 at 7:27 PM, aaron morton <aaron@thelastpickle.= com> wrote:
nodetool compactionstats, gives

=A0=A0=A0 pending= tasks: 13120
If there are no errors in the log, I = would say this is a bug.=A0

This happened after the node ran out of fi= le descriptors, so an edge case wouldn't surprise me.

I've rebuilt the node (blown the data way and am running a node= tool rebuild). Do you think it's worth posting an issue, or not enough = traceable evidence ?

cheers
=A0

Cheers
=
-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 19/06/2013, at 11:41 AM, Franc Carter <franc.carter@sirca.org.au> wrote:

On Wed, Jun 19, 2013 at 9:3= 4 AM, Bryan Talbot <btalbot@aeriagames.com> wrote:
Manual compaction for LCS doesn't really do much. =A0I= t certainly doesn't compact all those little files into bigger files. = =A0What makes you think that compactions are not occurring?

Yeah, that's what I thoug= ht, however:-

nodetool compactionstats, gives

=A0=A0=A0 pendi= ng tasks: 13120
=A0=A0 Active compaction remaining time :=A0=A0=A0=A0=A0= =A0=A0 n/a

when I run nodetool compact in a loop the pending tasks goes down gradu= ally.

This node also has vastly higher latencies (x10) than the othe= r nodes. I saw this with a previous CF than I 'manually compacted',= and when the pending tasks reached low numbers (stuck on 9) then latencies= were back to low milliseconds

cheers
=A0
-Bryan



On Tue, Jun 18, 2013 at 3:59 PM, Fra= nc Carter <franc.carter@sirca.org.au> wrote:
On Sat, Jun 15, 2013 at 11:49 AM, Franc= Carter <franc.carter@sirca.org.au> wrote:
On Sat, Jun 15, 2013 at 8:48 AM, Robert Coli <rcoli@eventbrite.com= > wrote:
On Wed, Jun 12, 2013 at 3:26 PM, Franc Carter <franc.carter@sirca.org.au>= ; wrote:
> We are running a test system with Leveled compaction on Cassandra-1.2.= 4.
> While doing an initial load of the data one of the nodes ran out of fi= le
> descriptors and since then it hasn't been automatically compacting= .

You have (at least) two options :

1) increase file descriptors available to Cassandra with ulimit, if possibl= e
2) increase the size of your sstables with levelled compaction, such
that you have fewer of them

Oops,= I wasn't clear enough.

I have increased the n= umber of file descriptors and no longer have a file descriptor issue. Howev= er the node still doesn't compact automatically. If I run a 'nodeto= ol compact' it will do a small amount of compaction and then stop. The = Column Family is using LCS

Any ideas on this - compaction is still n= ot automatically running for one of my nodes

thanks
=A0

cheers
=A0

--
Franc Carter <= /span>| Systems architect | Sirca Ltd
Level 4, 55 Harrington St, The Rocks NSW 2000
PO Box H58, Australia Square, Sydney NSW 1215<= /span>




--
Franc Carter = | Systems architect | Sirca Ltd
Level 4, 55 Harrington St, The Rocks NSW 2000
PO Box H58, Australia Square, Sydney NSW 1215<= /span>





--
Franc Carter | Systems architect |<= /span> Sirca Ltd
Level 4, 55 Harrington St, The Rocks NSW 2000
PO Box H58, Australia Square, Sydney NSW 1215<= /span>





--
Franc Carter | = Systems architect | Sirca Ltd
Level 4, 55 Harrington St, The Rocks NSW 2000
PO Box H58, Australia Square, Sydney NSW 1215<= /span>





--

Franc Carter<= /b> |<= /span> Systems architect | Sirca Ltd

franc.carter@sirca.org.au=A0|=A0www.sirca.org.au

Tel:= =A0+61 2 8355 2514

Level 4, 55 Harrington St, The Rocks NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215<= /span>


--bcaec50162af0f246804dfa80236--