From user-return-20184-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Aug 24 15:26:06 2011 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E096D823D for ; Wed, 24 Aug 2011 15:26:06 +0000 (UTC) Received: (qmail 49669 invoked by uid 500); 24 Aug 2011 15:26:04 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 49570 invoked by uid 500); 24 Aug 2011 15:26:03 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 49557 invoked by uid 99); 24 Aug 2011 15:26:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Aug 2011 15:26:03 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of meatforums@gmail.com designates 209.85.216.179 as permitted sender) Received: from [209.85.216.179] (HELO mail-qy0-f179.google.com) (209.85.216.179) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Aug 2011 15:25:56 +0000 Received: by qyk35 with SMTP id 35so845346qyk.10 for ; Wed, 24 Aug 2011 08:25:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=MHi0+F4Vk3MWHnYH5r7U5l0tYoQMzCJsP8AE9r959Zc=; b=V8TNJ8D+8HOx+XSqzjv9i1kFuTgfU6ENEhHI88iI96Lg4ChpVgIxKs2b1isBlGaO3R LQqSHAJGF2QPaxCDgEnckj6SXWCPjljEc9mhUrhfN2IQkTC7lExRLnRa+WuRPlPUw6DY oBsu0QkhiJJLXVuLXr0ilouNCs8tnLgOQtgZY= MIME-Version: 1.0 Received: by 10.229.48.147 with SMTP id r19mr878244qcf.18.1314199355439; Wed, 24 Aug 2011 08:22:35 -0700 (PDT) Received: by 10.229.16.134 with HTTP; Wed, 24 Aug 2011 08:22:35 -0700 (PDT) In-Reply-To: References: <47C34E69-F3ED-4253-8944-126D12449451@thelastpickle.com> Date: Wed, 24 Aug 2011 08:22:35 -0700 Message-ID: Subject: Re: Commit log fills up in less than a minute From: Anand Somani To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=00163683203ebebbdc04ab41e18f X-Virus-Checked: Checked by ClamAV on apache.org --00163683203ebebbdc04ab41e18f Content-Type: text/plain; charset=ISO-8859-1 So I have looked at the cluster from - Cassandra-client - describe cluster => shows correctly - 3 nodes - used the StorageService - JMX bean =>UnreachableNodes - shows 0 If all these show the correct ring state, why are hints being maintained, looks like that is the only way to find out about "phantom" nodes. On Wed, Aug 24, 2011 at 8:01 AM, Anand Somani wrote: > So, I restarted the cluster (not rolling), but it is still maintaining > hints for the IP's that are no longer part of the ring. nodetool ring shows > things correctly (as only 3 nodes). > When I check thru the jmx hintedhandoff manager, it shows it is maintaining > the hints for those non existent IP's. So the question is > - How can I remove these IP permanently, so hints do not get saved? > - Not all nodes see the same list of IP's > > > > > On Sun, Aug 21, 2011 at 3:10 PM, aaron morton wrote: > >> Yup, you can check the what HH is doing via JMX. >> >> there is a bug in 0.7 that can result in log files not been deleted >> https://issues.apache.org/jira/browse/CASSANDRA-2829 >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 22/08/2011, at 4:56 AM, Anand Somani wrote: >> >> We have a lot of space on /data, and looks like it was flushing data fine >> from file timestamps. >> >> We did have a bit of goofup with IP's when bringing up a down node (and >> the commit files have been around since then). Wonder if that is what >> triggered it and we have a bunch of hinted handoff's being backed up. >> >> For hinted hand off - how do I check if the nodes are collecting hints ( I >> do have it turned on). I noticed console bean HintedHandManager, is that the >> only way to find out. >> >> On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller < >> peter.schuller@infidyne.com> wrote: >> >>> > When does the actual commit-data file get deleted. >>> > >>> > The flush interval on all my memtables is 60 minutes >>> >>> They *should* be getting deleted when they no longer contain any data >>> that has not been flushed to disk. Are flushes definitely still >>> happening? Is it possible flushing has started failing (e.g. out of >>> disk)? >>> >>> The only way I can think of over nodes directly affecting the commit >>> log size on your node would be e.g. hinted handoff resulting in burst >>> of writes. >>> >>> -- >>> / Peter Schuller (@scode on twitter) >>> >> >> >> > --00163683203ebebbdc04ab41e18f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable So I have looked at the cluster from
  • Cassandra-client - describ= e cluster =3D> shows correctly - 3 nodes
  • used the StorageService= - JMX bean =3D>UnreachableNodes - shows 0

If all these sh= ow the correct ring state, why are hints being maintained, looks like that = is the only way to find out about "phantom" nodes.

On Wed, Aug 24, 2011 at 8:01 AM, Anand Soman= i <meatforums@= gmail.com> wrote:
So, I restarted the cluster (not rolling), but it is still maintaining hint= s for the IP's that are no longer part of the ring. nodetool ring shows= things correctly (as only 3 nodes).
When I check thru the jmx hintedha= ndoff manager, it shows it is maintaining the hints for those non existent = IP's. So the question is
=A0- How can I remove these IP permanently, so hints do not get saved?
= =A0- Not all nodes see the same list of IP's




On Sun, Aug 21, 2011 at = 3:10 PM, aaron morton <aaron@thelastpickle.com> wrote:=
Yup, you can check the what HH is doing via JMX.
<= br>
there is a bug in 0.7 that can result in log files not been dele= ted=A0https://issues.apache.org/jira/browse/CASSANDRA-2829

Cheers
=A0
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelast= pickle.com

On 22/08/2011, at 4:56 AM, Anand Somani wrote:

We have a lot of space on /data, and looks like it was= flushing data fine from file timestamps.

We did have a bit of goof= up with IP's when bringing up a down node (and the commit files have be= en around since then). Wonder if that is what triggered it and we have a bu= nch of hinted handoff's being backed up.

For hinted hand off - how do I check if the nodes are collecting hints = ( I do have it turned on). I noticed console bean HintedHandManager, is tha= t the only way to find out.

On Sun, Aug 2= 1, 2011 at 9:20 AM, Peter Schuller <peter.schuller@infidyne.com<= /a>> wrote:
> When do= es the actual commit-data file get deleted.
>
> The flush interval on all my memtables is 60 minutes

They *should* be getting deleted when they no longer contain any data=
that has not been flushed to disk. Are flushes definitely still
happening? Is it possible flushing has started failing (e.g. out of
disk)?

The only way I can think of over nodes directly affecting the commit
log size on your node would be e.g. hinted handoff resulting in burst
of writes.

--
/ Peter Schuller (@scode on twitter)




--00163683203ebebbdc04ab41e18f--