Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 37D937230 for ; Sun, 21 Aug 2011 22:11:11 +0000 (UTC) Received: (qmail 6135 invoked by uid 500); 21 Aug 2011 22:11:08 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 6102 invoked by uid 500); 21 Aug 2011 22:11:07 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 6094 invoked by uid 99); 21 Aug 2011 22:11:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 22:11:07 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a82.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 22:11:00 +0000 Received: from homiemail-a82.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a82.g.dreamhost.com (Postfix) with ESMTP id 1FBED282061 for ; Sun, 21 Aug 2011 15:10:39 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=XcKqKYnnKE iyIJBg/TFWCDbWWibg537yp4v8xFbT2wlv4WBeyhrw81uT/OKogh01BDPN3l/QQp gHxWJpxecZl3lFpr0Spl9I8qCuDh2TW8JuoQPdTLbpSQBDCcUcEQnqeUi/eFk7j/ mgVJx8jJTUFvKbUEaPc9GLAox7HhvQ0pY= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=R5C3SbZjosotJvZ+ z/Xt4uYuJHk=; b=SFJC/yKDYxPQPxvojLr6fgDybbU2SgIuTypHNlkWPvyzknr/ 60u+OOhHysx3cRzFUJExSu+2Xxqd6TGaMs9hPXglQyYh9VPYh1HQOkgq94ekIrpP WLGPM+sXiYHmEz2+jljTQK2oTWqTvn/GalYQhj4ZYUbGQHlmZOOjlTGLlqE= Received: from 202-126-206-198.vectorcommunications.net.nz (unknown [202.126.206.198]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a82.g.dreamhost.com (Postfix) with ESMTPSA id 3836728205F for ; Sun, 21 Aug 2011 15:10:38 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: multipart/alternative; boundary="Apple-Mail=_0C17E383-F54A-4600-999B-98ACE1D5819C" Subject: Re: Commit log fills up in less than a minute Date: Mon, 22 Aug 2011 10:10:33 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <47C34E69-F3ED-4253-8944-126D12449451@thelastpickle.com> X-Mailer: Apple Mail (2.1244.3) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_0C17E383-F54A-4600-999B-98ACE1D5819C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Yup, you can check the what HH is doing via JMX. there is a bug in 0.7 that can result in log files not been deleted = https://issues.apache.org/jira/browse/CASSANDRA-2829 Cheers =20 ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 22/08/2011, at 4:56 AM, Anand Somani wrote: > We have a lot of space on /data, and looks like it was flushing data = fine from file timestamps.=20 >=20 > We did have a bit of goofup with IP's when bringing up a down node = (and the commit files have been around since then). Wonder if that is = what triggered it and we have a bunch of hinted handoff's being backed = up. >=20 > For hinted hand off - how do I check if the nodes are collecting hints = ( I do have it turned on). I noticed console bean HintedHandManager, is = that the only way to find out. >=20 > On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller = wrote: > > When does the actual commit-data file get deleted. > > > > The flush interval on all my memtables is 60 minutes >=20 > They *should* be getting deleted when they no longer contain any data > that has not been flushed to disk. Are flushes definitely still > happening? Is it possible flushing has started failing (e.g. out of > disk)? >=20 > The only way I can think of over nodes directly affecting the commit > log size on your node would be e.g. hinted handoff resulting in burst > of writes. >=20 > -- > / Peter Schuller (@scode on twitter) >=20 --Apple-Mail=_0C17E383-F54A-4600-999B-98ACE1D5819C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 Yup, = you can check the what HH is doing via JMX.

there is = a bug in 0.7 that can result in log files not been deleted https://issu= es.apache.org/jira/browse/CASSANDRA-2829

Chee= rs
 
http://www.thelastpickle.com

On 22/08/2011, at 4:56 AM, Anand Somani wrote:

We have a = lot of space on /data, and looks like it was flushing data fine from = file timestamps.

We did have a bit of goofup with IP's when = bringing up a down node (and the commit files have been around since = then). Wonder if that is what triggered it and we have a bunch of hinted = handoff's being backed up.

For hinted hand off - how do I check if the nodes are collecting = hints ( I do have it turned on). I noticed console bean = HintedHandManager, is that the only way to find out.

On Sun, Aug 21, 2011 at 9:20 AM, Peter Schuller = <peter.schuller@infidyne.com> wrote:
> When does the actual commit-data file get deleted.
>
> The flush interval on all my memtables is 60 minutes

They *should* be getting deleted when they no longer contain any = data
that has not been flushed to disk. Are flushes definitely still
happening? Is it possible flushing has started failing (e.g. out of
disk)?

The only way I can think of over nodes directly affecting the commit
log size on your node would be e.g. hinted handoff resulting in = burst
of writes.

--
/ Peter Schuller (@scode on twitter)


= --Apple-Mail=_0C17E383-F54A-4600-999B-98ACE1D5819C--