Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1B0677283 for ; Sat, 16 Jul 2011 02:47:01 +0000 (UTC) Received: (qmail 34165 invoked by uid 500); 16 Jul 2011 02:46:58 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 33900 invoked by uid 500); 16 Jul 2011 02:46:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 33892 invoked by uid 99); 16 Jul 2011 02:46:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Jul 2011 02:46:54 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of schumi.han@gmail.com designates 209.85.213.172 as permitted sender) Received: from [209.85.213.172] (HELO mail-yx0-f172.google.com) (209.85.213.172) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Jul 2011 02:46:48 +0000 Received: by yxp4 with SMTP id 4so915309yxp.31 for ; Fri, 15 Jul 2011 19:46:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=8yJUQI/eic/yWqlEXiefmtpvIpOqDfwA6IVWeVRR+HY=; b=ZrlKod633sB8xr6CaK8FtlHeydiGigr8zquz1lS4lqlOzpyMpdTUN0yHdQow6AYN4X UiyI4jKfDGnS60+Hwv6Pt6hgJNkzvNkbbL2IXQEXx63tVjH3lOi1XzCBsTa1oxfJ9jzl QDLxI3LBd7s1R/D3k9754eAV2GQt0wHPtl6JM= MIME-Version: 1.0 Received: by 10.236.157.103 with SMTP id n67mr5069752yhk.97.1310784387173; Fri, 15 Jul 2011 19:46:27 -0700 (PDT) Received: by 10.236.42.194 with HTTP; Fri, 15 Jul 2011 19:46:27 -0700 (PDT) In-Reply-To: References: Date: Sat, 16 Jul 2011 10:46:27 +0800 Message-ID: Subject: Re: Commit log is not emptied after "nodetool drain" From: Zhu Han To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=485b397dd083c64e0104a826c561 X-Virus-Checked: Checked by ClamAV on apache.org --485b397dd083c64e0104a826c561 Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable 2011/7/15 Zhu Han > > 2011/7/15 Jonathan Ellis > >> If you have non-empty segments post-drain that is a bug. Is it >> reproducible? >> > > I think it is always reproducible on 0.6.x branch. Here is a simple > experiment: > Should I raise an issue ticket on it? > > 1) "bin/nodetool -h localhost" > > 2) During flush the memtables, we can observe the name of the old commit > log from log: > >> " INFO [main] 2011-07-15 11:57:46,742 ColumnFamilyStore.java (line 478) >> Data has reached its threshold; switching in a fresh Memtable at >> CommitLogContext(file=3D'/var/lib/cassandra/commitlog/CommitLog-13107022= 65959.log', >> position=3D125) " >> > > 3) Before the node is drained, new commitlog is created: > >> INFO [COMMIT-LOG-WRITER] 2011-07-15 11:58:11,383 CommitLogSegment.java >> (line 50) Creating new commitlog segment >> /var/lib/cassandra/commitlog/CommitLog-1310702291383.log >> INFO [RMI TCP Connection(2)-192.168.1.101] 2011-07-15 11:58:11,413 >> StorageService.java (line 391) Node is drained >> > > 4) After the node is drained and killed, there are still two commit log > under the directory > >> $ ls -lh /var/lib/cassandra/commitlog/ >> total 128K >> -rw-r--r-- 1 root root 439 2011-07-15 11:57 CommitLog-1310702265959.log >> -rw-r--r-- 1 root root 125 2011-07-15 11:58 CommitLog-1310702291383.log >> > > > > >> 2011/7/14 Zhu Han : >> > Jonathan, >> > >> > But all the old non-empty log segments are kept on the disk. And >> cassandra >> > takes some time to apply the operations from these closed log segments >> after >> > restart of the process. >> > >> > Is it expected? >> > >> > best regards, >> > =BA=AB=D6=F1(Zhu Han) >> > >> > =BC=E1=B9=FB=C6=CC=D7=D3, =D7=EE=BC=F2=B5=A5=D2=D7=D3=C3=B5=C4=D4=C6= =B4=E6=B4=A2 >> > =CD=AC=B2=BD=CE=C4=BC=FE, =B7=D6=CF=ED=D5=D5=C6=AC, =CE=C4=B5=B5=B1=B8= =B7=DD! >> > >> > >> > >> > 2011/7/15 Jonathan Ellis >> >> >> >> It's expected to have a new, empty segment after drain completes. >> >> >> >> 2011/7/14 Zhu Han : >> >> > The deployed version is based on 0.6.13. >> >> > >> >> > After "nodetool drain" is invoked on one of the nodes, the commit l= og >> is >> >> > not >> >> > emptied. Is this the expected behavior? If so, how can I rename a >> column >> >> > family on 0.6.x branch? >> >> > >> >> > Here is the log output: >> >> > " >> >> > INFO [COMMIT-LOG-WRITER] 2011-07-15 00:39:49,541 >> CommitLogSegment.java >> >> > (line >> >> > 50) Creating new commitlog segment >> >> > /var/lib/cassandra/commitlog/CommitLog-1310661589541.log >> >> > INFO [RMI TCP Connection(8)-202.120.2.16] 2011-07-15 00:39:49,544 >> >> > StorageService.java (line 391) Node is drained >> >> > " >> >> > >> >> > I saw an issue here, but it was reported against 0.8.x branch. >> >> > https://issues.apache.org/jira/browse/CASSANDRA-2874 >> >> > >> >> > best regards, >> >> > =BA=AB=D6=F1(Zhu Han) >> >> > >> >> > =BC=E1=B9=FB=C6=CC=D7=D3, =D7=EE=BC=F2=B5=A5=D2=D7=D3=C3=B5=C4=D4= =C6=B4=E6=B4=A2 >> >> > =CD=AC=B2=BD=CE=C4=BC=FE, =B7=D6=CF=ED=D5=D5=C6=AC, =CE=C4=B5=B5=B1= =B8=B7=DD! >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> Jonathan Ellis >> >> Project Chair, Apache Cassandra >> >> co-founder of DataStax, the source for professional Cassandra support >> >> http://www.datastax.com >> > >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com >> > > --485b397dd083c64e0104a826c561 Content-Type: text/html; charset=GB2312 Content-Transfer-Encoding: quoted-printable
2011/7/15 Zhu Han <schumi.han@gmail.com>

2011/7/15 Jonathan Ellis <= span dir=3D"ltr"><jbellis@gmail.com>
If you have non-empty segments post-drain that is a bug.  Is it reprod= ucible?

I think it is always reproducible on= 0.6.x branch. Here is a simple experiment:

Should I raise an issue ticket on it?
 

1) "bin/nodetool -h localhost"

2) During flush the memtables,  we can observe the name of the old= commit log from log:
" INFO [main] 2011-07-15 11:57:46,742 ColumnFamilyStore.java (line 478= ) Data has reached its threshold; switching in a fresh Memtable at CommitLo= gContext(file=3D'/var/lib/cassandra/commitlog/CommitLog-1310702265959.l= og', position=3D125) "

3) Before the node is drained, new commitlog is created:
 INFO [COMMIT-LO= G-WRITER] 2011-07-15 11:58:11,383 CommitLogSegment.java (line 50) Creating = new commitlog segment /var/lib/cassandra/commitlog/CommitLog-1310702291383.= log
 INFO [RMI TCP Connection(2)-192.168.1.101] 2011-07-15 11:58:11,413 St= orageService.java (line 391) Node is drained

4) After t= he node is drained and killed, there are still two commit log under the dir= ectory
$ ls -lh /var/lib/cassa= ndra/commitlog/
total 128K
-rw-r--r-- 1 root root 439 2011-07-15 11:5= 7 CommitLog-1310702265959.log
-rw-r--r-- 1 root root 125 2011-07-15 11:58 CommitLog-1310702291383.log
=




2011/7/14 Zhu Han <schumi.han@gmail.com>:
> Jonathan,
>
> But all the old non-empty log segments are kept on the disk. And cassa= ndra
> takes some time to apply the operations from these closed log segments= after
> restart of the process.
>
> Is it expected?
>
> best regards,
> =BA=AB=D6=F1(Zhu Han)
>
> =BC=E1=B9=FB=C6=CC=D7=D3, =D7=EE=BC=F2=B5=A5=D2=D7=D3=C3=B5=C4=D4=C6= =B4=E6=B4=A2
> =CD=AC=B2=BD=CE=C4=BC=FE, =B7=D6=CF=ED=D5=D5=C6=AC, =CE=C4=B5=B5=B1=B8= =B7=DD!
>
>
>
> 2011/7/15 Jonathan Ellis <jbellis@gmail.com>
>>
>> It's expected to have a new, empty segment after drain complet= es.
>>
>> 2011/7/14 Zhu Han <schumi.han@gmail.com>:
>> > The deployed version is based on 0.6.13.
>> >
>> > After "nodetool drain" is invoked on one of the nod= es, the commit log is
>> > not
>> > emptied. Is this the expected behavior? If so, how can I rena= me a column
>> > family on 0.6.x branch?
>> >
>> > Here is the log output:
>> > "
>> > INFO [COMMIT-LOG-WRITER] 2011-07-15 00:39:49,541 CommitLogSeg= ment.java
>> > (line
>> > 50) Creating new commitlog segment
>> > /var/lib/cassandra/commitlog/CommitLog-1310661589541.log
>> >  INFO [RMI TCP Connection(8)-202.120.2.16] 2011-07-15 00= :39:49,544
>> > StorageService.java (line 391) Node is drained
>> > "
>> >
>> > I saw an issue here, but it was reported against 0.8.x branch= .
>> > https://issues.apache.org/jira/browse/CASSANDRA-2874<= /a>
>> >
>> > best regards,
>> > =BA=AB=D6=F1(Zhu Han)
>> >
>> > =BC=E1=B9=FB=C6=CC=D7=D3, =D7=EE=BC=F2=B5=A5=D2=D7=D3=C3=B5= =C4=D4=C6=B4=E6=B4=A2
>> > =CD=AC=B2=BD=CE=C4=BC=FE, =B7=D6=CF=ED=D5=D5=C6=AC, =CE=C4=B5= =B5=B1=B8=B7=DD!
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra supp= ort
>> http://www.d= atastax.com
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.c= om


--485b397dd083c64e0104a826c561--