From user-return-30062-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Nov 12 09:15:40 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 60958D63A for ; Mon, 12 Nov 2012 09:15:40 +0000 (UTC) Received: (qmail 65312 invoked by uid 500); 12 Nov 2012 09:15:37 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 64791 invoked by uid 500); 12 Nov 2012 09:15:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 64740 invoked by uid 99); 12 Nov 2012 09:15:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Nov 2012 09:15:34 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of post@fantasista.no designates 213.236.237.140 as permitted sender) Received: from [213.236.237.140] (HELO mx1.mailserveren.com) (213.236.237.140) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Nov 2012 09:15:29 +0000 Received: from localhost ([127.0.0.1]) by mx1.mailserveren.com with esmtpa (Exim 4.77) (envelope-from ) id 1TXq6k-00067N-6E for user@cassandra.apache.org; Mon, 12 Nov 2012 10:15:06 +0100 Message-Id: <9a1cde3096ffb3bd89c73552f0fbe458bc321107@pop3.fantasista.no> From: "Vegard Berget" Reply-To: "Vegard Berget" To: user@cassandra.apache.org X-Mailer: Atmail 6.5.0.10885 X-Originating-IP: 46.19.16.3 in-reply-to: Subject: Re: Hinted Handoff runs every ten minutes Date: Mon, 12 Nov 2012 10:15:06 +0100 Content-Type: multipart/alternative; boundary="=_4a4a0ab94606159aad7eda9d0acfcb29" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --=_4a4a0ab94606159aad7eda9d0acfcb29 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable =C2=A0Hi,=0AHintsColumnFamily directory on Node 1 (the first to be upgra= ded):1.8K=0AOct 27 11:27 system-HintsColumnFamily-hf-2-Data.db79 Oct 27= 11:27=0Asystem-HintsColumnFamily-hf-2-Digest.sha1496 Oct 27 11:27=0Asys= tem-HintsColumnFamily-hf-2-Filter.db26 Oct 27 11:27=0Asystem-HintsColumn= Family-hf-2-Index.db4.3K Oct 27 11:27=0Asystem-HintsColumnFamily-hf-2-St= atistics.db=0A=0AHintsColumnFamily-directory on Node 2 (the second to be= upgraded):12K=0AOct 27 11:37 system-HintsColumnFamily-hf-3-Data.db79 Oc= t 27 11:37=0Asystem-HintsColumnFamily-hf-3-Digest.sha1976 Oct 27 11:37= =0Asystem-HintsColumnFamily-hf-3-Filter.db11 Oct 27 11:37=0Asystem-Hints= ColumnFamily-hf-3-Index.db4.3K Oct 27 11:37=0Asystem-HintsColumnFamily-h= f-3-Statistics.db=0A11:37 is exactly ten minutes after the node was star= ted after=0Aupgrading, according to the logs, which means that the first= node's=0Asstables is generated when the second node starts.=0A=0AThe ss= table2json is more or less equal for both nodes, but node 1=0Astarts wit= h "40000000000000000000000000000000" and node 2 with=0A"00".The rest loo= ks like this:"00": {"f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX":=0A{"deletedAt": -9= 223372036854775808, "subColumns": []},=0A"f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX= XXXXXXXX": {"deletedAt":=0A-9223372036854775808, "subColumns": []},(The= f7XXXXXXXXXXXXXXXXX is=0Arepeated with multiple different keys, but all= are above=0Af7XXXXXXXXXXXXXXXXXXXXX. =C2=A0The deletedAt is always equa= l - on both=0Anodes)=0AOf course, this could be caused by me doing a mis= take while upgrading.=0A=C2=A0I have not tried to compact, but repair ru= ns every night. =C2=A0It's a=0Alow traffic cluster to test cassandra, if= that matters.=0A.vegard,=0A=0A----- Original Message -----=0AFrom: user= @cassandra.apache.org=0ATo:=0ACc:=0ASent:Sun, 11 Nov 2012 13:16:01 -0600= =0ASubject:Re: Hinted Handoff runs every ten minutes=0A=0A How many hint= sstables are there? What does sstable2json show?=0A=0A On Thu, Nov 8, 2= 012 at 3:23 PM, Mike Heffner wrote:=0A > Is there a ticket open for thi= s for 1.1.6?=0A >=0A > We also noticed this after upgrading from 1.1.3 t= o 1.1.6. Every=0Anode runs a=0A > 0 row hinted handoff every 10 minutes.= N-1 nodes hint to the same=0Anode,=0A > while that node hints to anothe= r node.=0A >=0A >=0A > On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget w= rote:=0A >>=0A >> Hi,=0A >>=0A >> I have the exact same problem with 1.1= 6. HintsColumnFamily=0Aconsists of=0A >> one row (Rowkey 00, nothing mo= re). The "problem" started after=0Aupgrading=0A >> from 1.1.4 to 1.1.6.= Every ten minutes HintedHandoffManager starts=0Aand=0A >> finishes afte= r sending "0 rows".=0A >>=0A >> .vegard,=0A >>=0A >>=0A >>=0A >> ----- O= riginal Message -----=0A >> From:=0A >> user@cassandra.apache.org=0A >>= =0A >> To:=0A >> =0A >> Cc:=0A >>=0A >> Sent:=0A >> Mon, 29 Oct 2012 23:= 45:30 +0100=0A >>=0A >> Subject:=0A >> Re: Hinted Handoff runs every ten= minutes=0A >>=0A >>=0A >> Dne 29.10.2012 23:24, Stephen Pierce napsal(a= ):=0A >> > I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.=0A= >> >=0A >> > How can I check to see why it keeps running HintedHandoff?= =0A >> you have tombstone is system.HintsColumnFamily use list command i= n=0A >> cassandra-cli to check=0A >>=0A >=0A >=0A >=0A > --=0A >=0A > Mi= ke Heffner =0A > Librato, Inc.=0A >=0A >=0A=0A -- =0A Jonathan Ellis=0A= Project Chair, Apache Cassandra=0A co-founder of DataStax, the source f= or professional Cassandra support=0A http://www.datastax.com=0A --=_4a4a0ab94606159aad7eda9d0acfcb29 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable =C2=A0Hi,

HintsColumnFamil= y directory on Node 1 (the first to be upgraded):
1.8K Oc= t 27 11:27 system-HintsColumnFamily-hf-2-Data.db
79 Oct 27 11:= 27 system-HintsColumnFamily-hf-2-Digest.sha1
496 Oct 27 11:27= system-HintsColumnFamily-hf-2-Filter.db
26 Oct 27 11:27 syste= m-HintsColumnFamily-hf-2-Index.db
4.3K Oct 27 11:27 system-Hin= tsColumnFamily-hf-2-Statistics.db


HintsColumnFamily-directory on Node 2 (the second to be upgr= aded):
12K Oct 27 11:37 system-HintsColumnFamily-hf-3-Dat= a.db
79 Oct 27 11:37 system-HintsColumnFamily-hf-3-Digest.sha1=
976 Oct 27 11:37 system-HintsColumnFamily-hf-3-Filter.db
11 Oct 27 11:37 system-HintsColumnFamily-hf-3-Index.db
4= 3K Oct 27 11:37 system-HintsColumnFamily-hf-3-Statistics.db
<= br />
11:37 is exactly ten minutes after the node was started= after upgrading, according to the logs, which means that the first node= 's sstables is generated when the second node starts.



The sstable2json is more or l= ess equal for both nodes, but node 1 starts with "4000000000000000000000= 0000000000" and node 2 with "00".
The rest looks like this:
"00": {"f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX": {"deletedAt": -92233720= 36854775808, "subColumns": []}, "f7XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"= : {"deletedAt": -9223372036854775808, "subColumns": []},
(The= f7XXXXXXXXXXXXXXXXX is repeated with multiple different keys, but all a= re above f7XXXXXXXXXXXXXXXXXXXXX. =C2=A0The deletedAt is always equal -= on both nodes)

Of course, this could be caus= ed by me doing a mistake while upgrading. =C2=A0I have not tried to comp= act, but repair runs every night. =C2=A0It's a low traffic cluster to te= st cassandra, if that matters.

.vegard,
=



----- Original Messa= ge -----
From:
user@cassandra.apache.org
To:
<user@cassandra.apac= he.org>
Cc:

Sent:
Sun, 11 Nov 2012 13:16:01 -0600
Subject:
Re: Hinted Handoff runs= every ten minutes


=0AHow many hint sstables are there?= What does sstable2json show?

=0AOn Thu, Nov 8, 2012 at 3:23= PM, Mike Heffner <mike@librato.com> wrote:
=0A> Is there= a ticket open for this for 1.1.6?
=0A>
=0A> We also not= iced this after upgrading from 1.1.3 to 1.1.6. Every node runs a
= =0A> 0 row hinted handoff every 10 minutes. N-1 nodes hint to the sam= e node,
=0A> while that node hints to another node.
=0A>=
=0A>
=0A> On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berge= t <post@fantasista.no> wrote:
=0A>>
=0A>> Hi= ,
=0A>>
=0A>> I have the exact same problem with 1= 1.6. HintsColumnFamily consists of
=0A>> one row (Rowkey 00= , nothing more). The "problem" started after upgrading
=0A>>= ; from 1.1.4 to 1.1.6. Every ten minutes HintedHandoffManager starts an= d
=0A>> finishes after sending "0 rows".
=0A>>=0A>> .vegard,
=0A>>
=0A>>
=0A>&g= t;
=0A>> ----- Original Message -----
=0A>> From:<= br />=0A>> user@cassandra.apache.org
=0A>>
=0A>= > To:
=0A>> <user@cassandra.apache.org>
=0A>= > Cc:
=0A>>
=0A>> Sent:
=0A>> Mon, 2= 9 Oct 2012 23:45:30 +0100
=0A>>
=0A>> Subject:=0A>> Re: Hinted Handoff runs every ten minutes
=0A>>= ;
=0A>>
=0A>> Dne 29.10.2012 23:24, Stephen Pierce= napsal(a):
=0A>> > I'm running 1.1.5; the bug says it's f= ixed in 1.0.9/1.1.0.
=0A>> >
=0A>> > How can= I check to see why it keeps running HintedHandoff?
=0A>> you= have tombstone is system.HintsColumnFamily use list command in
=0A= >> cassandra-cli to check
=0A>>
=0A>
=0A&g= t;
=0A>
=0A> --
=0A>
=0A> Mike Heffner= <mike@librato.com>
=0A> Librato, Inc.
=0A>
=0A>



=0A--
=0AJonathan Ellis
=0APr= oject Chair, Apache Cassandra
=0Aco-founder of DataStax, the source= for professional Cassandra support
=0Ahttp://www.datastax.com
--=_4a4a0ab94606159aad7eda9d0acfcb29--