Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0153C9165 for ; Thu, 14 Mar 2013 04:58:39 +0000 (UTC) Received: (qmail 70172 invoked by uid 500); 14 Mar 2013 04:58:36 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 70122 invoked by uid 500); 14 Mar 2013 04:58:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 70086 invoked by uid 99); 14 Mar 2013 04:58:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Mar 2013 04:58:34 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a45.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Mar 2013 04:58:29 +0000 Received: from homiemail-a45.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a45.g.dreamhost.com (Postfix) with ESMTP id A60EC48074 for ; Wed, 13 Mar 2013 21:58:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=5WZOthEih/QjffIVsXRwquBPKC c=; b=HIlEDgXfLYqEbr4B4T21SAvjKC87EhgHIJU3j3HtSpBO6UV+M64XKG6eq5 moL1isWEF4OV4fN4ktAVhr5UGtYKOb1cvdJsxY0Z5mhLL9ws2vWfKvRQ0om9fLxq 3JkFCmk/iI8bwKqD6pJXcuNXCfjmpk66Bx3oVqFv994Cl6/1w= Received: from [192.168.168.142] (c-98-234-52-29.hsd1.ca.comcast.net [98.234.52.29]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a45.g.dreamhost.com (Postfix) with ESMTPSA id B124748073 for ; Wed, 13 Mar 2013 21:58:06 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_09E620D8-C19C-4858-AFAF-31191D2EBB1A" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: HintedHandoff IOError? Date: Wed, 13 Mar 2013 21:58:05 -0700 References: <9C2950D8-6B89-4649-BEF9-DFAACAF1EF30@ecyrd.com> <152C625F-470B-49CD-A7E1-CF3289313B8C@ecyrd.com> To: user@cassandra.apache.org In-Reply-To: <152C625F-470B-49CD-A7E1-CF3289313B8C@ecyrd.com> X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_09E620D8-C19C-4858-AFAF-31191D2EBB1A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > What is the sanctioned way of removing hints? rm -f = HintsColumnFamily*? Truncate from CLI? There is a JMX command to do it for a particular node.=20 But if you just want to remove all of them, stop and delete the files.=20= > the only one with zero size are the -tmp- files. It seems odd=85 Temp files are created during compaction and flushing sstables.=20 Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 11/03/2013, at 11:19 PM, Janne Jalkanen = wrote: >=20 > Oops, forgot to mention that, did I=85 Cass 1.1.10.=20 >=20 > What is the sanctioned way of removing hints? rm -f = HintsColumnFamily*? Truncate from CLI? >=20 > This is ls -l of my /system/HintsColumnFamily/ btw - the only one with = zero size are the -tmp- files. It seems odd=85 >=20 > -rw-rw-r-- 1 ubuntu ubuntu 86373144 Jan 26 21:39 = system-HintsColumnFamily-hf-11-Data.db > -rw-rw-r-- 1 ubuntu ubuntu 80 Jan 26 21:39 = system-HintsColumnFamily-hf-11-Digest.sha1 > -rw-rw-r-- 1 ubuntu ubuntu 976 Jan 26 21:39 = system-HintsColumnFamily-hf-11-Filter.db > -rw-rw-r-- 1 ubuntu ubuntu 11 Jan 26 21:39 = system-HintsColumnFamily-hf-11-Index.db > -rw-rw-r-- 1 ubuntu ubuntu 4348 Jan 26 21:39 = system-HintsColumnFamily-hf-11-Statistics.db > -rw-rw-r-- 1 ubuntu ubuntu 569 Feb 27 08:33 = system-HintsColumnFamily-hf-23-Data.db > -rw-rw-r-- 1 ubuntu ubuntu 80 Feb 27 08:33 = system-HintsColumnFamily-hf-23-Digest.sha1 > -rw-rw-r-- 1 ubuntu ubuntu 1936 Feb 27 08:33 = system-HintsColumnFamily-hf-23-Filter.db > -rw-rw-r-- 1 ubuntu ubuntu 11 Feb 27 08:33 = system-HintsColumnFamily-hf-23-Index.db > -rw-rw-r-- 1 ubuntu ubuntu 4356 Feb 27 08:33 = system-HintsColumnFamily-hf-23-Statistics.db > -rw-rw-r-- 1 ubuntu ubuntu 5500155 Feb 27 08:57 = system-HintsColumnFamily-hf-24-Data.db > -rw-rw-r-- 1 ubuntu ubuntu 80 Feb 27 08:57 = system-HintsColumnFamily-hf-24-Digest.sha1 > -rw-rw-r-- 1 ubuntu ubuntu 16 Feb 27 08:57 = system-HintsColumnFamily-hf-24-Filter.db > -rw-rw-r-- 1 ubuntu ubuntu 26 Feb 27 08:57 = system-HintsColumnFamily-hf-24-Index.db > -rw-rw-r-- 1 ubuntu ubuntu 4340 Feb 27 08:57 = system-HintsColumnFamily-hf-24-Statistics.db > -rw-rw-r-- 1 ubuntu ubuntu 0 Feb 27 08:57 = system-HintsColumnFamily-tmp-hf-25-Data.db > -rw-rw-r-- 1 ubuntu ubuntu 0 Feb 27 08:57 = system-HintsColumnFamily-tmp-hf-25-Index.db >=20 >=20 > /Janne >=20 > On Mar 12, 2013, at 08:07 , aaron morton = wrote: >=20 >> What version of cassandra are you using? >> I would stop each node and delete the hints. If it happens again I = could either indicate a failing disk or a bug.=20 >>=20 >> Cheers >>=20 >> ----------------- >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >>=20 >> @aaronmorton >> http://www.thelastpickle.com >>=20 >> On 11/03/2013, at 2:13 PM, Robert Coli = wrote: >>=20 >>> On Mon, Mar 11, 2013 at 7:05 AM, Janne Jalkanen >>> wrote: >>>> I keep seeing these in my log. Three-node cluster, one node is = working fine, but two other nodes have increased latencies and these in = the error logs (might of course be unrelated). No obvious GC pressure, = no disk errors that I can see. Ubuntu 12.04 on EC2, Java 7. Repair is = run regularly. >>>>=20 >>>> My two questions: 1) should I worry, and 2) what might be going on, = and 3) is there any way to get rid of these? Can I just blow my = HintedHandoff table to smithereens? >>>=20 >>> = http://svn.apache.org/repos/asf/cassandra/trunk/src/java/org/apache/cassan= dra/io/sstable/IndexHelper.java >>> " >>> public static Filter defreezeBloomFilter(FileDataInput file, long >>> maxSize, boolean useOldBuffer) throws IOException >>> { >>> int size =3D file.readInt(); >>> if (size > maxSize || size <=3D 0) >>> throw new EOFException("bloom filter claims to be " + = size >>> + " bytes, longer than entire row size " + maxSize); >>> ByteBuffer bytes =3D file.readBytes(size); >>> " >>>=20 >>> Based on the above, I would suspect either a zero byte -Filter.db = file >>> or a corrupt one. Probably worry a little bit, but only a little bit >>> unless your cluster is RF=3D1. >>>=20 >>> =3DRob >>=20 >=20 --Apple-Mail=_09E620D8-C19C-4858-AFAF-31191D2EBB1A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
What = is the sanctioned way of removing hints? rm -f HintsColumnFamily*? = Truncate from CLI?
There is a JMX command to do it for a particular = node. 
But = if you just want to remove all of them, stop and delete the = files. 
 the only one with = zero size are the -tmp- files.  It seems = odd=85
Temp files are created during compaction and = flushing sstables. 
Cheers


http://www.thelastpickle.com

On 11/03/2013, at 11:19 PM, Janne Jalkanen <Janne.Jalkanen@ecyrd.com> = wrote:

aaron@thelastpickle.com> = wrote:

What = version of cassandra are you using?
I would stop each node and = delete the hints. If it happens again I could either indicate a failing = disk or a bug. 

Cheers

http://www.thelastpickle.com

On 11/03/2013, at 2:13 PM, Robert Coli <robert.d.a.coli@gmail.com>= ; wrote:

On Mon, Mar 11, 2013 at 7:05 AM, Janne Jalkanen
<janne.jalkanen@ecyrd.com> = wrote:
I keep seeing these in my log. =  Three-node cluster, one node is working fine, but two other nodes = have increased latencies and these in the error logs (might of course be = unrelated). No obvious GC pressure, no disk errors that I can see. =  Ubuntu 12.04 on EC2, Java 7. Repair is run regularly.

My = two questions: 1) should I worry, and 2) what might be going on, and 3) = is there any way to get rid of these? Can I just blow my HintedHandoff = table to smithereens?

http://svn.apache.org/repos/asf/c= assandra/trunk/src/java/org/apache/cassandra/io/sstable/IndexHelper.java
"
public static Filter defreezeBloomFilter(FileDataInput file, = long
maxSize, boolean useOldBuffer) throws IOException
=    {
       int = size =3D file.readInt();
=        if (size > maxSize || size = <=3D 0)
=            throw = new EOFException("bloom filter claims to be " + size
+ " bytes, = longer than entire row size " + maxSize);
=        ByteBuffer bytes =3D = file.readBytes(size);
"

Based on the above, I would suspect = either a zero byte -Filter.db file
or a corrupt one. Probably worry a = little bit, but only a little bit
unless your cluster is = RF=3D1.

=3DRob

<= /div>

= --Apple-Mail=_09E620D8-C19C-4858-AFAF-31191D2EBB1A--