Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 244F9EDD1 for ; Tue, 12 Mar 2013 06:08:12 +0000 (UTC) Received: (qmail 98340 invoked by uid 500); 12 Mar 2013 06:08:09 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 98283 invoked by uid 500); 12 Mar 2013 06:08:09 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 98268 invoked by uid 99); 12 Mar 2013 06:08:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Mar 2013 06:08:09 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a48.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Mar 2013 06:08:04 +0000 Received: from homiemail-a48.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a48.g.dreamhost.com (Postfix) with ESMTP id 39E0B4F805C for ; Mon, 11 Mar 2013 23:07:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=CyusKER1Ep591RlflTzpWlDCiM c=; b=gVLBkYHzwjrsauuzxbwuYSOl+hC94yxBJOyJ4YfZ/BqvSSZlgEnyn+Wejl BZJC1ohjDvCKLJCIjhhD9R8gZjwCCGwLYQ22bhCwdynp1R/9Pehz4h/665Z7f5y1 WcLPZrq4PzSbG3wdv/umQ2iIHVgMsnYpBT0xMEgvPVG7gQkis= Received: from [192.168.168.131] (c-98-234-52-29.hsd1.ca.comcast.net [98.234.52.29]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a48.g.dreamhost.com (Postfix) with ESMTPSA id EF4924F805B for ; Mon, 11 Mar 2013 23:07:43 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_9860BB21-6ECB-4C88-ABE5-5CA21220C9D1" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: HintedHandoff IOError? Date: Mon, 11 Mar 2013 23:07:42 -0700 References: <9C2950D8-6B89-4649-BEF9-DFAACAF1EF30@ecyrd.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_9860BB21-6ECB-4C88-ABE5-5CA21220C9D1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 What version of cassandra are you using? I would stop each node and delete the hints. If it happens again I could = either indicate a failing disk or a bug.=20 Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 11/03/2013, at 2:13 PM, Robert Coli = wrote: > On Mon, Mar 11, 2013 at 7:05 AM, Janne Jalkanen > wrote: >> I keep seeing these in my log. Three-node cluster, one node is = working fine, but two other nodes have increased latencies and these in = the error logs (might of course be unrelated). No obvious GC pressure, = no disk errors that I can see. Ubuntu 12.04 on EC2, Java 7. Repair is = run regularly. >>=20 >> My two questions: 1) should I worry, and 2) what might be going on, = and 3) is there any way to get rid of these? Can I just blow my = HintedHandoff table to smithereens? >=20 > = http://svn.apache.org/repos/asf/cassandra/trunk/src/java/org/apache/cassan= dra/io/sstable/IndexHelper.java > " > public static Filter defreezeBloomFilter(FileDataInput file, long > maxSize, boolean useOldBuffer) throws IOException > { > int size =3D file.readInt(); > if (size > maxSize || size <=3D 0) > throw new EOFException("bloom filter claims to be " + size > + " bytes, longer than entire row size " + maxSize); > ByteBuffer bytes =3D file.readBytes(size); > " >=20 > Based on the above, I would suspect either a zero byte -Filter.db file > or a corrupt one. Probably worry a little bit, but only a little bit > unless your cluster is RF=3D1. >=20 > =3DRob --Apple-Mail=_9860BB21-6ECB-4C88-ABE5-5CA21220C9D1 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 What = version of cassandra are you using?
I would stop each node and = delete the hints. If it happens again I could either indicate a failing = disk or a bug. 

Cheers

http://www.thelastpickle.com

On 11/03/2013, at 2:13 PM, Robert Coli <robert.d.a.coli@gmail.com>= ; wrote:

On Mon, Mar 11, 2013 at 7:05 AM, Janne Jalkanen
<janne.jalkanen@ecyrd.com> = wrote:
I keep seeing these in my log. =  Three-node cluster, one node is working fine, but two other nodes = have increased latencies and these in the error logs (might of course be = unrelated). No obvious GC pressure, no disk errors that I can see. =  Ubuntu 12.04 on EC2, Java 7. Repair is run regularly.

My = two questions: 1) should I worry, and 2) what might be going on, and 3) = is there any way to get rid of these? Can I just blow my HintedHandoff = table to smithereens?

http://svn.apache.org/repos/asf/c= assandra/trunk/src/java/org/apache/cassandra/io/sstable/IndexHelper.java
"
public static Filter defreezeBloomFilter(FileDataInput file, = long
maxSize, boolean useOldBuffer) throws IOException
=    {
       int = size =3D file.readInt();
=        if (size > maxSize || size = <=3D 0)
=            throw = new EOFException("bloom filter claims to be " + size
+ " bytes, = longer than entire row size " + maxSize);
=        ByteBuffer bytes =3D = file.readBytes(size);
"

Based on the above, I would suspect = either a zero byte -Filter.db file
or a corrupt one. Probably worry a = little bit, but only a little bit
unless your cluster is = RF=3D1.

=3DRob

= --Apple-Mail=_9860BB21-6ECB-4C88-ABE5-5CA21220C9D1--