Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 44D52B9FB for ; Wed, 4 Jan 2012 08:20:00 +0000 (UTC) Received: (qmail 30264 invoked by uid 500); 4 Jan 2012 08:19:58 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 29384 invoked by uid 500); 4 Jan 2012 08:19:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 29350 invoked by uid 99); 4 Jan 2012 08:19:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2012 08:19:29 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a55.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2012 08:19:21 +0000 Received: from homiemail-a55.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a55.g.dreamhost.com (Postfix) with ESMTP id EBA4012C0DB for ; Wed, 4 Jan 2012 00:18:55 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=W6+MXpDEKA W7pJPQXfoZzw2SgFY+ive0ii3oqzjajUWTZJ6NTFyyVGOV5rdYzEdGpotNT/TU/h 8EHigexW9qohtjlVDA7JTk7zPPj0vusJSIsoPNTPzUNVBPG1WpEmDzLatz8AepcZ yEwhd5Q8a+bE9EhK+x/tUp1Ip7VV6mqQs= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=gZVYE34Fq0YR7biw C53IhViGbkk=; b=0t4D7Cd4QlCedhrKlkiVVfQyJYAu3pCR+lupr5imM8pACroc 8K6qMeVlwWdVtDYAg3NpZ/72PW6+R92+WtJV7/8eg6OmTBfG8UMqQykoFooLNpgx Q0zcbM4sk8nVZbKJWr0ci+QOfqG1WP1/m5eUIdSIlrbHXJGItWpKNtMJKT8= Received: from [172.16.1.4] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a55.g.dreamhost.com (Postfix) with ESMTPSA id 2132D12C0DD for ; Wed, 4 Jan 2012 00:18:54 -0800 (PST) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: multipart/alternative; boundary="Apple-Mail=_B3D72DD5-2BEF-454C-ADF6-A6F980959A22" Subject: Re: Dealing with "Corrupt (negative) value length encountered" Date: Wed, 4 Jan 2012 21:18:51 +1300 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <85BC5E7D-DA1D-4186-B516-D02767DFE2F2@thelastpickle.com> X-Mailer: Apple Mail (2.1251.1) --Apple-Mail=_B3D72DD5-2BEF-454C-ADF6-A6F980959A22 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > I was able to scrub the node the repair that failed was running on. = Are you saying the error could be displayed on that node but the bad = data coming from another node ? Yes. The error occurred the node was receiving a data stream from = another, you will need to clean the source of the data. You can either = crawl through the logs or scrub the entire cluster.=20 Cheers =20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 4/01/2012, at 9:15 AM, Philippe wrote: > I was able to scrub the node the repair that failed was running on. = Are you saying the error could be displayed on that node but the bad = data coming from another node ? >=20 > Log inspection also showed many of these, they seem to happen around = when a stream transfer finishes. > ERROR [Thread-550876] 2012-01-03 16:35:31,922 = AbstractCassandraDaemon.java (line 139) Fatal exception in thread = Thread[Thread-550876,5,main] > java.lang.IllegalArgumentException > at = sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:586) > at = org.apache.cassandra.streaming.IncomingStreamReader.readnwrite(IncomingStr= eamReader.java:110) > at = org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStrea= mReader.java:85) > at = org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamRea= der.java:61) > at = org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnectio= n.java:189) > at = org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.j= ava:117) >=20 > Thanks >=20 > 2012/1/2 aaron morton > I would try to nodetool scrub the data on the node that that sent the = bad data in the stream. You may be able to work which node from the = logs, or it may be easier to just scrub them all.=20 >=20 > Hope that helps.=20 > =20 > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com >=20 > On 31/12/2011, at 12:20 AM, Philippe wrote: >=20 >> Hello, >> Running a combination of 0.8.6 and 0.8.8 with RF=3D3, I am getting = the following while repairing one node (all other nodes completed = successfully). >> Can I just stop the instance, erase the SSTable and restart cleanup ? >> Thanks >>=20 >> ERROR [Thread-402484] 2011-12-29 14:51:03,687 = AbstractCassandraDaemon.java (line 139) Fatal exception in thread = Thread[Thread-402484,5,main] >> java.lang.RuntimeException: java.util.concurrent.ExecutionException: = java.io.IOError: java.io.IOException: Corrupt (negative) value length = encountered >> at = org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSes= sion.java:154) >> at = org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamRea= der.java:63) >> at = org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnectio= n.java:189) >> at = org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.j= ava:117) >> Caused by: java.util.concurrent.ExecutionException: java.io.IOError: = java.io.IOException: Corrupt (negative) value length encountered >> at = java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> at = org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSes= sion.java:138) >> ... 3 more >>=20 >=20 >=20 --Apple-Mail=_B3D72DD5-2BEF-454C-ADF6-A6F980959A22 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 I was able to scrub the node the repair = that failed was running on. Are you saying the error could be displayed = on that node but the bad data coming from another node = ?Yes. The error occurred the node was receiving a = data stream from another, you will need to clean the source of the data. = You can either crawl through the logs or scrub the entire = cluster. 

Cheers
 
http://www.thelastpickle.com

On 4/01/2012, at 9:15 AM, Philippe wrote:

I was able = to scrub the node the repair that failed was running on. Are you saying = the error could be displayed on that node but the bad data coming from = another node ?

Log inspection also showed many of = these, they seem to happen around when a stream transfer finishes.
ERROR [Thread-550876] 2012-01-03 16:35:31,922 = AbstractCassandraDaemon.java (line 139) Fatal exception in thread = Thread[Thread-550876,5,main]
java.lang.IllegalArgumentException<= /div>
        at = sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:586)
        at = org.apache.cassandra.streaming.IncomingStreamReader.readnwrite(IncomingStr= eamReader.java:110)
        at = org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStrea= mReader.java:85)
        at = org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamRea= der.java:61)
        at = org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnectio= n.java:189)
        at = org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.j= ava:117)

Thanks

2012/1/2 = aaron morton <aaron@thelastpickle.com>
I would try to nodetool scrub the = data on the node that that sent the bad data in the stream. You may be = able to work which node from the logs, or it may be easier to just scrub = them all. 

Hope that helps. 
 
-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 31/12/2011, at 12:20 AM, Philippe = wrote:

Hello,
Running a = combination of 0.8.6 and 0.8.8 with RF=3D3, I am getting the following = while repairing one node (all other nodes completed successfully).
Can I just stop the instance, erase the SSTable and restart cleanup = ?
Thanks

ERROR [Thread-402484] 2011-12-29 = 14:51:03,687 AbstractCassandraDaemon.java (line 139) Fatal exception in = thread = Thread[Thread-402484,5,main]
java.lang.RuntimeException: = java.util.concurrent.ExecutionException: java.io.IOError: = java.io.IOException: Corrupt (negative) value length encountered
        at = org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSes= sion.java:154)
        at = org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamRea= der.java:63)
        at = org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnectio= n.java:189)
        at = org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.j= ava:117)
Caused by: java.util.concurrent.ExecutionException: = java.io.IOError: java.io.IOException: Corrupt (negative) value length = encountered
        at = java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
        at = java.util.concurrent.FutureTask.get(FutureTask.java:83)
  =       at = org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSes= sion.java:138)
        ... 3 = more

=



= --Apple-Mail=_B3D72DD5-2BEF-454C-ADF6-A6F980959A22--