Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Thu, 11 Apr 2013 03:33:16 +0000 (UTC)
From: "Yuki Morishita (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12640335.1364922661845.147627.1365651196696@arcas>
In-Reply-To: <JIRA.12640335.1364922661845@arcas>
References: <JIRA.12640335.1364922661845@arcas>
Subject: [jira] [Commented] (CASSANDRA-5418) streaming fails
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628632#comment-13628632 ] 

Yuki Morishita commented on CASSANDRA-5418:
-------------------------------------------

Igor, thanks for the patch.
I think that would probably work, since the only code path that could write extra bytes is there, but I want to confirm by writing unit test for this. I'm working on it right now.
                
> streaming fails
> ---------------
>
>                 Key: CASSANDRA-5418
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5418
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.2, 1.2.3
>         Environment: 5 nodes, vnodes enabled, encryption disabled, compression enabled, RackInferring snitch, Centos 6, Oracle JVM with JNA enabled.
>            Reporter: Igor Ivanov
>            Priority: Critical
>         Attachments: 5418-1.2.txt
>
>
> When I run *nodetool repair* on cas01 node it get's stuck at some point.
> I see following exceptions in cas01 system.log:
> {quote}
> ERROR [Streaming to /10.10.45.60:28] 2013-04-02 09:03:55,353 CassandraDaemon.java (line 132) Exception in thread Thread[Streaming to /10.10.45.60:28,5,main]
> java.lang.RuntimeException: java.io.EOFException
> 	at com.google.common.base.Throwables.propagate(Throwables.java:160)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 	at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(Unknown Source)
> 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193)
> 	at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114)
> 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	... 3 more
> ERROR [Thread-2076] 2013-04-02 09:07:12,261 CassandraDaemon.java (line 132) Exception in thread Thread[Thread-2076,5,main]
> java.lang.AssertionError: incorrect row data size 130921 written to /var/lib/cassandra/data/EDITED/content_list/footballsite-content_list-tmp-ib-3660-Data.db; correct is 131074
> 	at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
> 	at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
> 	at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> 	at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
> 	at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
> 	at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> {quote}
> On other machines there are some exceptions too:
> {quote}
> ERROR [Thread-1424] 2013-04-02 09:07:12,248 CassandraDaemon.java (line 132) Exception in thread Thread[Thread-1424,5,main]
> java.lang.AssertionError: incorrect row data size 130921 written to /var/lib/cassandra/data/EDITED/content_list/footballsite-content_list-tmp-ib-2268-Data.db; correct is 131074
> 	at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
> 	at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
> 	at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> 	at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
> 	at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
> 	at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> ERROR [Streaming to /10.10.45.58:55] 2013-04-02 09:07:12,263 CassandraDaemon.java (line 132) Exception in thread Thread[Streaming to /10.10.45.58:55,5,main]
> java.lang.RuntimeException: java.io.EOFException
> 	at com.google.common.base.Throwables.propagate(Throwables.java:160)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 	at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(Unknown Source)
> 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193)
> 	at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114)
> 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	... 3 more
> {quote}
> Then I see frozen status in *nodetool netstats* and repair never completes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira