cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuki Morishita (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5418) streaming fails
Date Wed, 10 Apr 2013 15:26:16 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627903#comment-13627903
] 

Yuki Morishita commented on CASSANDRA-5418:
-------------------------------------------

Igor, can you provide more info about this?
Do you see the same AssertionError for every CFs or the specific one? If the latter, can you
post the definition of that CF?

                
> streaming fails
> ---------------
>
>                 Key: CASSANDRA-5418
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5418
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.2, 1.2.3
>         Environment: 5 nodes, vnodes enabled, encryption disabled, compression enabled,
RackInferring snitch, Centos 6, Oracle JVM with JNA enabled.
>            Reporter: Igor Ivanov
>            Priority: Critical
>
> When I run *nodetool repair* on cas01 node it get's stuck at some point.
> I see following exceptions in cas01 system.log:
> {quote}
> ERROR [Streaming to /10.10.45.60:28] 2013-04-02 09:03:55,353 CassandraDaemon.java (line
132) Exception in thread Thread[Streaming to /10.10.45.60:28,5,main]
> java.lang.RuntimeException: java.io.EOFException
> 	at com.google.common.base.Throwables.propagate(Throwables.java:160)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 	at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(Unknown Source)
> 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193)
> 	at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114)
> 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	... 3 more
> ERROR [Thread-2076] 2013-04-02 09:07:12,261 CassandraDaemon.java (line 132) Exception
in thread Thread[Thread-2076,5,main]
> java.lang.AssertionError: incorrect row data size 130921 written to /var/lib/cassandra/data/EDITED/content_list/footballsite-content_list-tmp-ib-3660-Data.db;
correct is 131074
> 	at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
> 	at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
> 	at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> 	at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
> 	at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
> 	at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> {quote}
> On other machines there are some exceptions too:
> {quote}
> ERROR [Thread-1424] 2013-04-02 09:07:12,248 CassandraDaemon.java (line 132) Exception
in thread Thread[Thread-1424,5,main]
> java.lang.AssertionError: incorrect row data size 130921 written to /var/lib/cassandra/data/EDITED/content_list/footballsite-content_list-tmp-ib-2268-Data.db;
correct is 131074
> 	at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
> 	at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
> 	at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> 	at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
> 	at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
> 	at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> ERROR [Streaming to /10.10.45.58:55] 2013-04-02 09:07:12,263 CassandraDaemon.java (line
132) Exception in thread Thread[Streaming to /10.10.45.58:55,5,main]
> java.lang.RuntimeException: java.io.EOFException
> 	at com.google.common.base.Throwables.propagate(Throwables.java:160)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 	at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(Unknown Source)
> 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:193)
> 	at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:114)
> 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	... 3 more
> {quote}
> Then I see frozen status in *nodetool netstats* and repair never completes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message