cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terje Marthinussen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException
Date Fri, 10 Jun 2011 14:19:07 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047211#comment-13047211
] 

Terje Marthinussen commented on CASSANDRA-2752:
-----------------------------------------------

Chris: Same as the "compactionexecutor" or the infinite loop? (made CASSANDRA-2758 for that
just now).

Seems like the sstables here has been truncated for some reason.
Rowindexer iterates through a bunch of rows just fine.
Then it reaches the problem row.

For this row, it can get the key, it jumps pass the bloomfilter etc and when it is about to
read the column count, it fails and is trying to read at an offset which equals the length
of the file..
So far, all the stacktraces I have seen are all are on the column count read. 

A wild guess may be that it has failed to write the actual content (columns) of the last row
that was stream?

Unfortunately it does not to happen all the time, but it does only happen on the CF with secondary
indexes.





> repair fails with java.io.EOFException
> --------------------------------------
>
>                 Key: CASSANDRA-2752
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.8.0
>            Reporter: Terje Marthinussen
>
> Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
> INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java (line 234)
Queueing comparison #<Differencer #<TreeRequest manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e,
/1
> .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java (line 468)
Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out of sync for (JP,XXX) on
(Token(bytes[6e]),Token(bytes[313039])]
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java (line 485)
Performing streaming repair of 2 ranges for #<TreeRequest manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
> fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]>
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) Stream context
metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db sections=1 progress=0/36592 - 0%],
1 sstables.
>  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java (line 174) Streaming
to /1.10.42.82
> ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[CompactionExecutor:9,1,main]
> java.io.EOFException
>         at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
>         at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
>         at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
>         at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
>         at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
>         at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> On .82
> ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[CompactionExecutor:12,1,main]
> java.io.EOFException
>         at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
>         at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
>         at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
>         at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
>         at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
>         at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 113) Fatal
exception in thread Thread[Thread-132,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException
>         at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
>         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
>         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
>         at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:136)
>         ... 3 more
> Caused by: java.io.EOFException
>         at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
>         at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
>         at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
>         at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
>         at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
>         at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Looks to me like the receiving side fails first.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message