cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Borgqvist (JIRA) <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5191) BufferOverflowException in CommitLogSegment
Date Tue, 29 Jan 2013 14:13:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565389#comment-13565389
] 

André Borgqvist commented on CASSANDRA-5191:
--------------------------------------------

I've added a catch and log print on row 265 in CommitLogSegment.

try
{
    buffer.putLong(checksum.getValue());	
}catch(BufferOverflowException e)
{
    logger.error("BUFFEROVERFLOW: mutation="+rowMutation + ", checksum="+checksum.getValue()
+ ", bufferLen=" + buffer.remaining());
    throw e;
}   

Result in log (column names contains colons from our application logic which looks a bit weird
in the printout, for example 'c:res:State' is a column name):

ERROR [COMMIT-LOG-WRITER] 2013-01-29 12:40:15,003 CommitLogSegment.java (line 271) BUFFEROVERFLOW:
mutation=RowMutation(keyspace='cake', key='376366343966643633653264313631626438383736316366316432333133653331653030633633336566653035626333',
modifications=[ColumnFamily(vouchers [c:res:State:false:7@1359459603537005,c:res:TemporaryState:true:4@1359459603537007,c:res:subscriberId:false:12@1359459603537003,c:res:transactionId:false:5@1359459603537001,])]),
checksum=3781223592, bufferLen=4


Noticed that "marked for deletion" was true on c:res:TemporaryState:true:4@1359459603537007
and I know we write with a very short ttl (5s), I suspected this has something to do with
it.

So I made a small test program that loops and inserts a column with a ttl of 5s, and reproduced
the error with it.
The program is run toward node 1 in the cluster and node 2 crashes after some minutes (I started
three instances on the test program to try to speed it up).

ERROR [COMMIT-LOG-WRITER] 2013-01-29 14:52:43,304 CommitLogSegment.java (line 271) BUFFEROVERFLOW:
mutation=RowMutation(keyspace='cake', key='31383037363038', modifications=[ColumnFamily(vouchers
[testcol:true:4@1359467551818,])]), checksum=2632894560, bufferLen=6

Just noticed NTP is not correctly configured and the system clock on node 2 is 12 seconds
ahead.



                
> BufferOverflowException in CommitLogSegment
> -------------------------------------------
>
>                 Key: CASSANDRA-5191
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5191
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.9
>         Environment: RHEL 2.6.32-220.el6.x86_64, jdk1.6.0_27
>            Reporter: André Borgqvist
>
> Running mixed reads, writes and deletes on a single column family in a two node cluster.
After a few minutes the following appears in the system log:
> ERROR [COMMIT-LOG-WRITER] 2013-01-25 12:49:55,955 AbstractCassandraDaemon.java (line
135) Exception in thread Thread[COMMIT-LOG-WRITER,5,main]
> java.nio.BufferOverflowException
> 	at java.nio.Buffer.nextPutIndex(Buffer.java:499)
> 	at java.nio.DirectByteBuffer.putLong(DirectByteBuffer.java:756)
> 	at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:265)
> 	at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:382)
> 	at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:50)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> 	at java.lang.Thread.run(Thread.java:662)
> Possibly related to https://issues.apache.org/jira/browse/CASSANDRA-3615

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message