cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Frank Cooper <coop...@yahoo-inc.com>
Subject RE: Server cannot startup after shutdown
Date Wed, 26 Aug 2009 06:03:46 GMT
Hi, Jonathan,

I have been trying to shutdown and restart Cassandra again this morning. I still get the malformed
entry bug (which you say below your patch fixes.) I also get:

ERROR - Exception encountered during startup.
java.lang.NegativeArraySizeException
        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:274)
        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171)
Exception encountered during startup.
java.lang.NegativeArraySizeException
        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:274)
        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171)

No out of memory error this time, though.

I'm also curious about your comment "I introduced a regression where it couldn't handle the
last entry in the commitlog being incomplete." Does the last entry in the commit log being
incomplete mean that the last update or set of updates are not fully committed to the log?
And therefore they are lost? I thought since I had set "<CommitLogSync>true</CommitLogSync>"
that all updates would be fully flushed before returning to the caller.

(BTW thanks for all the help with setting up Cassandra, it really made it easier to run experiments...)

brian
________________________________________
From: Jonathan Ellis [jbellis@gmail.com]
Sent: Monday, August 24, 2009 12:51 PM
To: cassandra-user@incubator.apache.org
Subject: Re: Server cannot startup after shutdown

Oops, my bad -- that patch has been sitting unreviewed in
CASSANDRA-370.  I thought it was in trunk by now.  I'll try to get
someone to review that today.

-Jonathan

On Wed, Aug 19, 2009 at 9:46 PM, Jonathan Ellis<jbellis@gmail.com> wrote:
> The malformed input bug was fixed after beta1 and should be in a
> nightly build by now.  (I introduced a regression where it couldn't
> handle the last entry in the commitlog being incomplete.  So upgrading
> should be able to restart on the existing commitlogs.)
>
> The OOM puzzles me a little; I'm not sure how it could be unable to
> replay a mutation that it was able to write to the commitlog in the
> first place.  You could try setting the memtable object and memory
> thresholds lower temporarily and see if that leaves enough extra free
> to do the replay.
>
> -Jonathan
>
> On Wed, Aug 19, 2009 at 7:12 PM, Brian Frank
> Cooper<cooperb@yahoo-inc.com> wrote:
>> Hi folks,
>>
>> I'm using 0.4 beta1 and had six servers loaded with 20 GB of data per server. (In
this test, 10 KB per record, and 2 GB heap space allocated to the JVM.) I stopped the servers
(using what I think is the recommended method, the kill command). Upon trying to restart,
some servers threw a UTFDataFormatException, while others threw an OutOfMemoryError exception.
None of them started.
>>
>> Is this a known issue?
>>
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:274)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171)
>>
>> ERROR - Exception encountered during startup.
>> java.io.UTFDataFormatException: malformed input around byte 5497
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:639)
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>>        at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:218)
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:285)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171)
>> Exception encountered during startup.
>> java.io.UTFDataFormatException: malformed input around byte 5497
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:639)
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>>        at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:218)
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:285)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171)
>>
>> Thanks for the help!
>>
>> Brian
>

Mime
View raw message