incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Frank Cooper <coop...@yahoo-inc.com>
Subject RE: Server cannot startup after shutdown
Date Thu, 20 Aug 2009 22:58:49 GMT
I haven't had a chance to play with that yet. I'm just trying to get a bunch of data loaded
so I can run some tests. Once the tests are done I will look at starting and stopping servers
again.

The tokens thing helped out quite a lot.

brian

-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com] 
Sent: Thursday, August 20, 2009 10:50 AM
To: cassandra-user@incubator.apache.org
Subject: Re: Server cannot startup after shutdown

I saw that you're testing with different tokens now -- how did the
replay OOM work out?

On Wed, Aug 19, 2009 at 10:54 PM, Brian Frank
Cooper<cooperb@yahoo-inc.com> wrote:
> Thanks for the reply. I'll try playing with the memory settings.
>
> brian
> ________________________________________
> From: Jonathan Ellis [jbellis@gmail.com]
> Sent: Wednesday, August 19, 2009 7:46 PM
> To: cassandra-user@incubator.apache.org
> Subject: Re: Server cannot startup after shutdown
>
> The malformed input bug was fixed after beta1 and should be in a
> nightly build by now.  (I introduced a regression where it couldn't
> handle the last entry in the commitlog being incomplete.  So upgrading
> should be able to restart on the existing commitlogs.)
>
> The OOM puzzles me a little; I'm not sure how it could be unable to
> replay a mutation that it was able to write to the commitlog in the
> first place.  You could try setting the memtable object and memory
> thresholds lower temporarily and see if that leaves enough extra free
> to do the replay.
>
> -Jonathan
>
> On Wed, Aug 19, 2009 at 7:12 PM, Brian Frank
> Cooper<cooperb@yahoo-inc.com> wrote:
>> Hi folks,
>>
>> I'm using 0.4 beta1 and had six servers loaded with 20 GB of data per server. (In
this test, 10 KB per record, and 2 GB heap space allocated to the JVM.) I stopped the servers
(using what I think is the recommended method, the kill command). Upon trying to restart,
some servers threw a UTFDataFormatException, while others threw an OutOfMemoryError exception.
None of them started.
>>
>> Is this a known issue?
>>
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:274)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171)
>>
>> ERROR - Exception encountered during startup.
>> java.io.UTFDataFormatException: malformed input around byte 5497
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:639)
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>>        at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:218)
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:285)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171)
>> Exception encountered during startup.
>> java.io.UTFDataFormatException: malformed input around byte 5497
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:639)
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>>        at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:218)
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:285)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:171)
>>
>> Thanks for the help!
>>
>> Brian
>

Mime
View raw message