Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=HIFWvgzALwcx4bJ6tIHk0C0Ocuby2BvCwFXyZHFTU0dOnguE6VaFQtbCpUtnULkJ
From: Brian Frank Cooper <cooperb@yahoo-inc.com>
To: "cassandra-user@incubator.apache.org"
	<cassandra-user@incubator.apache.org>
Date: Tue, 25 Aug 2009 23:03:46 -0700
Subject: RE: Server cannot startup after shutdown
Thread-Topic: Server cannot startup after shutdown
Thread-Index: Acok9KRtIliV3wwxQfCIEtwgNksUbABHUYZ+
Message-ID: 
 <C2D6929236FAC846B7A4FE1EC39910C64F1F66FC20@SP1-EX07VS01.ds.corp.yahoo.com>
References: 
 <C2D6929236FAC846B7A4FE1EC39910C64F1F66FBF3@SP1-EX07VS01.ds.corp.yahoo.com>
	 <e06563880908191946t1d3d96dej1f27e0eadd7a5475@mail.gmail.com>,<e06563880908241251v492733e3hdf12d66f63a3ff0b@mail.gmail.com>
In-Reply-To: <e06563880908241251v492733e3hdf12d66f63a3ff0b@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi, Jonathan,

I have been trying to shutdown and restart Cassandra again this morning. I =
still get the malformed entry bug (which you say below your patch fixes.) I=
 also get:

ERROR - Exception encountered during startup.
java.lang.NegativeArraySizeException
        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:274)
        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManag=
er.java:63)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaem=
on.java:96)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemo=
n.java:171)
Exception encountered during startup.
java.lang.NegativeArraySizeException
        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:274)
        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManag=
er.java:63)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaem=
on.java:96)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemo=
n.java:171)

No out of memory error this time, though.

I'm also curious about your comment "I introduced a regression where it cou=
ldn't handle the last entry in the commitlog being incomplete." Does the la=
st entry in the commit log being incomplete mean that the last update or se=
t of updates are not fully committed to the log? And therefore they are los=
t? I thought since I had set "<CommitLogSync>true</CommitLogSync>" that all=
 updates would be fully flushed before returning to the caller.

(BTW thanks for all the help with setting up Cassandra, it really made it e=
asier to run experiments...)

brian
________________________________________
From: Jonathan Ellis [jbellis@gmail.com]
Sent: Monday, August 24, 2009 12:51 PM
To: cassandra-user@incubator.apache.org
Subject: Re: Server cannot startup after shutdown

Oops, my bad -- that patch has been sitting unreviewed in
CASSANDRA-370.  I thought it was in trunk by now.  I'll try to get
someone to review that today.

-Jonathan

On Wed, Aug 19, 2009 at 9:46 PM, Jonathan Ellis<jbellis@gmail.com> wrote:
> The malformed input bug was fixed after beta1 and should be in a
> nightly build by now.  (I introduced a regression where it couldn't
> handle the last entry in the commitlog being incomplete.  So upgrading
> should be able to restart on the existing commitlogs.)
>
> The OOM puzzles me a little; I'm not sure how it could be unable to
> replay a mutation that it was able to write to the commitlog in the
> first place.  You could try setting the memtable object and memory
> thresholds lower temporarily and see if that leaves enough extra free
> to do the replay.
>
> -Jonathan
>
> On Wed, Aug 19, 2009 at 7:12 PM, Brian Frank
> Cooper<cooperb@yahoo-inc.com> wrote:
>> Hi folks,
>>
>> I'm using 0.4 beta1 and had six servers loaded with 20 GB of data per se=
rver. (In this test, 10 KB per record, and 2 GB heap space allocated to the=
 JVM.) I stopped the servers (using what I think is the recommended method,=
 the kill command). Upon trying to restart, some servers threw a UTFDataFor=
matException, while others threw an OutOfMemoryError exception. None of the=
m started.
>>
>> Is this a known issue?
>>
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:274)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryMan=
ager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDa=
emon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDae=
mon.java:171)
>>
>> ERROR - Exception encountered during startup.
>> java.io.UTFDataFormatException: malformed input around byte 5497
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:639)
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>>        at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:218=
)
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:285)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryMan=
ager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDa=
emon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDae=
mon.java:171)
>> Exception encountered during startup.
>> java.io.UTFDataFormatException: malformed input around byte 5497
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:639)
>>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>>        at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:218=
)
>>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:285)
>>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryMan=
ager.java:63)
>>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDa=
emon.java:96)
>>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDae=
mon.java:171)
>>
>> Thanks for the help!
>>
>> Brian
>