incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clint Kelly <clint.ke...@gmail.com>
Subject Re: Cassandra process exiting mysteriously
Date Tue, 12 Aug 2014 15:47:09 GMT
Hi Or,

For now I removed the test that was failing like this from our suite
and made a note to revisit it in a couple of weeks.  Unfortunately I
still don't know what the issue is.  I'll post here if I figure out it
(please do the same!).  My working hypothesis now is that we had some
kind of OOM problem.

Best regards,
Clint

On Tue, Aug 12, 2014 at 12:23 AM, Or Sher <or.sher1@gmail.com> wrote:
> Clint, did you find anything?
> I just noticed it happens to us too on only one node in our CI cluster.
> I don't think there is  a special usage before it happens... The last line
> in the log before the shutdown lines in at least an hour before..
> We're using C* 2.0.9.
>
>
> On Thu, Aug 7, 2014 at 12:49 AM, Clint Kelly <clint.kelly@gmail.com> wrote:
>>
>> Hi Rob,
>>
>> Thanks for the clarification; this is really useful.  I'll run some
>> experiments to see if the problem is a JVM OOM on our build machine.
>>
>> Best regards,
>> Clint
>>
>> On Wed, Aug 6, 2014 at 1:14 PM, Robert Coli <rcoli@eventbrite.com> wrote:
>> > On Wed, Aug 6, 2014 at 1:12 PM, Robert Coli <rcoli@eventbrite.com>
>> > wrote:
>> >>
>> >> On Wed, Aug 6, 2014 at 1:11 AM, Duncan Sands <duncan.sands@gmail.com>
>> >> wrote:
>> >>>
>> >>> this doesn't look like an OOM to me.  If the kernel OOM kills
>> >>> Cassandra
>> >>> then Cassandra instantly vaporizes, and there will be nothing in the
>> >>> Cassandra logs (you will find information about the OOM in the system
>> >>> logs
>> >>> though, eg in dmesg).  In the log snippet above you see an orderly
>> >>> shutdown,
>> >>> this is completely different to the instant OOM kill.
>> >>
>> >>
>> >> Not really.
>> >>
>> >> https://issues.apache.org/jira/browse/CASSANDRA-7507
>> >
>> >
>> > To be clear, there's two different OOMs here, I am talking about the JVM
>> > OOM, not system level. As CASSANDRA-7507 indicates, JVM OOM does not
>> > necessarily result in the cassandra process dying, and can in fact
>> > trigger
>> > clean shutdown.
>> >
>> > System level OOM will in fact send the equivalent of KILL, which will
>> > not
>> > trigger the clean shutdown hook in Cassandra.
>> >
>> > =Rob
>
>
>
>
> --
> Or Sher

Mime
View raw message