incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hendry" <dan.hendry.j...@gmail.com>
Subject RE: Severe Reliability Problems - 0.7 RC2
Date Mon, 20 Dec 2010 23:44:01 GMT
Thanks for all the response, I have included requested information below. As
a side note, I THINK I have fixed the problem by using "disk_access_mode:
mmap_index_only". At the very least, none of the nodes has died since
setting the option.

 

> What kernel version are you running?

 

3/4 running 2.6.32-21-server, 1/4 running 2.6.35-23-server

 

> Also, you're virtualized (given %steal), right?

 

No, %steal is 0, these are all dedicated machines

 

> What filesystem are you using?

 

EXT4

 

> Are there any clues in /var/log/messages?

 

Nothing out of the ordinary

 

> How much swap space do you have configured?

 

2 GB and 24 GB of system memory.

 

Dan

 

From: Chris Goffinet [mailto:cg@chrisgoffinet.com] 
Sent: December-20-10 17:32
To: user@cassandra.apache.org
Subject: Re: Severe Reliability Problems - 0.7 RC2

 

What kernel version are you running? I have seen with I/O intense nodes with
2.6.18 to 2.6.24 the kernel has a bug where it locks the JVM and spins to
100%.

On Mon, Dec 20, 2010 at 1:14 PM, Brandon Williams <driftx@gmail.com> wrote:

On Mon, Dec 20, 2010 at 2:13 PM, Dan Hendry <dan.hendry.junk@gmail.com>
wrote:

Yes, I have tried that (although only twice). Same impact as a regular kill:
nothing happens and I get no stacktrace output. It is however on my list of
things to try again the next time a node dies. I am also not able to attach
jstack to the process.

 

Kill -3 will only produce output in foreground mode, jstack will work in
either foreground or background.

 

-Brandon 

 

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.872 / Virus Database: 271.1.1/3327 - Release Date: 12/20/10
02:34:00


Mime
View raw message