Thanks. Will try upgrading jvm.

Is 1.0 or 0.8.6 is backwards compatible with 0.8.1? 

Thanks,
Jahangir.

On Wed, Nov 2, 2011 at 9:39 AM, Sylvain Lebresne <sylvain@datastax.com> wrote:
On Wed, Nov 2, 2011 at 2:03 PM, Jahangir Mohammed
<md.jahangir27@gmail.com> wrote:
> Hello All,
> JVM is crashing on the cassandra nodes. Re-start doesn't help for long.
> Ring information:
> $ bin/nodetool -h A ring;
> Address         DC          Rack        Status State   Load            Owns
>    Token
>
>    127605887595351923798765477786913079297
> A   DC1         RAC1        Up     Normal  83.65 GB        25.00%  0
> B    DC2         RAC1        Down   Normal  170.09 GB       0.00%   1
> C   DC1         RAC1        Up     Normal  94.6 GB         25.00%
>  42535295865117307932921825928971026432
> D    DC2         RAC1        Up     Normal  87 GB           0.00%
> 42535295865117307932921825928971026433
> E   DC1         RAC1        Up     Normal  98.05 GB        25.00%
>  85070591730234615865843651857942052864
> F    DC2         RAC1        Up     Normal  95.55 GB        0.00%
> 85070591730234615865843651857942052865
> G   DC1         RAC1        Up     Normal  111.22 GB       25.00%
>  127605887595351923798765477786913079296
> H    DC2         RAC1        Up     Normal  42.05 GB        0.00%
> 127605887595351923798765477786913079297
> Details:
> 10GB Heap space.
> Memory on each node = 98 GB
> Disk space on each node = 400 GB
> JVM Crashes with segmentation faults. Have to do frequent re-starts of the
> nodes.
> Space on B is 170 GB and is getting CPU bound on re-start. but didn't get
> added to ring for almost 7 hours now.
> Java version:
>  java -version
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> JVM Crash Error log:
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00002abc7ec41fbc, pid=14232, tid=1104185664
> #
> # JRE version: 6.0_24-b07
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # V  [libjvm.so+0x30ffbc]
> #
> # If you would like to submit a bug report, please visit:
> #   http://java.sun.com/webapps/bugreport/crash.jsp
> #
> ---------------  T H R E A D  ---------------
> Current thread (0x000000004d374000):  GCTaskThread [stack:
> 0x0000000000000000,0x0000000000000000] [id=14243]
> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
> si_addr=0x0000000000000010
> Registers:
>
> Any ideas/suggestions? Any preferred JVM version? There is nothing in
> cassandra logs to identify what's going on. And the update is pretty old,


That's a JVM segfault so has nothing to do with Cassandra. And hotspot
update 19 is pretty old so that's not so surprising. You usually want to go
with as recent as you can to avoid those but if memory servers there is
known problem with JVM < 21 that Cassandra hits. But really I would go
with update 27 or 29 without much hesitation.

Btw, Cassandra 0.8.1 is pretty old too.

> Thanks,
> Jahangir.