cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Lerman (JIRA)" <>
Subject [jira] Created: (CASSANDRA-1224) Cassandra NPE on insert after one node goes down.
Date Thu, 24 Jun 2010 15:59:49 GMT
Cassandra NPE on insert after one node goes down.

                 Key: CASSANDRA-1224
             Project: Cassandra
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.6.1
         Environment: Gentoo  Linux
            Reporter: Jeff Lerman
            Priority: Minor

Hi all,

I posted this in a different thread and was instructed to create a new bug.  As far as I can
tell it is not too major of an issue as it may have been cause by us prematurely taking down
a node.

I just had this happen in Cassandra 0.6.1. We're only running two nodes as of now and our
second one was barely accepting any requests and only being replicated to for the most part.
The load went up to 9 consistently so we investigated and noticed its "Load" on nodetool was
2x as large as our other instance. I went and cleared out the data and commitlogs, set autobootstrap
to true and put it back in.

This is where our case gets funky...we noticed the other instance's load going up a lot and
saw that the one I just readded was not doing much. After awhile of contemplating, I took
down the second one again. Minutes later I found an open case about the anticompaction happening
before full bootstrapping occurs. I found the data/stream dir on the working instance and
saw that it was complete...but I had already taken down the second one! So I deleted the stream
dir to save space and figured I'd start the process again tomorrow.

A few hours later I am getting these Internal errors on writes:

ERROR [pool-1-thread-287117] 2010-06-23 19:16:51,754 (line 1492) Internal error
processing insert

That was the entire trace.   We tried to kill -3 Cassandra...waited hours and it never killed.
 Did a kill -6 but got no usable dump.   Perhaps it is possible for someone to recreate this

I also noticed that the virtual memory Cassandra was taking up tacked on the extra 10+GB for
the stream file.  It never released this either which is bad.



This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message