cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From arun sirimalla <arunsi...@gmail.com>
Subject Re: Unexpected behavior after adding successffully a new node
Date Tue, 12 May 2015 16:59:04 GMT
Analia,

Try running repair on node 3.

On Tue, May 12, 2015 at 7:39 AM, Analia Lorenzatto <
analialorenzatto@gmail.com> wrote:

> Hello guys,
>
>
> I have a cluster 2.1.0-2 comprised of 3 nodes.  The replication factor=2.
> We successfully added the third node last week.  After that, We ran clean
> ups on one node at that time.  Then we ran repairs on all the nodes, and
> finally compactions on all the CFs.
>
> Last night, I noticed the cluster started behaving in a weird way.  The
> last node (successfully added last week) were being reported up and down
> all the time.  I could see a lot of messages like this on logs:
>
> WARN  [SharedPool-Worker-33] 2015-05-11 21:31:45,125
> AbstractTracingAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[SharedPool-Worker-33,5,main]: {}
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /mnt/cassandra/data/matchings-85b4929048e211e4a949a3ed319cbedc/matchings-ka-3914-Data.db
> (No such file or directory)
>
> At the same time the consumption of heap used was on the top, up to the
> point the rest of the cluster saw this node as down.  After that, I just
> restarted the cassandra service with no problems on that node.
>
> Now, I can see the three nodes on the cluster Up and Normal, but this last
> node (which was rebooted) does not have data.  But it has all the structure
> of cassandra data.
>
> I can query against the new node and I get the same result as if do the
> query against the others nodes.  But, on this new node I do not have any
> SStables:
>
>
>
> root@prd-rtbkit-cassandra-03:/var/log/cassandra# nodetool status
> Datacenter: us-east
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address     Load       Tokens  Owns (effective)  Host ID
>                 Rack
> UN  10.0.0.a  390.28 GB  256     66.7%
> eed9e9f5-f279-4b2f-b521-c056cbf65b52  1c
> UN  10.0.0.b  382.36 GB  256     68.3%
> 19492c26-4458-4a0b-af04-72e0aab6598e  1c
> UN  10.0.0.c  40.61 MB   256     64.9%
> b8da952c-24b3-444a-a34e-7a1804eee6e6  1c
>
> What do you recommend to do? Leave this as if, remove it and try to join
> this or a new one?
> Thanks in advance!!
>
> --
> Saludos / Regards.
>
> Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life".  By Captain Jean-Luc Picard.
>



-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick

Champion of Big Data (Cloudera)
http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html

2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Mime
View raw message