cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Shuler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10291) Bootstrap hangs on adding new node
Date Thu, 10 Sep 2015 18:08:45 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739252#comment-14739252
] 

Michael Shuler commented on CASSANDRA-10291:
--------------------------------------------

(dropped a bunch of extra jira fields)

Looking at the log, it appears to me that you may be suffering from network issues where the
streaming fails. Have you made any configuration adjustments or discussed this with people
on the mailing list or irc? Recovering from a bootstrap failure usually means just bootstrap
again. Since you have a pretty large amount of data on an already-loaded cluster with questionable
network interruption, you might get better help and ideas from other users on the mailing
list or irc - not sure if this is exactly a bug.

> Bootstrap hangs on adding new node 
> -----------------------------------
>
>                 Key: CASSANDRA-10291
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10291
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Debian 7 64 bit
> HotSpot JDK 1.7.0_79
> Cassandra-2.2.1 via apt-get 
> 1x Intel Quad-Core Xeon E3-1230 / 16GB / 4x1TB SATA / 3x1TB RAID0 data drive 
>            Reporter: Ara Sadoyan
>         Attachments: cassandra.log
>
>
> Adding new node in heavy loaded environment freeze bootstrap. No errors are reported
in log files.  Some of other other nodes throws "String didn't validate" error, but I;m not
sure that this is related. 
> After restarting node it start bootstrap again and hangs after some time . 
> nodetool netstats shows : 
> /data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701   bytes(27%)  received
 from idx:0/192.168.220.16
> /data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671   bytes(38%)  received
 from idx:0/192.168.220.22
> /data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398   bytes(48%)  received
 from idx:0/192.168.220.12
> /data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610  bytes(85%)  received
 from idx:0/192.168.220.18
> /data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085   bytes(48%)  received 
from idx:0/192.168.220.20
> but listing files on local FS shows "No such file or directory"
> This happens only if there is significant amount of data. I have 1.5 TB per node on 13
node cluster, we use STCS compaction strategy and flat network topology . 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message