cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ara Sadoyan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10291) Bootstrap hangs on adding new node
Date Thu, 10 Sep 2015 22:17:45 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739722#comment-14739722
] 

Ara Sadoyan commented on CASSANDRA-10291:
-----------------------------------------

This cluster is running for almost a year and there were no problems in 2.1 .
Problem appeared after I have upgraded to 2.2. Today I did bootstrap again and failed same
way, but there was nothing in logs.
Regarding network. These servers are directly connected to gbit switch .
I have lowered down stream speed so commulative traffic to bootstrapping node was about 700
mbit, just to ensure that I do not have bandwidth problem. So i guess this is a bug. 

> Bootstrap hangs on adding new node 
> -----------------------------------
>
>                 Key: CASSANDRA-10291
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10291
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Debian 7 64 bit
> HotSpot JDK 1.7.0_79
> Cassandra-2.2.1 via apt-get 
> 1x Intel Quad-Core Xeon E3-1230 / 16GB / 4x1TB SATA / 3x1TB RAID0 data drive 
>            Reporter: Ara Sadoyan
>         Attachments: cassandra.log
>
>
> Adding new node in heavy loaded environment freeze bootstrap. No errors are reported
in log files.  Some of other other nodes throws "String didn't validate" error, but I;m not
sure that this is related. 
> After restarting node it start bootstrap again and hangs after some time . 
> nodetool netstats shows : 
> /data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701   bytes(27%)  received
 from idx:0/192.168.220.16
> /data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671   bytes(38%)  received
 from idx:0/192.168.220.22
> /data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398   bytes(48%)  received
 from idx:0/192.168.220.12
> /data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610  bytes(85%)  received
 from idx:0/192.168.220.18
> /data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085   bytes(48%)  received 
from idx:0/192.168.220.20
> but listing files on local FS shows "No such file or directory"
> This happens only if there is significant amount of data. I have 1.5 TB per node on 13
node cluster, we use STCS compaction strategy and flat network topology . 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message