cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ara Sadoyan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10291) Bootstrap hangs on adding new node
Date Fri, 11 Sep 2015 11:01:46 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ara Sadoyan updated CASSANDRA-10291:
------------------------------------
    Attachment: system.log

Full system log during node bootstrap 

> Bootstrap hangs on adding new node 
> -----------------------------------
>
>                 Key: CASSANDRA-10291
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10291
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Debian 7 64 bit
> HotSpot JDK 1.7.0_79
> Cassandra-2.2.1 via apt-get 
> 1x Intel Quad-Core Xeon E3-1230 / 16GB / 4x1TB SATA / 3x1TB RAID0 data drive 
>            Reporter: Ara Sadoyan
>         Attachments: system.log
>
>
> Adding new node in heavy loaded environment freeze bootstrap. No errors are reported
in log files.  Some of other other nodes throws "String didn't validate" error, but I;m not
sure that this is related. 
> After restarting node it start bootstrap again and hangs after some time . 
> nodetool netstats shows : 
> /data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701   bytes(27%)  received
 from idx:0/192.168.220.16
> /data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671   bytes(38%)  received
 from idx:0/192.168.220.22
> /data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398   bytes(48%)  received
 from idx:0/192.168.220.12
> /data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610  bytes(85%)  received
 from idx:0/192.168.220.18
> /data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085   bytes(48%)  received 
from idx:0/192.168.220.20
> but listing files on local FS shows "No such file or directory"
> This happens only if there is significant amount of data. I have 1.5 TB per node on 13
node cluster, we use STCS compaction strategy and flat network topology . 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message