cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ara Sadoyan (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-10291) Bootstrap hangs on adding new node
Date Mon, 12 Oct 2015 21:26:05 GMT


Ara Sadoyan commented on CASSANDRA-10291:

Guys the real problem is not logged.
These particular streams were restreamed, but actaul hanged streams were never been logged.
These streams were just hanged silently.
Please do not concentrate on these two streams. 
AFAIK upgradesstables should show corrupted sstables , but when I did that it passe smoothly
on all nodes.
Soon I will upgrade cluster to 2.2.2 and try to reproduce issue.

> Bootstrap hangs on adding new node 
> -----------------------------------
>                 Key: CASSANDRA-10291
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Debian 7 64 bit
> HotSpot JDK 1.7.0_79
> Cassandra-2.2.1 via apt-get 
> 1x Intel Quad-Core Xeon E3-1230 / 16GB / 4x1TB SATA / 3x1TB RAID0 data drive 
>            Reporter: Ara Sadoyan
>         Attachments: logs_netstats.tar.gz, nodetool.txt, system.log
> Adding new node in heavy loaded environment freeze bootstrap. No errors are reported
in log files.  Some of other other nodes throws "String didn't validate" error, but I;m not
sure that this is related. 
> After restarting node it start bootstrap again and hangs after some time . 
> nodetool netstats shows : 
> /data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701   bytes(27%)  received
 from idx:0/
> /data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671   bytes(38%)  received
 from idx:0/
> /data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398   bytes(48%)  received
 from idx:0/
> /data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610  bytes(85%)  received
 from idx:0/
> /data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085   bytes(48%)  received 
from idx:0/
> but listing files on local FS shows "No such file or directory"
> This happens only if there is significant amount of data. I have 1.5 TB per node on 13
node cluster, we use STCS compaction strategy and flat network topology . 

This message was sent by Atlassian JIRA

View raw message