Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of gciuloaica@gmail.com designates
 209.85.215.175 as permitted sender)
Message-ID: <5167B3CD.8010202@gmail.com>
Date: Fri, 12 Apr 2013 10:12:13 +0300
From: Gabriel Ciuloaica <gciuloaica@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8;
 rv:17.0) Gecko/20130328 Thunderbird/17.0.5
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Adding new nodes to an existing cluster
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

 From yesterday, I'm trying to add a new node to an existing 3 nodes 
Cassandra cluster, running version 1.2.3.  Today I have started clean, 
removed the node that I have tried to add yesterday and try to add 
another node. Verified both cassandra.yaml and 
cassandra-topology.properties are identical on all nodes (I'm using 
NetworkTopologyStrategy and PropertyFile snitch).
I see the nodetool netstats on the new node shows that it is streaming 
from the other nodes, but other than increasing number on Responses 
Pool, there is no change in the report.  Logs are set to Debug and most 
of new lines are "Applying mutations".

Below is the only ERROR that I can find in both cassandra.log and 
system.log.  So, it seems that streaming is working but not 
completing... For the sstable that I get the error, the percentage is 
not increasing in netstats, not for any other sstables. Now it is 
running for few hours and no progress. Any clue ?
ERROR 06:09:03,934 Exception in thread Thread[Thread-25,5,main]
java.lang.AssertionError: incorrect row data size 114969 written to 
/data/woow_backend/offline/woow_backend-offline-tmp-ib-2-Data.db; 
correct is 115006
         at 
org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
         at 
org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
         at 
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
         at 
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
         at 
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
         at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)


nodetool netstats:

Mode: JOINING
Not sending any streams.
Streaming from: /10.11.1.108
    woow_backend: 
/data/woow_backend/user_rooms/woow_backend-user_rooms-ib-1-Data.db 
sections=1 progress=0/566 - 0%
    woow_backend: 
/data/woow_backend/gc_archive/woow_backend-gc_archive-ib-2-Data.db 
sections=1 progress=0/3478 - 0%
    woow_backend: 
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-258-Data.db 
sections=1 progress=0/313 - 0%
    woow_backend: 
/data/woow_backend/roster_groups/woow_backend-roster_groups-ib-23-Data.db sections=1 
progress=0/120 - 0%
    woow_backend: 
/data/woow_backend/roster_versions/woow_backend-roster_versions-ib-34-Data.db 
sections=2 progress=0/195 - 0%
    woow_backend: 
/data/woow_backend/rosters/woow_backend-rosters-ib-27-Data.db 
sections=93 progress=0/881196 - 0%
    woow_backend: 
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-257-Data.db 
sections=104 progress=0/5816889 - 0%
    woow_backend: 
/data/woow_backend/avatars/woow_backend-avatars-ib-68-Data.db 
sections=71 progress=0/5841975 - 0%
    woow_backend: 
/data/woow_backend/rosters/woow_backend-rosters-ib-28-Data.db sections=2 
progress=0/234 - 0%
    woow_backend: 
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-259-Data.db 
sections=3 progress=0/6910 - 0%
    woow_backend: 
/data/woow_backend/offline/woow_backend-offline-ib-129-Data.db 
sections=77 progress=22499/462727 - 4%
    woow_backend: 
/data/woow_backend/rooms/woow_backend-rooms-ib-2-Data.db sections=1 
progress=0/308 - 0%
    woow_backend: 
/data/woow_backend/roster_versions/woow_backend-roster_versions-ib-33-Data.db 
sections=93 progress=0/34246 - 0%
    woow_backend: 
/data/woow_backend/rooms/woow_backend-rooms-ib-1-Data.db sections=1 
progress=0/310 - 0%
Pool Name                    Active   Pending       Completed
Commands                        n/a         0             15
Responses                       n/a         0          44808

Thanks,
Gabi