incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Ciuloaica <gciuloa...@gmail.com>
Subject Adding new nodes to an existing cluster
Date Fri, 12 Apr 2013 07:12:13 GMT
Hi,

 From yesterday, I'm trying to add a new node to an existing 3 nodes 
Cassandra cluster, running version 1.2.3.  Today I have started clean, 
removed the node that I have tried to add yesterday and try to add 
another node. Verified both cassandra.yaml and 
cassandra-topology.properties are identical on all nodes (I'm using 
NetworkTopologyStrategy and PropertyFile snitch).
I see the nodetool netstats on the new node shows that it is streaming 
from the other nodes, but other than increasing number on Responses 
Pool, there is no change in the report.  Logs are set to Debug and most 
of new lines are "Applying mutations".

Below is the only ERROR that I can find in both cassandra.log and 
system.log.  So, it seems that streaming is working but not 
completing... For the sstable that I get the error, the percentage is 
not increasing in netstats, not for any other sstables. Now it is 
running for few hours and no progress. Any clue ?
ERROR 06:09:03,934 Exception in thread Thread[Thread-25,5,main]
java.lang.AssertionError: incorrect row data size 114969 written to 
/data/woow_backend/offline/woow_backend-offline-tmp-ib-2-Data.db; 
correct is 115006
         at 
org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
         at 
org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
         at 
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
         at 
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
         at 
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
         at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)



nodetool netstats:

Mode: JOINING
Not sending any streams.
Streaming from: /10.11.1.108
    woow_backend: 
/data/woow_backend/user_rooms/woow_backend-user_rooms-ib-1-Data.db 
sections=1 progress=0/566 - 0%
    woow_backend: 
/data/woow_backend/gc_archive/woow_backend-gc_archive-ib-2-Data.db 
sections=1 progress=0/3478 - 0%
    woow_backend: 
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-258-Data.db 
sections=1 progress=0/313 - 0%
    woow_backend: 
/data/woow_backend/roster_groups/woow_backend-roster_groups-ib-23-Data.db sections=1 
progress=0/120 - 0%
    woow_backend: 
/data/woow_backend/roster_versions/woow_backend-roster_versions-ib-34-Data.db 
sections=2 progress=0/195 - 0%
    woow_backend: 
/data/woow_backend/rosters/woow_backend-rosters-ib-27-Data.db 
sections=93 progress=0/881196 - 0%
    woow_backend: 
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-257-Data.db 
sections=104 progress=0/5816889 - 0%
    woow_backend: 
/data/woow_backend/avatars/woow_backend-avatars-ib-68-Data.db 
sections=71 progress=0/5841975 - 0%
    woow_backend: 
/data/woow_backend/rosters/woow_backend-rosters-ib-28-Data.db sections=2 
progress=0/234 - 0%
    woow_backend: 
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-259-Data.db 
sections=3 progress=0/6910 - 0%
    woow_backend: 
/data/woow_backend/offline/woow_backend-offline-ib-129-Data.db 
sections=77 progress=22499/462727 - 4%
    woow_backend: 
/data/woow_backend/rooms/woow_backend-rooms-ib-2-Data.db sections=1 
progress=0/308 - 0%
    woow_backend: 
/data/woow_backend/roster_versions/woow_backend-roster_versions-ib-33-Data.db 
sections=93 progress=0/34246 - 0%
    woow_backend: 
/data/woow_backend/rooms/woow_backend-rooms-ib-1-Data.db sections=1 
progress=0/310 - 0%
Pool Name                    Active   Pending       Completed
Commands                        n/a         0             15
Responses                       n/a         0          44808

Thanks,
Gabi

Mime
View raw message