incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Eriksson <krum...@gmail.com>
Subject Re: Adding new nodes to an existing cluster
Date Fri, 12 Apr 2013 08:11:27 GMT
looks very similar to this:
https://issues.apache.org/jira/browse/CASSANDRA-5418

/Marcus


On Fri, Apr 12, 2013 at 9:12 AM, Gabriel Ciuloaica <gciuloaica@gmail.com>wrote:

> Hi,
>
> From yesterday, I'm trying to add a new node to an existing 3 nodes
> Cassandra cluster, running version 1.2.3.  Today I have started clean,
> removed the node that I have tried to add yesterday and try to add another
> node. Verified both cassandra.yaml and cassandra-topology.properties are
> identical on all nodes (I'm using NetworkTopologyStrategy and PropertyFile
> snitch).
> I see the nodetool netstats on the new node shows that it is streaming
> from the other nodes, but other than increasing number on Responses Pool,
> there is no change in the report.  Logs are set to Debug and most of new
> lines are "Applying mutations".
>
> Below is the only ERROR that I can find in both cassandra.log and
> system.log.  So, it seems that streaming is working but not completing...
> For the sstable that I get the error, the percentage is not increasing in
> netstats, not for any other sstables. Now it is running for few hours and
> no progress. Any clue ?
> ERROR 06:09:03,934 Exception in thread Thread[Thread-25,5,main]
> java.lang.AssertionError: incorrect row data size 114969 written to
> /data/woow_backend/offline/**woow_backend-offline-tmp-ib-2-**Data.db;
> correct is 115006
>         at org.apache.cassandra.io.**sstable.SSTableWriter.**
> appendFromStream(**SSTableWriter.java:285)
>         at org.apache.cassandra.**streaming.**
> IncomingStreamReader.streamIn(**IncomingStreamReader.java:179)
>         at org.apache.cassandra.**streaming.**IncomingStreamReader.read(**
> IncomingStreamReader.java:122)
>         at org.apache.cassandra.net.**IncomingTcpConnection.stream(**
> IncomingTcpConnection.java:**238)
>         at org.apache.cassandra.net.**IncomingTcpConnection.**
> handleStream(**IncomingTcpConnection.java:**178)
>         at org.apache.cassandra.net.**IncomingTcpConnection.run(**
> IncomingTcpConnection.java:78)
>
>
>
> nodetool netstats:
>
> Mode: JOINING
> Not sending any streams.
> Streaming from: /10.11.1.108
>    woow_backend: /data/woow_backend/user_rooms/**
> woow_backend-user_rooms-ib-1-**Data.db sections=1 progress=0/566 - 0%
>    woow_backend: /data/woow_backend/gc_archive/**
> woow_backend-gc_archive-ib-2-**Data.db sections=1 progress=0/3478 - 0%
>    woow_backend: /data/woow_backend/msg_**archive/woow_backend-msg_**archive-ib-258-Data.db
> sections=1 progress=0/313 - 0%
>    woow_backend: /data/woow_backend/roster_**groups/woow_backend-roster_**groups-ib-23-Data.db
> sections=1 progress=0/120 - 0%
>    woow_backend: /data/woow_backend/roster_**versions/woow_backend-roster_
> **versions-ib-34-Data.db sections=2 progress=0/195 - 0%
>    woow_backend: /data/woow_backend/rosters/**woow_backend-rosters-ib-27-*
> *Data.db sections=93 progress=0/881196 - 0%
>    woow_backend: /data/woow_backend/msg_**archive/woow_backend-msg_**archive-ib-257-Data.db
> sections=104 progress=0/5816889 - 0%
>    woow_backend: /data/woow_backend/avatars/**woow_backend-avatars-ib-68-*
> *Data.db sections=71 progress=0/5841975 - 0%
>    woow_backend: /data/woow_backend/rosters/**woow_backend-rosters-ib-28-*
> *Data.db sections=2 progress=0/234 - 0%
>    woow_backend: /data/woow_backend/msg_**archive/woow_backend-msg_**archive-ib-259-Data.db
> sections=3 progress=0/6910 - 0%
>    woow_backend: /data/woow_backend/offline/**woow_backend-offline-ib-129-
> **Data.db sections=77 progress=22499/462727 - 4%
>    woow_backend: /data/woow_backend/rooms/woow_**backend-rooms-ib-2-Data.db
> sections=1 progress=0/308 - 0%
>    woow_backend: /data/woow_backend/roster_**versions/woow_backend-roster_
> **versions-ib-33-Data.db sections=93 progress=0/34246 - 0%
>    woow_backend: /data/woow_backend/rooms/woow_**backend-rooms-ib-1-Data.db
> sections=1 progress=0/310 - 0%
> Pool Name                    Active   Pending       Completed
> Commands                        n/a         0             15
> Responses                       n/a         0          44808
>
> Thanks,
> Gabi
>

Mime
View raw message