incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Cistaro <jcist...@netflix.com>
Subject Re: replace_token code?
Date Mon, 10 Sep 2012 16:45:22 GMT
We have seen various issues from these replaced nodes hanging around.  For clusters where a
lot of nodes have been replaced, we see these replaced nodes having an impact on heap/GC and
a lot of tcp timeouts/retransmits (because the old nodes no longer exist).  As a result, we
have begun cleaning these up using unsafeAssassinateEndpoint via jmx.  We have only started
using recently.  So far no bad side effects.  This also helps because those replaced nodes
can appear as "unreachable nodes" wrt schema and sometimes prevent things like CF truncation.

Using unsafeAssassinateEndpoint will clean these from unreachable nodes and will mark them
as LEFT in gossip info.  There is a ttl for them in gossipinfo and they should go away after
3 days.  Once they are marked LEFT, you should stop seeing those up/same/dead messages.

unsafeAssassinateEndpoint is "unsafe" in that, if you specify IP of a real node in cluster,
that node will be assassinated.  Otherwise, if you specify nodes that have been replaced,
it is supposed to work correctly.

Hope this helps,
jc


From: Yang <teddyyyy123@gmail.com<mailto:teddyyyy123@gmail.com>>
Reply-To: <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Mon, 10 Sep 2012 01:10:56 -0700
To: <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: replace_token code?

it looks that by specifying replace_token, the old owner is not removed from gossip (which
I had thought it would do).
Then it's understandable that the old owner would resurface later and we get some warning
saying that the same token is owned by both.


I ran an example with a 2-node cluster, with RF=2.  host 10.72.201.80 was running for a while
and had some data, then i shut it down, and
booted up 10.190.221.204 with replace_token of the old token owned by the previous host.
the following log sequence shows that the new host does acquire the token, but it does not
at the same time remove 80 forcefully from gossip.
instead, a few seconds later, it believed that .80 became live again.
I don't have much understanding of the Gossip protocol, but roughly know that it's probability-based,
looks we need an "assertive"/"NOW"
membership control message for replace_token.




thanks
yang


 WARN [main] 2012-09-10 08:00:21,855 TokenMetadata.java (line 160) Token 166594924822352415786406422619018814804
changing ownership from /10.72.201.80<http://10.72.201.80> to /10.190.221.204<http://10.190.221.204>
 INFO [main] 2012-09-10 08:00:21,855 StorageService.java (line 753) JOINING: Starting to bootstrap...
 INFO [CompactionExecutor:2] 2012-09-10 08:00:21,875 CompactionTask.java (line 109) Compacting
[SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-1-Data.db'),
SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-3-Data.db'),
SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-4-Data.db'),
SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-2-Data.db')]
 INFO [CompactionExecutor:2] 2012-09-10 08:00:21,979 CompactionTask.java (line 221) Compacted
to [/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-5-Data.db,].  499 to 394
(~78% of original) bytes for 3 keys at 0.003997MB/s.  Time: 94ms.
 INFO [Thread-4] 2012-09-10 08:00:22,070 StreamInSession.java (line 214) Finished streaming
session 1 from /10.72.102.61<http://10.72.102.61>
 INFO [main] 2012-09-10 08:00:22,073 ColumnFamilyStore.java (line 643) Enqueuing flush of
Memtable-LocationInfo@30624226(77/96 serialized/live bytes, 2 ops)
 INFO [FlushWriter:2] 2012-09-10 08:00:22,074 Memtable.java (line 266) Writing Memtable-LocationInfo@30624226(77/96
serialized/live bytes, 2 ops)
 INFO [FlushWriter:2] 2012-09-10 08:00:22,082 Memtable.java (line 307) Completed flushing
/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-6-Data.db (163 bytes) for commitlog
position ReplayPosition(segmentId=6360937126265, position=3854)
 INFO [main] 2012-09-10 08:00:22,087 StorageService.java (line 1098) Node /10.190.221.204<http://10.190.221.204>
state jump to normal
 INFO [main] 2012-09-10 08:00:22,088 StorageService.java (line 666) Bootstrap/Replace/Move
completed! Now serving reads.
 INFO [main] 2012-09-10 08:00:22,089 Mx4jTool.java (line 72) Will not load MX4J, mx4j-tools.jar
is not in the classpath
 INFO [main] 2012-09-10 08:00:22,119 CassandraDaemon.java (line 124) Binding thrift service
to /10.190.221.204:9160<http://10.190.221.204:9160>
 INFO [main] 2012-09-10 08:00:22,122 CassandraDaemon.java (line 133) Using TFastFramedTransport
with a max frame size of 15728640 bytes.
 INFO [main] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 160) Using synchronous/threadpool
thrift server on /10.190.221.204<http://10.190.221.204> : 9160
 INFO [Thread-5] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 212) Listening for thrift
clients...
 INFO [RMI TCP Connection(4)-127.0.0.1] 2012-09-10 08:00:59,014 ColumnFamilyStore.java (line
643) Enqueuing flush of Memtable-Versions@11775580(128/160 serialized/live bytes, 3 ops)
 INFO [FlushWriter:2] 2012-09-10 08:00:59,016 Memtable.java (line 266) Writing Memtable-Versions@11775580(128/160
serialized/live bytes, 3 ops)
 INFO [FlushWriter:2] 2012-09-10 08:00:59,024 Memtable.java (line 307) Completed flushing
/mnt/cassandra/data/system/Versions/system-Versions-hd-1-Data.db (256 bytes) for commitlog
position ReplayPosition(segmentId=6360937126265, position=3854)
 INFO [GossipStage:1] 2012-09-10 08:01:23,036 Gossiper.java (line 850) Node /10.72.201.80<http://10.72.201.80>
is now part of the cluster
 INFO [GossipStage:1] 2012-09-10 08:01:23,037 Gossiper.java (line 816) InetAddress /10.72.201.80<http://10.72.201.80>
is now UP
 INFO [GossipStage:1] 2012-09-10 08:01:23,038 StorageService.java (line 1126) Nodes /10.72.201.80<http://10.72.201.80>
and /10.190.221.204<http://10.190.221.204> have the same token 166594924822352415786406422619018814804.
 Ignoring /10.72.201.80<http://10.72.201.80>
 INFO [GossipTasks:1] 2012-09-10 08:01:32,967 Gossiper.java (line 830) InetAddress /10.72.201.80<http://10.72.201.80>
is now dead.
 INFO [GossipTasks:1] 2012-09-10 08:01:53,976 Gossiper.java (line 644) FatClient /10.72.201.80<http://10.72.201.80>
has been silent for 30000ms, removing from gossip



Mime
View raw message