incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject Re: replace_token code?
Date Mon, 10 Sep 2012 17:59:28 GMT
Thanks Jim, looks I'll have to read into the code to understand what is
happening under the hood

yang

On Mon, Sep 10, 2012 at 9:45 AM, Jim Cistaro <jcistaro@netflix.com> wrote:

>  We have seen various issues from these replaced nodes hanging around.
>  For clusters where a lot of nodes have been replaced, we see these
> replaced nodes having an impact on heap/GC and a lot of tcp
> timeouts/retransmits (because the old nodes no longer exist).  As a result,
> we have begun cleaning these up using unsafeAssassinateEndpoint via jmx.
>  We have only started using recently.  So far no bad side effects.  This
> also helps because those replaced nodes can appear as "unreachable nodes"
> wrt schema and sometimes prevent things like CF truncation.
>
>  Using unsafeAssassinateEndpoint will clean these from unreachable nodes
> and will mark them as LEFT in gossip info.  There is a ttl for them in
> gossipinfo and they should go away after 3 days.  Once they are marked
> LEFT, you should stop seeing those up/same/dead messages.
>
>  unsafeAssassinateEndpoint is "unsafe" in that, if you specify IP of a
> real node in cluster, that node will be assassinated.  Otherwise, if you
> specify nodes that have been replaced, it is supposed to work correctly.
>
>  Hope this helps,
> jc
>
>
>   From: Yang <teddyyyy123@gmail.com>
> Reply-To: <user@cassandra.apache.org>
> Date: Mon, 10 Sep 2012 01:10:56 -0700
> To: <user@cassandra.apache.org>
> Subject: replace_token code?
>
>  it looks that by specifying replace_token, the old owner is not removed
> from gossip (which I had thought it would do).
> Then it's understandable that the old owner would resurface later and we
> get some warning saying that the same token is owned by both.
>
>
>  I ran an example with a 2-node cluster, with RF=2.  host 10.72.201.80
> was running for a while and had some data, then i shut it down, and
> booted up 10.190.221.204 with replace_token of the old token owned by the
> previous host.
> the following log sequence shows that the new host does acquire the token,
> but it does not at the same time remove 80 forcefully from gossip.
> instead, a few seconds later, it believed that .80 became live again.
> I don't have much understanding of the Gossip protocol, but roughly know
> that it's probability-based, looks we need an "assertive"/"NOW"
> membership control message for replace_token.
>
>
>
>
>  thanks
> yang
>
>
>   WARN [main] 2012-09-10 08:00:21,855 TokenMetadata.java (line 160) Token
> 166594924822352415786406422619018814804 changing ownership from /
> 10.72.201.80 to /10.190.221.204
>  INFO [main] 2012-09-10 08:00:21,855 StorageService.java (line 753)
> JOINING: Starting to bootstrap...
>  INFO [CompactionExecutor:2] 2012-09-10 08:00:21,875 CompactionTask.java
> (line 109) Compacting
> [SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-1-Data.db'),
> SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-3-Data.db'),
> SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-4-Data.db'),
> SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-2-Data.db')]
>  INFO [CompactionExecutor:2] 2012-09-10 08:00:21,979 CompactionTask.java
> (line 221) Compacted to
> [/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-5-Data.db,].
>  499 to 394 (~78% of original) bytes for 3 keys at 0.003997MB/s.  Time:
> 94ms.
>  INFO [Thread-4] 2012-09-10 08:00:22,070 StreamInSession.java (line 214)
> Finished streaming session 1 from /10.72.102.61
>  INFO [main] 2012-09-10 08:00:22,073 ColumnFamilyStore.java (line 643)
> Enqueuing flush of Memtable-LocationInfo@30624226(77/96 serialized/live
> bytes, 2 ops)
>  INFO [FlushWriter:2] 2012-09-10 08:00:22,074 Memtable.java (line 266)
> Writing Memtable-LocationInfo@30624226(77/96 serialized/live bytes, 2 ops)
>  INFO [FlushWriter:2] 2012-09-10 08:00:22,082 Memtable.java (line 307)
> Completed flushing
> /mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-6-Data.db
> (163 bytes) for commitlog position ReplayPosition(segmentId=6360937126265,
> position=3854)
>  INFO [main] 2012-09-10 08:00:22,087 StorageService.java (line 1098) Node /
> 10.190.221.204 state jump to normal
>  INFO [main] 2012-09-10 08:00:22,088 StorageService.java (line 666)
> Bootstrap/Replace/Move completed! Now serving reads.
>  INFO [main] 2012-09-10 08:00:22,089 Mx4jTool.java (line 72) Will not load
> MX4J, mx4j-tools.jar is not in the classpath
>  INFO [main] 2012-09-10 08:00:22,119 CassandraDaemon.java (line 124)
> Binding thrift service to /10.190.221.204:9160
>  INFO [main] 2012-09-10 08:00:22,122 CassandraDaemon.java (line 133) Using
> TFastFramedTransport with a max frame size of 15728640 bytes.
>  INFO [main] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 160) Using
> synchronous/threadpool thrift server on /10.190.221.204 : 9160
>  INFO [Thread-5] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 212)
> Listening for thrift clients...
>  INFO [RMI TCP Connection(4)-127.0.0.1] 2012-09-10 08:00:59,014
> ColumnFamilyStore.java (line 643) Enqueuing flush of
> Memtable-Versions@11775580(128/160 serialized/live bytes, 3 ops)
>  INFO [FlushWriter:2] 2012-09-10 08:00:59,016 Memtable.java (line 266)
> Writing Memtable-Versions@11775580(128/160 serialized/live bytes, 3 ops)
>  INFO [FlushWriter:2] 2012-09-10 08:00:59,024 Memtable.java (line 307)
> Completed flushing
> /mnt/cassandra/data/system/Versions/system-Versions-hd-1-Data.db (256
> bytes) for commitlog position ReplayPosition(segmentId=6360937126265,
> position=3854)
>   INFO [GossipStage:1] 2012-09-10 08:01:23,036 Gossiper.java (line 850)
> Node /10.72.201.80 is now part of the cluster
>  INFO [GossipStage:1] 2012-09-10 08:01:23,037 Gossiper.java (line 816)
> InetAddress /10.72.201.80 is now UP
>  INFO [GossipStage:1] 2012-09-10 08:01:23,038 StorageService.java (line
> 1126) Nodes /10.72.201.80 and /10.190.221.204 have the same token
> 166594924822352415786406422619018814804.  Ignoring /10.72.201.80
>  INFO [GossipTasks:1] 2012-09-10 08:01:32,967 Gossiper.java (line 830)
> InetAddress /10.72.201.80 is now dead.
>  INFO [GossipTasks:1] 2012-09-10 08:01:53,976 Gossiper.java (line 644)
> FatClient /10.72.201.80 has been silent for 30000ms, removing from gossip
>
>
>

Mime
View raw message