Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of teddyyyy123@gmail.com
 designates 209.85.216.172 as permitted sender)
MIME-Version: 1.0
From: Yang <teddyyyy123@gmail.com>
Date: Mon, 10 Sep 2012 01:10:56 -0700
Message-ID: 
 <CAAnh3__m55He0sgo5bc-0KAvOZ7sdOR-1vG-O5tbUh4194Nm+Q@mail.gmail.com>
Subject: replace_token code?
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=20cf303b430f7906fb04c95480c9

--20cf303b430f7906fb04c95480c9
Content-Type: text/plain; charset=ISO-8859-1

it looks that by specifying replace_token, the old owner is not removed
from gossip (which I had thought it would do).
Then it's understandable that the old owner would resurface later and we
get some warning saying that the same token is owned by both.


I ran an example with a 2-node cluster, with RF=2.  host 10.72.201.80 was
running for a while and had some data, then i shut it down, and
booted up 10.190.221.204 with replace_token of the old token owned by the
previous host.
the following log sequence shows that the new host does acquire the token,
but it does not at the same time remove 80 forcefully from gossip.
instead, a few seconds later, it believed that .80 became live again.
I don't have much understanding of the Gossip protocol, but roughly know
that it's probability-based, looks we need an "assertive"/"NOW"
membership control message for replace_token.


thanks
yang


 WARN [main] 2012-09-10 08:00:21,855 TokenMetadata.java (line 160) Token
166594924822352415786406422619018814804 changing ownership from /
10.72.201.80 to /10.190.221.204
 INFO [main] 2012-09-10 08:00:21,855 StorageService.java (line 753)
JOINING: Starting to bootstrap...
 INFO [CompactionExecutor:2] 2012-09-10 08:00:21,875 CompactionTask.java
(line 109) Compacting
[SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-1-Data.db'),
SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-3-Data.db'),
SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-4-Data.db'),
SSTableReader(path='/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-2-Data.db')]
 INFO [CompactionExecutor:2] 2012-09-10 08:00:21,979 CompactionTask.java
(line 221) Compacted to
[/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-5-Data.db,].
 499 to 394 (~78% of original) bytes for 3 keys at 0.003997MB/s.  Time:
94ms.
 INFO [Thread-4] 2012-09-10 08:00:22,070 StreamInSession.java (line 214)
Finished streaming session 1 from /10.72.102.61
 INFO [main] 2012-09-10 08:00:22,073 ColumnFamilyStore.java (line 643)
Enqueuing flush of Memtable-LocationInfo@30624226(77/96 serialized/live
bytes, 2 ops)
 INFO [FlushWriter:2] 2012-09-10 08:00:22,074 Memtable.java (line 266)
Writing Memtable-LocationInfo@30624226(77/96 serialized/live bytes, 2 ops)
 INFO [FlushWriter:2] 2012-09-10 08:00:22,082 Memtable.java (line 307)
Completed flushing
/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-6-Data.db
(163 bytes) for commitlog position ReplayPosition(segmentId=6360937126265,
position=3854)
 INFO [main] 2012-09-10 08:00:22,087 StorageService.java (line 1098) Node /
10.190.221.204 state jump to normal
 INFO [main] 2012-09-10 08:00:22,088 StorageService.java (line 666)
Bootstrap/Replace/Move completed! Now serving reads.
 INFO [main] 2012-09-10 08:00:22,089 Mx4jTool.java (line 72) Will not load
MX4J, mx4j-tools.jar is not in the classpath
 INFO [main] 2012-09-10 08:00:22,119 CassandraDaemon.java (line 124)
Binding thrift service to /10.190.221.204:9160
 INFO [main] 2012-09-10 08:00:22,122 CassandraDaemon.java (line 133) Using
TFastFramedTransport with a max frame size of 15728640 bytes.
 INFO [main] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 160) Using
synchronous/threadpool thrift server on /10.190.221.204 : 9160
 INFO [Thread-5] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 212)
Listening for thrift clients...
 INFO [RMI TCP Connection(4)-127.0.0.1] 2012-09-10 08:00:59,014
ColumnFamilyStore.java (line 643) Enqueuing flush of
Memtable-Versions@11775580(128/160 serialized/live bytes, 3 ops)
 INFO [FlushWriter:2] 2012-09-10 08:00:59,016 Memtable.java (line 266)
Writing Memtable-Versions@11775580(128/160 serialized/live bytes, 3 ops)
 INFO [FlushWriter:2] 2012-09-10 08:00:59,024 Memtable.java (line 307)
Completed flushing
/mnt/cassandra/data/system/Versions/system-Versions-hd-1-Data.db (256
bytes) for commitlog position ReplayPosition(segmentId=6360937126265,
position=3854)
 INFO [GossipStage:1] 2012-09-10 08:01:23,036 Gossiper.java (line 850) Node
/10.72.201.80 is now part of the cluster
 INFO [GossipStage:1] 2012-09-10 08:01:23,037 Gossiper.java (line 816)
InetAddress /10.72.201.80 is now UP
 INFO [GossipStage:1] 2012-09-10 08:01:23,038 StorageService.java (line
1126) Nodes /10.72.201.80 and /10.190.221.204 have the same token
166594924822352415786406422619018814804.  Ignoring /10.72.201.80
 INFO [GossipTasks:1] 2012-09-10 08:01:32,967 Gossiper.java (line 830)
InetAddress /10.72.201.80 is now dead.
 INFO [GossipTasks:1] 2012-09-10 08:01:53,976 Gossiper.java (line 644)
FatClient /10.72.201.80 has been silent for 30000ms, removing from gossip

--20cf303b430f7906fb04c95480c9
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div>it looks that by specifying replace_token, the old owner is not remove=
d from gossip (which I had thought it would do).</div><div>Then it&#39;s un=
derstandable that the old owner would resurface later and we get some warni=
ng saying that the same token is owned by both.</div>

<div><br></div><div><br></div><div>I ran an example with a 2-node cluster, =
with RF=3D2. =A0host=A010.72.201.80 was running for a while and had some da=
ta, then i shut it down, and=A0</div><div>booted up=A010.190.221.204=A0with=
 replace_token of the old token owned by the previous host.</div>

<div>the following log sequence shows that the new host does acquire the to=
ken, but it does not at the same time remove 80 forcefully from gossip.</di=
v><div>instead, a few seconds later, it believed that .80 became live again=
.</div>

<div>I don&#39;t have much understanding of the Gossip protocol, but roughl=
y know that it&#39;s probability-based, looks we need an &quot;assertive&qu=
ot;/&quot;NOW&quot;=A0</div><div>membership control message for replace_tok=
en.</div>

<div><br></div><div><br></div><div><br></div><div><br></div><div>thanks</di=
v><div>yang</div><div><br></div><div><br></div><div>=A0WARN [main] 2012-09-=
10 08:00:21,855 TokenMetadata.java (line 160) Token 16659492482235241578640=
6422619018814804 changing ownership from /<a href=3D"http://10.72.201.80">1=
0.72.201.80</a> to /<a href=3D"http://10.190.221.204">10.190.221.204</a></d=
iv>

<div>=A0INFO [main] 2012-09-10 08:00:21,855 StorageService.java (line 753) =
JOINING: Starting to bootstrap...</div><div>=A0INFO [CompactionExecutor:2] =
2012-09-10 08:00:21,875 CompactionTask.java (line 109) Compacting [SSTableR=
eader(path=3D&#39;/mnt/cassandra/data/system/LocationInfo/system-LocationIn=
fo-hd-1-Data.db&#39;), SSTableReader(path=3D&#39;/mnt/cassandra/data/system=
/LocationInfo/system-LocationInfo-hd-3-Data.db&#39;), SSTableReader(path=3D=
&#39;/mnt/cassandra/data/system/LocationInfo/system-LocationInfo-hd-4-Data.=
db&#39;), SSTableReader(path=3D&#39;/mnt/cassandra/data/system/LocationInfo=
/system-LocationInfo-hd-2-Data.db&#39;)]</div>

<div>=A0INFO [CompactionExecutor:2] 2012-09-10 08:00:21,979 CompactionTask.=
java (line 221) Compacted to [/mnt/cassandra/data/system/LocationInfo/syste=
m-LocationInfo-hd-5-Data.db,]. =A0499 to 394 (~78% of original) bytes for 3=
 keys at 0.003997MB/s. =A0Time: 94ms.</div>

<div>=A0INFO [Thread-4] 2012-09-10 08:00:22,070 StreamInSession.java (line =
214) Finished streaming session 1 from /<a href=3D"http://10.72.102.61">10.=
72.102.61</a></div><div>=A0INFO [main] 2012-09-10 08:00:22,073 ColumnFamily=
Store.java (line 643) Enqueuing flush of Memtable-LocationInfo@30624226(77/=
96 serialized/live bytes, 2 ops)</div>

<div>=A0INFO [FlushWriter:2] 2012-09-10 08:00:22,074 Memtable.java (line 26=
6) Writing Memtable-LocationInfo@30624226(77/96 serialized/live bytes, 2 op=
s)</div><div>=A0INFO [FlushWriter:2] 2012-09-10 08:00:22,082 Memtable.java =
(line 307) Completed flushing /mnt/cassandra/data/system/LocationInfo/syste=
m-LocationInfo-hd-6-Data.db (163 bytes) for commitlog position ReplayPositi=
on(segmentId=3D6360937126265, position=3D3854)</div>

<div>=A0INFO [main] 2012-09-10 08:00:22,087 StorageService.java (line 1098)=
 Node /<a href=3D"http://10.190.221.204">10.190.221.204</a> state jump to n=
ormal</div><div>=A0INFO [main] 2012-09-10 08:00:22,088 StorageService.java =
(line 666) Bootstrap/Replace/Move completed! Now serving reads.</div>

<div>=A0INFO [main] 2012-09-10 08:00:22,089 Mx4jTool.java (line 72) Will no=
t load MX4J, mx4j-tools.jar is not in the classpath</div><div>=A0INFO [main=
] 2012-09-10 08:00:22,119 CassandraDaemon.java (line 124) Binding thrift se=
rvice to /<a href=3D"http://10.190.221.204:9160">10.190.221.204:9160</a></d=
iv>

<div>=A0INFO [main] 2012-09-10 08:00:22,122 CassandraDaemon.java (line 133)=
 Using TFastFramedTransport with a max frame size of 15728640 bytes.</div><=
div>=A0INFO [main] 2012-09-10 08:00:22,125 CassandraDaemon.java (line 160) =
Using synchronous/threadpool thrift server on /<a href=3D"http://10.190.221=
.204">10.190.221.204</a> : 9160</div>

<div>=A0INFO [Thread-5] 2012-09-10 08:00:22,125 CassandraDaemon.java (line =
212) Listening for thrift clients...</div><div>=A0INFO [RMI TCP Connection(=
4)-127.0.0.1] 2012-09-10 08:00:59,014 ColumnFamilyStore.java (line 643) Enq=
ueuing flush of Memtable-Versions@11775580(128/160 serialized/live bytes, 3=
 ops)</div>

<div>=A0INFO [FlushWriter:2] 2012-09-10 08:00:59,016 Memtable.java (line 26=
6) Writing Memtable-Versions@11775580(128/160 serialized/live bytes, 3 ops)=
</div><div>=A0INFO [FlushWriter:2] 2012-09-10 08:00:59,024 Memtable.java (l=
ine 307) Completed flushing /mnt/cassandra/data/system/Versions/system-Vers=
ions-hd-1-Data.db (256 bytes) for commitlog position ReplayPosition(segment=
Id=3D6360937126265, position=3D3854)</div>

<div><div>=A0INFO [GossipStage:1] 2012-09-10 08:01:23,036 Gossiper.java (li=
ne 850) Node /<a href=3D"http://10.72.201.80">10.72.201.80</a> is now part =
of the cluster</div><div>=A0INFO [GossipStage:1] 2012-09-10 08:01:23,037 Go=
ssiper.java (line 816) InetAddress /<a href=3D"http://10.72.201.80">10.72.2=
01.80</a> is now UP</div>

<div>=A0INFO [GossipStage:1] 2012-09-10 08:01:23,038 StorageService.java (l=
ine 1126) Nodes /<a href=3D"http://10.72.201.80">10.72.201.80</a> and /<a h=
ref=3D"http://10.190.221.204">10.190.221.204</a> have the same token 166594=
924822352415786406422619018814804. =A0Ignoring /<a href=3D"http://10.72.201=
.80">10.72.201.80</a></div>

<div>=A0INFO [GossipTasks:1] 2012-09-10 08:01:32,967 Gossiper.java (line 83=
0) InetAddress /<a href=3D"http://10.72.201.80">10.72.201.80</a> is now dea=
d.</div><div>=A0INFO [GossipTasks:1] 2012-09-10 08:01:53,976 Gossiper.java =
(line 644) FatClient /<a href=3D"http://10.72.201.80">10.72.201.80</a> has =
been silent for 30000ms, removing from gossip</div>

</div><div><br></div><div><br></div>

--20cf303b430f7906fb04c95480c9--