cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Tolbert <andrew.tolb...@datastax.com>
Subject Re: Unexpected error while refreshing token map, keeping previous version (IllegalArgumentException: Multiple entries with same key ?
Date Thu, 20 Jun 2019 15:34:03 GMT
I just configured a 3 node cluster in this way and was able to reproduce
the warning message:

cqlsh> select peer, rpc_address from system.peers;

 peer      | rpc_address
-----------+-------------
 127.0.0.3 |   127.0.0.1
 127.0.0.2 |   127.0.0.1

(2 rows)

cqlsh> select rpc_address from system.local;

 rpc_address
-------------
   127.0.0.1

10:22:40.399 [s0-admin-0] WARN  c.d.o.d.i.c.metadata.DefaultMetadata - [s0]
Unexpected error while refreshing token map, keeping previous version
java.lang.IllegalArgumentException: Multiple entries with same key:
Murmur3Token(-100881582699237014)=/127.0.0.1:9042 and
Murmur3Token(-100881582699237014)=/127.0.0.1:9042
at
com.datastax.oss.driver.shaded.guava.common.collect.ImmutableMap.conflictException(ImmutableMap.java:215)
at
com.datastax.oss.driver.shaded.guava.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:209)
at
com.datastax.oss.driver.shaded.guava.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:147)
at
com.datastax.oss.driver.shaded.guava.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:110)
at
com.datastax.oss.driver.shaded.guava.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:393)
at
com.datastax.oss.driver.internal.core.metadata.token.DefaultTokenMap.buildTokenToPrimaryAndRing(DefaultTokenMap.java:261)
at
com.datastax.oss.driver.internal.core.metadata.token.DefaultTokenMap.build(DefaultTokenMap.java:57)
at
com.datastax.oss.driver.internal.core.metadata.DefaultMetadata.rebuildTokenMap(DefaultMetadata.java:146)
at
com.datastax.oss.driver.internal.core.metadata.DefaultMetadata.withNodes(DefaultMetadata.java:104)
at
com.datastax.oss.driver.internal.core.metadata.InitialNodeListRefresh.compute(InitialNodeListRefresh.java:96)
at
com.datastax.oss.driver.internal.core.metadata.MetadataManager.apply(MetadataManager.java:475)
at
com.datastax.oss.driver.internal.core.metadata.MetadataManager$SingleThreaded.refreshNodes(MetadataManager.java:299)
at
com.datastax.oss.driver.internal.core.metadata.MetadataManager$SingleThreaded.access$1700(MetadataManager.java:265)
at
com.datastax.oss.driver.internal.core.metadata.MetadataManager.lambda$refreshNodes$0(MetadataManager.java:155)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

Interestingly enough, the version 3 of the driver only recognizes 1 node,
where version 4 is able to detect 3 nodes separately.  It's probably not a
scenario that was given a lot of thought since this is a misconfiguration.
Will think about how it should be handled and log tickets in any case, as
would be nice to surface to the user that something isn't right in a more
clear way.

Can you please confirm when you have chance that this is indeed a
configuration issue with rpc_address?  Just to make sure I'm not ignoring a
possible bug ;)

Thanks,
Andy


On Thu, Jun 20, 2019 at 10:20 AM Andy Tolbert <andrew.tolbert@datastax.com>
wrote:

> One thing that strikes me is that the endpoint reported is '127.0.0.1'.
> Is it possible that you have rpc_address set to 127.0.0.1 on each of your
> three nodes in cassandra.yaml?  The driver uses the system.peers table to
> identify nodes in the cluster and associates them by rpc_address.  Can you
> verify this by executing 'select peer, rpc_address from system.peers' to
> see what is being reported as the rpc_address and let me know?
>
> In any case, the driver should probably handle this better, I'll create a
> driver ticket.
>
> Thanks,
> Andy
>
> On Thu, Jun 20, 2019 at 10:03 AM Jeff Jirsa <jjirsa@gmail.com> wrote:
>
>> There’s a reasonable chance this is a bug in the Datastax driver - may
>> want to start there when debugging .
>>
>> It’s also just a warn, and the two entries with the same token are the
>> same endpoint which doesn’t seem concerning to me, but I don’t know the
>> Datastax driver that well
>>
>> On Jun 20, 2019, at 7:40 AM, Котельников Александр <a.kotelnikov@crpt.ru>
>> wrote:
>>
>> It appears that no such warning is issued if I connected to Cassandra
>> from a remote server, not locally.
>>
>>
>>
>> *From: *Котельников Александр <a.kotelnikov@crpt.ru>
>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Date: *Thursday, 20 June 2019 at 10:46
>> *To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Subject: *Unexpected error while refreshing token map, keeping previous
>> version (IllegalArgumentException: Multiple entries with same key ?
>>
>>
>>
>> Hey!
>>
>>
>>
>> I’ve  just configured a test 3-node Cassandra cluster and run very
>> trivial java test against it.
>>
>>
>>
>> I see the following warning from java-driver on each CqlSession
>> initialization:
>>
>>
>>
>> 13:54:13.913 [loader-admin-0] WARN  c.d.o.d.i.c.metadata.DefaultMetadata
>> - [loader] Unexpected error while refreshing token map, keeping previous
>> version (IllegalArgumentException: Multiple entries with same key:
>> Murmur3Token(-1060405237057176857)=/127.0.0.1:9042 and
>> Murmur3Token(-1060405237057176857)=/127.0.0.1:9042)
>>
>>
>>
>> What does It mean? Why?
>>
>>
>>
>> Cassandra 3.11.4, driver 4.0.1.
>>
>>
>>
>> nodetool status
>>
>> Datacenter: datacenter1
>>
>> =======================
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  Address       Load       Tokens       Owns (effective)  Host
>> ID                               Rack
>>
>> UN  10.73.66.36   419.36 MiB  256          100.0%
>> fafa2737-9024-437b-9a59-c1c037bce244  rack1
>>
>> UN  10.73.66.100  336.47 MiB  256          100.0%
>> d5323ad0-f8cd-42d4-b34d-9afcd002ea47  rack1
>>
>> UN  10.73.67.196  336.4 MiB  256          100.0%
>> 74dffe0c-32a4-4071-8b36-5ada5afa4a7d  rack1
>>
>>
>>
>> The issue persists if I reset the cluster, just the token changes its
>> value.
>>
>> Alexander
>>
>>

Mime
View raw message