Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Mon, 16 Dec 2013 18:43:36 +0000 (UTC)
From: "Russell Alexander Spitzer (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12685014.1387219108383.44466.1387219416797@arcas>
In-Reply-To: <JIRA.12685014.1387219108383@arcas>
References: <JIRA.12685014.1387219108383@arcas>
Subject: [jira] [Commented] (CASSANDRA-6493) Exceptions when a second
 Datacenter is Added
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/CASSANDRA-6493?page=3Dcom.atlas=
sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D=
13849448#comment-13849448 ]=20

Russell Alexander Spitzer commented on CASSANDRA-6493:
------------------------------------------------------

https://cassci.datastax.com/job/cassandra-addremovedc/25/console

The "Node down Detected" are messages from a thread which runs nodetool sta=
tus every ~2 seconds and counts how many nodes report themselves as up, the=
 lack of a command line output shows the command failed.=20

> Exceptions when a second Datacenter is Added
> --------------------------------------------
>
>                 Key: CASSANDRA-6493
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6493
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu, EC2 M1.large
>            Reporter: Russell Alexander Spitzer
>
> On adding a second datacenter several exceptions were raised.
> Test outline:
> Start 25 Node DC1
> Keyspace Setup Replication 3
> Begin insert against DC1 Using Stress
> While the inserts are occuring
> Start up 25 Node DC2
> Alter Keyspace to include Replication in 2nd DC
> Run rebuild on DC2
> Wait for stress to finish
> Run repair on Cluster
> ... Some other operations
> At the point when the second datacenter is added several warnings go off =
because nodetool status is not functioning, and a few moments later the sta=
rt operation reports a failure because a node has not successfully turned o=
n.=20
> The first start attempt yielded the following exception on a node in the =
second DC.
> {code}
> CassandraDaemon.java (line 464) Exception encountered during startup
> java.lang.AssertionError: -7560216458456714666 not found in -922206027867=
3125462, -9220751250790085193, ..... ALL THE TOKENS ...,=E2=80=82 921857585=
1928340117, 9219681798686280387
> at org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMetadat=
a.java:752)
> at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(TokenMe=
tadata.java:696)
> at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(TokenMet=
adata.java:703)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddre=
sses(AbstractReplicationStrategy.java:187)
> at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(Rang=
eStreamer.java:147)
> at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:12=
1)
> at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
> at org.apache.cassandra.service.StorageService.bootstrap(StorageService.j=
ava:979)
> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageServi=
ce.java:745)
> =09at org.apache.cassandra.service.StorageService.initServer(StorageServi=
ce.java:586)
> =09at org.apache.cassandra.service.StorageService.initServer(StorageServi=
ce.java:483)
> =09at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.=
java:348)
> =09at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaem=
on.java:447)
> =09at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.j=
ava:490)
> {code}
> The test automatically tries to restart nodes if they fail during startup=
, The second attempt for this node succeeded but a 'nodetool status' still =
failed and a different node in the second DC logged the following and faile=
d to start up.
> {code}
> ERROR [main] 2013-12-16 18:02:04,869 CassandraDaemon.java (line 464) Exce=
ption encountered during startup
> java.util.ConcurrentModificationException
> =09at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
> =09at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
> =09at org.apache.commons.lang.StringUtils.join(StringUtils.java:3382)
> =09at org.apache.commons.lang.StringUtils.join(StringUtils.java:3444)
> =09at org.apache.cassandra.locator.TokenMetadata.getPredecessor(TokenMeta=
data.java:752)
> =09at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangesFor(Toke=
nMetadata.java:696)
> =09at org.apache.cassandra.locator.TokenMetadata.getPrimaryRangeFor(Token=
Metadata.java:703)
> =09at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAd=
dresses(AbstractReplicationStrategy.java:187)
> =09at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithSourcesFor(R=
angeStreamer.java:147)
> =09at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java=
:121)
> =09at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:8=
1)
> =09at org.apache.cassandra.service.StorageService.bootstrap(StorageServic=
e.java:979)
> =09at org.apache.cassandra.service.StorageService.joinTokenRing(StorageSe=
rvice.java:745)
> =09at org.apache.cassandra.service.StorageService.initServer(StorageServi=
ce.java:586)
> =09at org.apache.cassandra.service.StorageService.initServer(StorageServi=
ce.java:483)
> =09at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.=
java:348)
> =09at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaem=
on.java:447)
> =09at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.j=
ava:490)
> ERROR [StorageServiceShutdownHook] 2013-12-16 18:02:04,876 CassandraDaemo=
n.java (line 191) Exception in thread Thread[StorageServiceShutdownHook,5,m=
ain]
> java.lang.NullPointerException
> =09at org.apache.cassandra.service.StorageService.stopNativeTransport(Sto=
rageService.java:358)
> =09at org.apache.cassandra.service.StorageService.shutdownClientServers(S=
torageService.java:373)
> =09at org.apache.cassandra.service.StorageService.access$000(StorageServi=
ce.java:89)
> =09at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageSe=
rvice.java:551)
> =09at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java=
:28)
> =09at java.lang.Thread.run(Thread.java:724)
> {code}


--
This message was sent by Atlassian JIRA
(v6.1.4#6159)