cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <>
Subject [jira] [Comment Edited] (CASSANDRA-11038) Is node being restarted treated as node joining?
Date Wed, 10 Feb 2016 17:02:18 GMT


Sylvain Lebresne edited comment on CASSANDRA-11038 at 2/10/16 5:01 PM:

And this does seem to still be a thing since we have [a dtest|]
that actively assert that a {{NEW_NODE}} is send in that case (not sure why no red flags were
raised when writing that test but that sound obviously wrong to me).

was (Author: slebresne):
And this does seem to still be a thing since we have [a dtest|]
that actively assert that a {{NEW_NODE}} is send in that case.

> Is node being restarted treated as node joining?
> ------------------------------------------------
>                 Key: CASSANDRA-11038
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Distributed Metadata
>            Reporter: cheng ren
>            Priority: Minor
> Hi, 
> What we found recently is that every time we restart a node, all other nodes in the cluster
treat the restarted node as a new node joining and issue node joining notification to clients.
We have traced the code path being hit when a peer node detected a restarted node:
> src/java/org/apache/cassandra/gms/
> {code}
>     private void handleMajorStateChange(InetAddress ep, EndpointState epState)
>     {
>         if (!isDeadState(epState))
>         {
>             if (endpointStateMap.get(ep) != null)
>       "Node {} has restarted, now UP", ep);
>             else
>       "Node {} is now part of the cluster", ep);
>         }
>         if (logger.isTraceEnabled())
>             logger.trace("Adding endpoint state for " + ep);
>         endpointStateMap.put(ep, epState);
>         // the node restarted: it is up to the subscriber to take whatever action is
>         for (IEndpointStateChangeSubscriber subscriber : subscribers)
>             subscriber.onRestart(ep, epState);
>         if (!isDeadState(epState))
>             markAlive(ep, epState);
>         else
>         {
>             logger.debug("Not marking " + ep + " alive due to dead state");
>             markDead(ep, epState);
>         }
>         for (IEndpointStateChangeSubscriber subscriber : subscribers)
>             subscriber.onJoin(ep, epState);
>     }
> {code}
> subscriber.onJoin(ep, epState) ends up with calling onJoinCluster in
> {code}
> src/java/org/apache/cassandra/transport/
>         public void onJoinCluster(InetAddress endpoint)
>         {
> server.connectionTracker.send(Event.TopologyChange.newNode(getRpcAddress(endpoint), server.socket.getPort()));
>         }
> {code}
> We have a full trace of code path and skip some intermedia function calls here for being
> Upon receiving the node joining notification, clients would go and scan system peer table
to fetch the latest topology information. Since we have tens of thousands of client connections,
scans from all of them put an enormous load to our cluster. 
> Although in the newer version of driver, client skips fetching peer table if the new
node has already existed in local metadata, we are still curious why node being restarted
is handled as node joining on server side? Did we hit a bug or this is the way supposed to
be? Our old java driver version is 1.0.4 and cassandra version is 2.0.12.
> Thanks!

This message was sent by Atlassian JIRA

View raw message