cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brandon Williams <dri...@gmail.com>
Subject Re: ApplicationState Schema has drifted from DatabaseDescriptor
Date Wed, 09 Feb 2011 22:55:44 GMT
On Wed, Feb 9, 2011 at 4:31 PM, Aaron Morton <aaron@thelastpickle.com>wrote:

> Thanks Gary. I'll keep an eye on things and see if it happens again.
>
> From reading the code I'm wondering if there is a small chance of a race
> condition in HintedHandoffManager.waitForSchemaAgreement() .
>
> Could the following happen? I'm a little unsure on exactly how the endpoint
> state is removed from the map in Gossiper.
>
> 1) node 1 starts
> 2) Gossiper calls StorageService.onAlive() when the endpoints are detected
> as alive.
> 3) HintedHandoffManager.deliverHints() adds a runnable to the HintedHandoff
> TP
> 4) This happens several times, and node 1 gets busy delivering hints but
> there is only 1 thread in the thread pool.
> 5) Node n is removed from the cluster and the endpoint state is deleted in
> the Gossiper on node 1
> 6) Node 1 gets around to processing the hints for node n and
> Gossiper.getEndpointStateForEndpoint() returns null for node n
>

Yes, this is currently possible, but you have to decommission the node
before the schema check/sleep portion of HH is over, which is unlikely in
practice.  It will be especially unlikely after
https://issues.apache.org/jira/browse/CASSANDRA-2115.

-Brandon

Mime
View raw message