cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-5171) Save EC2Snitch topology information in system table
Date Wed, 22 May 2013 15:53:22 GMT


Jonathan Ellis updated CASSANDRA-5171:

             Priority: Critical  (was: Trivial)
    Affects Version/s:     (was: 1.2.0)
        Fix Version/s:     (was: 1.2.1)
           Issue Type: Bug  (was: Improvement)
              Summary: Save EC2Snitch topology information in system table  (was: Enhance
Ec2Snitch gossip info.)

This was reverted in CASSANDRA-5432, but I think the problem it solves is actually pretty
severe, so I'm reopening it.

The problem is that pretty much everything from TokenMetadata to NetworkTopologyStrategy assumes
that once we see a node, the snitch can tell us where it lives, and in particular that once
the snitch tells us where a node lives it won't change its answer.

So this is problematic:

    public String getDatacenter(InetAddress endpoint)
        if (endpoint.equals(FBUtilities.getBroadcastAddress()))
            return ec2region;
        EndpointState state = Gossiper.instance.getEndpointStateForEndpoint(endpoint);
        if (state == null || state.getApplicationState(ApplicationState.DC) == null)
            return DEFAULT_DC;
        return state.getApplicationState(ApplicationState.DC).value;

That is, if we don't know where a node belongs (e.g., we just restarted and haven't been gosipped
to yet), assume it's in {{DEFAULT_DC}}.

This can lead to data loss.  Consider node X in DC1, where keyspace KS is replicated.  Suddenly
X is yanked out of DC1 and placed in DC2, where KS is not replicated.  Nobody will bother
querying X for the data in KS that was formerly replicated to it.  Even repair will not see
> Save EC2Snitch topology information in system table
> ---------------------------------------------------
>                 Key: CASSANDRA-5171
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.1
>         Environment: EC2
>            Reporter: Vijay
>            Assignee: Vijay
>            Priority: Critical
>             Fix For: 1.2.6
>         Attachments: 0001-CASSANDRA-5171.patch
> EC2Snitch currently waits for the Gossip information to understand the cluster information
every time we restart. It will be nice to use already available system table info similar
to GPFS.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message