cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Jordan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5692) Race condition in detecting version on a mixed 1.1/1.2 cluster
Date Tue, 15 Oct 2013 14:54:42 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795253#comment-13795253
] 

Jeremiah Jordan commented on CASSANDRA-5692:
--------------------------------------------

[~sbtourist] no thread dumps.  given nodes e1-e5.  Queries to e1, and "describe cluster" on
e1 showed all nodes.  Queries to e5 would usually timeout, and "describe cluster;" on e5 would
show e1-e3 as "UNAVAILABLE".  netstat -tn | grep <e1 ip> on e5 showed:

{noformat}
e5:7000 <-> e1:<high port>
e5:<high port> <-> e1:7000
e5:<high port 2> <-> e1:7000
{noformat}

Where e5 to a node which responded to the "describe cluster;" showed:

{noformat}
e5:7000 <-> e4:<high port>
e5:7000 <-> e4:<high port 2>
e5:<high port> <-> e4:7000
e5:<high port 2> <-> e4:7000
{noformat}

And in the TRACE level logs for IncomingTcpConnection.java when restarting e5 I only see one
connection come in from e1, but there are two that come in from e4.  When I enabled TRACE
level logs for e1, it was just spitting out something about "version 5" over and over really
fast.  At that point we restarted e1, and it came up and everything was happy from e1->e5,
so then we did a rolling restart on the cluster and went to bed.


> Race condition in detecting version on a mixed 1.1/1.2 cluster
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-5692
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5692
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.1.9, 1.2.5
>            Reporter: Sergio Bossa
>            Assignee: Sergio Bossa
>            Priority: Minor
>             Fix For: 1.2.7, 2.0 beta 1
>
>         Attachments: 5692-0005.patch, 5692-0006.patch
>
>
> On a mixed 1.1 / 1.2 cluster, starting 1.2 nodes fires sometimes a race condition in
version detection, where the 1.2 node wrongly detects version 6 for a 1.1 node.
> It works as follows:
> 1) The just started 1.2 node quickly opens an OutboundTcpConnection toward a 1.1 node
before receiving any messages from the latter.
> 2) Given the version is correctly detected only when the first message is received, the
version is momentarily set at 6.
> 3) This opens an OutboundTcpConnection from 1.2 to 1.1 at version 6, which gets stuck
in the connect() method.
> Later, the version is correctly fixed, but all outbound connections from 1.2 to 1.1 are
stuck at this point.
> Evidence from 1.2 logs:
> TRACE 13:48:31,133 Assuming current protocol version for /127.0.0.2
> DEBUG 13:48:37,837 Setting version 5 for /127.0.0.2



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message