cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "graham sanderson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7734) Schema pushes (seemingly) randomly not happening
Date Tue, 12 Aug 2014 03:59:11 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093698#comment-14093698
] 

graham sanderson commented on CASSANDRA-7734:
---------------------------------------------

Note totality of related logging from 172.16.26.11 (the first node in a six node cluster)

{code}
date
Tue Aug 12 05:56:52 CEST 2014
{code}

{code}
grep " -> " /var/log/cassandra/system.log*
/var/log/cassandra/system.log:INFO  [Thread-10] 2014-08-11 06:43:38,440 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.16
/var/log/cassandra/system.log:INFO  [Thread-11] 2014-08-11 06:43:38,441 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.13
/var/log/cassandra/system.log:INFO  [Thread-12] 2014-08-11 06:43:38,445 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.14
/var/log/cassandra/system.log:INFO  [Thread-11] 2014-08-11 06:43:38,461 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.13
/var/log/cassandra/system.log:INFO  [Thread-12] 2014-08-11 06:43:38,462 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.14
/var/log/cassandra/system.log:INFO  [Thread-13] 2014-08-11 06:43:38,464 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.14
/var/log/cassandra/system.log:INFO  [Thread-10] 2014-08-11 06:43:38,468 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.16
/var/log/cassandra/system.log:INFO  [Thread-15] 2014-08-11 06:43:38,472 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.16
/var/log/cassandra/system.log:INFO  [Thread-16] 2014-08-11 06:43:38,475 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.13
/var/log/cassandra/system.log:INFO  [Thread-19] 2014-08-11 06:43:38,538 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.12
/var/log/cassandra/system.log:INFO  [Thread-20] 2014-08-11 06:43:38,591 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.15
/var/log/cassandra/system.log:INFO  [Thread-20] 2014-08-11 06:43:38,618 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.15
/var/log/cassandra/system.log:INFO  [Thread-21] 2014-08-11 06:43:38,621 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.15
/var/log/cassandra/system.log:INFO  [Thread-19] 2014-08-11 06:43:39,221 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.12
/var/log/cassandra/system.log:INFO  [Thread-23] 2014-08-11 06:43:39,225 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.12
/var/log/cassandra/system.log:INFO  [Thread-24] 2014-08-11 06:51:55,694 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.12
/var/log/cassandra/system.log:INFO  [Thread-37] 2014-08-11 06:53:26,679 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.12
/var/log/cassandra/system.log:INFO  [Thread-37] 2014-08-11 06:53:26,836 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.12
/var/log/cassandra/system.log:INFO  [Thread-38] 2014-08-11 06:53:26,839 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.12
/var/log/cassandra/system.log:INFO  [Thread-16] 2014-08-11 06:54:28,603 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.13
/var/log/cassandra/system.log:INFO  [Thread-43] 2014-08-11 06:55:41,762 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.13
/var/log/cassandra/system.log:INFO  [Thread-43] 2014-08-11 06:55:41,833 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.13
/var/log/cassandra/system.log:INFO  [Thread-44] 2014-08-11 06:55:41,843 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.13
/var/log/cassandra/system.log:INFO  [Thread-14] 2014-08-11 06:57:32,128 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.14
/var/log/cassandra/system.log:INFO  [Thread-49] 2014-08-11 06:58:35,688 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.14
/var/log/cassandra/system.log:INFO  [Thread-21] 2014-08-11 07:00:01,122 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.15
/var/log/cassandra/system.log:INFO  [Thread-53] 2014-08-11 07:01:01,692 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.15
/var/log/cassandra/system.log:INFO  [Thread-15] 2014-08-11 07:01:26,955 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.16
/var/log/cassandra/system.log:INFO  [Thread-57] 2014-08-11 07:02:48,342 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.16
/var/log/cassandra/system.log:INFO  [Thread-57] 2014-08-11 07:02:48,637 MessagingService.java
(line 782) Reseting version 7 -> null for /172.16.26.16
/var/log/cassandra/system.log:INFO  [Thread-58] 2014-08-11 07:02:48,641 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.16
/var/log/cassandra/system.log:INFO  [Thread-92] 2014-08-11 07:29:34,332 MessagingService.java
(line 767) Updating version null -> 7 for /172.16.26.11
{code}

Thus far, there are periods (but not prolonged periods) where the version is null. That said,
I don't think this endpoint->version mapping acts the way the recent code updates expect

> Schema pushes (seemingly) randomly not happening
> ------------------------------------------------
>
>                 Key: CASSANDRA-7734
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7734
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: graham sanderson
>
> We have been seeing problems since upgrade to 2.0.9 from 2.0.5.
> Basically after a while, new schema changes (we periodically add tables) start propagating
very slowly to some nodes and fast to others. It looks from the logs and trace that in this
case the "push" of the schema never happens (note a node has decided not to push to another
node, it doesn't seem to start again) from the originating node to some of the other nodes.
In this case though, we do see the other node end up pulling the schema some time later when
it notices its schema is out of date.
> Here is code from 2.0.9 MigrationManager.announce
> {code}
>        for (InetAddress endpoint : Gossiper.instance.getLiveMembers())
>         {
>             // only push schema to nodes with known and equal versions
>             if (!endpoint.equals(FBUtilities.getBroadcastAddress()) &&
>                     MessagingService.instance().knowsVersion(endpoint) &&
>                     MessagingService.instance().getRawVersion(endpoint) == MessagingService.current_version)
>                 pushSchemaMutation(endpoint, schema);
>         }
> {code}
> and from 2.0.5
> {code}
>         for (InetAddress endpoint : Gossiper.instance.getLiveMembers())
>         {
>             if (endpoint.equals(FBUtilities.getBroadcastAddress()))
>                 continue; // we've dealt with localhost already
>             // don't send schema to the nodes with the versions older than current major
>             if (MessagingService.instance().getVersion(endpoint) < MessagingService.current_version)
>                 continue;
>             pushSchemaMutation(endpoint, schema);
> 	}
> {code}
> the old getVersion() call would return MessagingService.current_version if the version
was unknown, so the push would occur in this case. I don't have logging to prove this, but
have strong suspicion that the version may end up null in some cases (which would have allowed
schema propagation in 2.0.5, but not by somewhere after that and <= 2.0.9)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message