Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4A62EC260 for ; Tue, 12 Aug 2014 03:59:12 +0000 (UTC) Received: (qmail 23712 invoked by uid 500); 12 Aug 2014 03:59:12 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 23676 invoked by uid 500); 12 Aug 2014 03:59:12 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 23654 invoked by uid 99); 12 Aug 2014 03:59:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Aug 2014 03:59:12 +0000 Date: Tue, 12 Aug 2014 03:59:11 +0000 (UTC) From: "graham sanderson (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-7734) Schema pushes (seemingly) randomly not happening MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093698#comment-14093698 ] graham sanderson commented on CASSANDRA-7734: --------------------------------------------- Note totality of related logging from 172.16.26.11 (the first node in a six node cluster) {code} date Tue Aug 12 05:56:52 CEST 2014 {code} {code} grep " -> " /var/log/cassandra/system.log* /var/log/cassandra/system.log:INFO [Thread-10] 2014-08-11 06:43:38,440 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.16 /var/log/cassandra/system.log:INFO [Thread-11] 2014-08-11 06:43:38,441 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.13 /var/log/cassandra/system.log:INFO [Thread-12] 2014-08-11 06:43:38,445 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.14 /var/log/cassandra/system.log:INFO [Thread-11] 2014-08-11 06:43:38,461 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.13 /var/log/cassandra/system.log:INFO [Thread-12] 2014-08-11 06:43:38,462 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.14 /var/log/cassandra/system.log:INFO [Thread-13] 2014-08-11 06:43:38,464 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.14 /var/log/cassandra/system.log:INFO [Thread-10] 2014-08-11 06:43:38,468 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.16 /var/log/cassandra/system.log:INFO [Thread-15] 2014-08-11 06:43:38,472 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.16 /var/log/cassandra/system.log:INFO [Thread-16] 2014-08-11 06:43:38,475 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.13 /var/log/cassandra/system.log:INFO [Thread-19] 2014-08-11 06:43:38,538 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.12 /var/log/cassandra/system.log:INFO [Thread-20] 2014-08-11 06:43:38,591 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.15 /var/log/cassandra/system.log:INFO [Thread-20] 2014-08-11 06:43:38,618 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.15 /var/log/cassandra/system.log:INFO [Thread-21] 2014-08-11 06:43:38,621 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.15 /var/log/cassandra/system.log:INFO [Thread-19] 2014-08-11 06:43:39,221 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.12 /var/log/cassandra/system.log:INFO [Thread-23] 2014-08-11 06:43:39,225 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.12 /var/log/cassandra/system.log:INFO [Thread-24] 2014-08-11 06:51:55,694 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.12 /var/log/cassandra/system.log:INFO [Thread-37] 2014-08-11 06:53:26,679 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.12 /var/log/cassandra/system.log:INFO [Thread-37] 2014-08-11 06:53:26,836 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.12 /var/log/cassandra/system.log:INFO [Thread-38] 2014-08-11 06:53:26,839 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.12 /var/log/cassandra/system.log:INFO [Thread-16] 2014-08-11 06:54:28,603 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.13 /var/log/cassandra/system.log:INFO [Thread-43] 2014-08-11 06:55:41,762 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.13 /var/log/cassandra/system.log:INFO [Thread-43] 2014-08-11 06:55:41,833 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.13 /var/log/cassandra/system.log:INFO [Thread-44] 2014-08-11 06:55:41,843 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.13 /var/log/cassandra/system.log:INFO [Thread-14] 2014-08-11 06:57:32,128 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.14 /var/log/cassandra/system.log:INFO [Thread-49] 2014-08-11 06:58:35,688 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.14 /var/log/cassandra/system.log:INFO [Thread-21] 2014-08-11 07:00:01,122 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.15 /var/log/cassandra/system.log:INFO [Thread-53] 2014-08-11 07:01:01,692 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.15 /var/log/cassandra/system.log:INFO [Thread-15] 2014-08-11 07:01:26,955 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.16 /var/log/cassandra/system.log:INFO [Thread-57] 2014-08-11 07:02:48,342 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.16 /var/log/cassandra/system.log:INFO [Thread-57] 2014-08-11 07:02:48,637 MessagingService.java (line 782) Reseting version 7 -> null for /172.16.26.16 /var/log/cassandra/system.log:INFO [Thread-58] 2014-08-11 07:02:48,641 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.16 /var/log/cassandra/system.log:INFO [Thread-92] 2014-08-11 07:29:34,332 MessagingService.java (line 767) Updating version null -> 7 for /172.16.26.11 {code} Thus far, there are periods (but not prolonged periods) where the version is null. That said, I don't think this endpoint->version mapping acts the way the recent code updates expect > Schema pushes (seemingly) randomly not happening > ------------------------------------------------ > > Key: CASSANDRA-7734 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7734 > Project: Cassandra > Issue Type: Bug > Reporter: graham sanderson > > We have been seeing problems since upgrade to 2.0.9 from 2.0.5. > Basically after a while, new schema changes (we periodically add tables) start propagating very slowly to some nodes and fast to others. It looks from the logs and trace that in this case the "push" of the schema never happens (note a node has decided not to push to another node, it doesn't seem to start again) from the originating node to some of the other nodes. In this case though, we do see the other node end up pulling the schema some time later when it notices its schema is out of date. > Here is code from 2.0.9 MigrationManager.announce > {code} > for (InetAddress endpoint : Gossiper.instance.getLiveMembers()) > { > // only push schema to nodes with known and equal versions > if (!endpoint.equals(FBUtilities.getBroadcastAddress()) && > MessagingService.instance().knowsVersion(endpoint) && > MessagingService.instance().getRawVersion(endpoint) == MessagingService.current_version) > pushSchemaMutation(endpoint, schema); > } > {code} > and from 2.0.5 > {code} > for (InetAddress endpoint : Gossiper.instance.getLiveMembers()) > { > if (endpoint.equals(FBUtilities.getBroadcastAddress())) > continue; // we've dealt with localhost already > // don't send schema to the nodes with the versions older than current major > if (MessagingService.instance().getVersion(endpoint) < MessagingService.current_version) > continue; > pushSchemaMutation(endpoint, schema); > } > {code} > the old getVersion() call would return MessagingService.current_version if the version was unknown, so the push would occur in this case. I don't have logging to prove this, but have strong suspicion that the version may end up null in some cases (which would have allowed schema propagation in 2.0.5, but not by somewhere after that and <= 2.0.9) -- This message was sent by Atlassian JIRA (v6.2#6252)