Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A2D999EBD for ; Thu, 10 Nov 2011 21:45:18 +0000 (UTC) Received: (qmail 56081 invoked by uid 500); 10 Nov 2011 21:45:16 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 56054 invoked by uid 500); 10 Nov 2011 21:45:16 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 56046 invoked by uid 99); 10 Nov 2011 21:45:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Nov 2011 21:45:16 +0000 X-ASF-Spam-Status: No, hits=-0.6 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of tsmith84@gmail.com designates 209.85.213.44 as permitted sender) Received: from [209.85.213.44] (HELO mail-yw0-f44.google.com) (209.85.213.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Nov 2011 21:45:08 +0000 Received: by ywt34 with SMTP id 34so1487364ywt.31 for ; Thu, 10 Nov 2011 13:44:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=+q5y+Si7Ljrl0LiYE/sjI0X1ZZenVmq85pSDPk7YHfs=; b=NFglvBK6auVAx7OX/Ap3OYHr97zC5Zi58XMTUfi5o+Iwmty+W9C8lyasjrggYvLoSt kv1AyH5IeRieMv/UKy3M6PhsldC94GRnz2FPbApe6gH3iPAyWCy8rJMjCDSCvISXVK9s U0EkGgwbbnYr+2H4XruDklcuH2kswysPO539M= MIME-Version: 1.0 Received: by 10.101.85.2 with SMTP id n2mr1333445anl.95.1320961488104; Thu, 10 Nov 2011 13:44:48 -0800 (PST) Received: by 10.100.133.10 with HTTP; Thu, 10 Nov 2011 13:44:48 -0800 (PST) Date: Thu, 10 Nov 2011 13:44:48 -0800 Message-ID: Subject: Not all nodes see the complete ring From: Timothy Smith To: user@cassandra.apache.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org I=92m curious if anyone has ever seen this happen or has any idea how it would happen. I have a 10 cluster node with 5 nodes in each data center running .6 (we're working on the upgrade now). I had several nodes with forgotten deletes so I failed the nodes and bootstrapped them back into the cluster one at a time. Everything seemed fine, but now I=92m noticing that all systems in 1 of my data centers see all 10 nodes and all systems in the other data center see just 9. I=92m figuring now is the time to fail the node that only half the other nodes can see, but what would cause this to happen? -Tim Smith DC1 10.x.x.45 Up 162065212751151145161126595807335373 |<--| 10.x.x.44 Up 5452449782323250074504667089218893518 | = ^ 10.x.x.43 Up 8114257989534302620064490155463988554 v | 10.x.x.46 Up 21422567192334579300859480282267974118 | = ^ 10.x.x.60 Up 54861697885175209049354960363878287097 v = | 10.x.x.69 Up 154328840302872203985032035664154382201 | ^ 10.x.x.62 Up 156995951391754654763374624484548356765 v | 10.x.x.61 Up 158321671343439891722659169797597266747 | ^ 10.x.x.47 Up 159657096314745030420102789742477598562 |-->= | DC2 10.x.x.45 Up 162065212751151145161126595807335373 |<--| 10.x.x.44 Up 5452449782323250074504667089218893518 | = ^ 10.x.x.43 Up 8114257989534302620064490155463988554 v | 10.x.x.46 Up 21422567192334579300859480282267974118 | ^ 10.x.x.60 Up 54861697885175209049354960363878287097 v = | 10.x.x.71 Up 149032703168324939856639604911542585192 | = ^ 10.x.x.69 Up 154328840302872203985032035664154382201 v = | 10.x.x.62 Up 156995951391754654763374624484548356765 | = ^ 10.x.x.61 Up 158321671343439891722659169797597266747 v | 10.x.x.47 Up 159657096314745030420102789742477598562 |-->|