Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B25C6E45C for ; Thu, 14 Feb 2013 03:40:13 +0000 (UTC) Received: (qmail 23555 invoked by uid 500); 14 Feb 2013 03:40:13 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 23531 invoked by uid 500); 14 Feb 2013 03:40:13 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 23512 invoked by uid 99); 14 Feb 2013 03:40:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Feb 2013 03:40:13 +0000 Date: Thu, 14 Feb 2013 03:40:13 +0000 (UTC) From: "Brandon Williams (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-5129) newly bootstrapping nodes hang indefinitely in STATUS:BOOT while JOINING cluster MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578126#comment-13578126 ] Brandon Williams commented on CASSANDRA-5129: --------------------------------------------- Easily repros with toy data from stress: {noformat} INFO 03:30:47,313 JOINING: Starting to bootstrap... INFO 03:30:48,522 Submitting index build of [Standard1.Idx1] for data in SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ib-1-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ib-2-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ib-3-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ib-4-Data.db') INFO 03:30:48,526 Enqueuing flush of Memtable-compactions_in_progress@893461718(177/177 serialized/live bytes, 7 ops) INFO 03:30:48,527 Writing Memtable-compactions_in_progress@893461718(177/177 serialized/live bytes, 7 ops) INFO 03:30:48,546 Completed flushing /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ib-1-Data.db (176 bytes) for commitlog position ReplayPosition(segmentId=1360812614633, position=75619) INFO 03:30:48,547 Compacting [SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ib-3-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ib-1-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ib-4-Data.db'), SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ib-2-Data.db')] {noformat} and stays like that forever. > newly bootstrapping nodes hang indefinitely in STATUS:BOOT while JOINING cluster > ---------------------------------------------------------------------------------- > > Key: CASSANDRA-5129 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5129 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 1.2.0 > Environment: Ubuntu 12.04 > Reporter: Michael Kjellman > > bootstrapping a new node causes it to hang indefinitely in STATUS:BOOT > Nodes streaming to the new node report > {code} > Mode: NORMAL > Nothing streaming to /10.8.30.16 > Not receiving any streams. > Pool Name Active Pending Completed > Commands n/a 0 1843990 > Responses n/a 2 661750 > {code} > the node being streamed to stuck in the JOINING state reports: > {code} > Mode: JOINING > Not sending any streams. > Nothing streaming from /10.8.30.103 > Nothing streaming from /10.8.30.102 > Pool Name Active Pending Completed > Commands n/a 0 10 > Responses n/a 0 613577 > {code} > it appears that the nodes in the "nothing streaming" state never sends a "finished streaming" to the joining node. > no exceptions are thrown during the streaming on either node while the node is in this state. > {code:name="full gossip state of bootstrapping node"} > /10.8.30.16 > NET_VERSION:6 > RELEASE_VERSION:1.2.0 > STATUS:BOOT,127605887595351923798765477786913079289 > RACK:RAC1 > RPC_ADDRESS:0.0.0.0 > DC:DC1 > SCHEMA:5cd8420d-ce3c-3625-8293-67558a24816b > HOST_ID:e20817ce-7454-4dc4-a1c6-b1dec35c4491 > LOAD:1.11824041581E11 > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira