Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4099A104EC for ; Mon, 29 Apr 2013 23:22:17 +0000 (UTC) Received: (qmail 91614 invoked by uid 500); 29 Apr 2013 23:22:17 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 91594 invoked by uid 500); 29 Apr 2013 23:22:17 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 91582 invoked by uid 99); 29 Apr 2013 23:22:17 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Apr 2013 23:22:17 +0000 Date: Mon, 29 Apr 2013 23:22:17 +0000 (UTC) From: "John Watson (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-5525) Adding nodes to 1.2 cluster w/ vnodes streamed more data than average node load MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Watson updated CASSANDRA-5525: ----------------------------------- Attachment: Screen Shot 2013-04-25 at 12.35.24 PM.png Actual disk space usage. Gray line is the joining node that out of disk space. Dark blue is the joining node I killed. > Adding nodes to 1.2 cluster w/ vnodes streamed more data than average node load > ------------------------------------------------------------------------------- > > Key: CASSANDRA-5525 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5525 > Project: Cassandra > Issue Type: Bug > Reporter: John Watson > Attachments: Screen Shot 2013-04-25 at 12.35.24 PM.png > > > 12 node cluster upgraded from 1.1.9 to 1.2.3, enabled 'num_tokens: 256', restarted and ran upgradesstables and cleanup. > Tried to join 2 additional nodes into the ring. > However, 1 of the new nodes ran out of disk space. This started causing 'no host id' alerts when attempting to store hints for that node. > {noformat} > ERROR 10:12:02,408 Exception in thread Thread[MutationStage:190,5,main] > java.lang.AssertionError: Missing host ID > {noformat} > The other node I killed to stop it from continuing to join. Since the live cluster was now in some sort of broken state dropping mutation messages on a 3 nodes. This was fixed by restarting them, however 1 node never stopped, so had to decomm it (leaving the original cluster at 11 nodes.) > Ring pre-join: > {noformat} > Load Tokens Owns (effective) Host ID > 147.55 GB 256 16.7% 754f9f4c-4ba7-4495-97e7-1f5b6755cb27 > 124.99 GB 256 16.7% 93f4400a-09d9-4ca0-b6a6-9bcca2427450 > 136.63 GB 256 16.7% ff821e8e-b2ca-48a9-ac3f-8234b16329ce > 141.78 GB 253 100.0% 339c474f-cf19-4ada-9a47-8b10912d5eb3 > 137.74 GB 256 16.7% 6d726cbf-147d-426e-a735-e14928c95e45 > 135.9 GB 256 16.7% e59a02b3-8b91-4abd-990e-b3cb2a494950 > 165.96 GB 256 16.7% 83ca527c-60c5-4ea0-89a8-de53b92b99c8 > 135.41 GB 256 16.7% c3ea4026-551b-4a14-a346-480e8c1fe283 > 143.38 GB 256 16.7% df7ba879-74ad-400b-b371-91b45dcbed37 > 178.05 GB 256 25.0% 78192d73-be0b-4d49-a129-9bec0770efed > 194.92 GB 256 25.0% 361d7e31-b155-4ce1-8890-451b3ddf46cf > 150.5 GB 256 16.7% 9889280a-1433-439e-bb84-6b7e7f44d761 > {noformat} > Ring after decomm bad node: > {noformat} > Load Tokens Owns (effective) Host ID > 80.95 GB 256 16.7% 754f9f4c-4ba7-4495-97e7-1f5b6755cb27 > 87.15 GB 256 16.7% 93f4400a-09d9-4ca0-b6a6-9bcca2427450 > 98.16 GB 256 16.7% ff821e8e-b2ca-48a9-ac3f-8234b16329ce > 142.6 GB 253 100.0% 339c474f-cf19-4ada-9a47-8b10912d5eb3 > 77.64 GB 256 16.7% e59a02b3-8b91-4abd-990e-b3cb2a494950 > 194.31 GB 256 25.0% 6d726cbf-147d-426e-a735-e14928c95e45 > 221.94 GB 256 33.3% 83ca527c-60c5-4ea0-89a8-de53b92b99c8 > 87.61 GB 256 16.7% c3ea4026-551b-4a14-a346-480e8c1fe283 > 101.02 GB 256 16.7% df7ba879-74ad-400b-b371-91b45dcbed37 > 172.44 GB 256 25.0% 78192d73-be0b-4d49-a129-9bec0770efed > 108.5 GB 256 16.7% 9889280a-1433-439e-bb84-6b7e7f44d761 > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira