Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8F1D34919 for ; Fri, 13 May 2011 15:47:15 +0000 (UTC) Received: (qmail 56256 invoked by uid 500); 13 May 2011 15:47:13 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 56233 invoked by uid 500); 13 May 2011 15:47:13 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 56225 invoked by uid 99); 13 May 2011 15:47:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 15:47:13 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [216.32.181.186] (HELO ch1outboundpool.messaging.microsoft.com) (216.32.181.186) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 15:47:03 +0000 Received: from mail87-ch1-R.bigfish.com (216.32.181.170) by CH1EHSOBE015.bigfish.com (10.43.70.65) with Microsoft SMTP Server id 14.1.225.8; Fri, 13 May 2011 15:46:41 +0000 Received: from mail87-ch1 (localhost.localdomain [127.0.0.1]) by mail87-ch1-R.bigfish.com (Postfix) with ESMTP id A9E50F2825B for ; Fri, 13 May 2011 15:46:41 +0000 (UTC) X-SpamScore: -6 X-BigFish: VPS-6(zz1432Nzz1202hzzz2dh2a8h668h839h61h) X-Spam-TCS-SCL: 0:0 X-Forefront-Antispam-Report: KIP:(null);UIP:(null);IPVD:NLI;H:VA3DIAHUB001.RED001.local;RD:smtp801.microsoftonline.com;EFVD:NLI Received: from mail87-ch1 (localhost.localdomain [127.0.0.1]) by mail87-ch1 (MessageSwitch) id 1305301601597980_27039; Fri, 13 May 2011 15:46:41 +0000 (UTC) Received: from CH1EHSMHS015.bigfish.com (snatpool1.int.messaging.microsoft.com [10.43.68.248]) by mail87-ch1.bigfish.com (Postfix) with ESMTP id 852D496004B for ; Fri, 13 May 2011 15:46:41 +0000 (UTC) Received: from VA3DIAHUB001.RED001.local (65.55.171.153) by CH1EHSMHS015.bigfish.com (10.43.70.15) with Microsoft SMTP Server (TLS) id 14.1.225.8; Fri, 13 May 2011 15:46:39 +0000 Received: from [192.168.1.153] (69.70.138.86) by smtp.mail.microsoftonline.com (10.32.16.40) with Microsoft SMTP Server (TLS) id 8.3.83.0; Fri, 13 May 2011 08:46:38 -0700 Message-ID: <4DCD525C.2040904@wajam.com> Date: Fri, 13 May 2011 11:46:36 -0400 From: Gabriel Tataranu User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10 MIME-Version: 1.0 To: "user@cassandra.apache.org" Subject: Re: Excessive allocation during hinted handoff References: <4DCADB98.1060600@wajam.com> <3FC5A5DD-EB3E-405D-AEB7-675616D66D52@thelastpickle.com> <4DCBEC0B.5050309@wajam.com> <4DCC01EF.4060904@wajam.com> <69F92198-4EAF-4808-B9FC-9FD13EC4B75C@thelastpickle.com> In-Reply-To: <69F92198-4EAF-4808-B9FC-9FD13EC4B75C@thelastpickle.com> Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit X-OriginatorOrg: wajam.com X-Virus-Checked: Checked by ClamAV on apache.org > The number of Completed HH tasks is interesting. AFAIK a task is started when the node detects another in the cluster has returned. Were you doing some other restarts around the cluster ? Not at all. The restarts seem to happen as normal operation. > > I don't want to divert from the GC issue, just wondering if something else is going on as well. Like the node is been asked to record a lot of hints. NP, this is another issue that a bit weird. Nodes goes MIA and then return few seconds latter. Not sure why. In other news, I've discovered than one of the nodes had some corruption in one of the SSTables - some 1TB record. I'm looking into cleaning up the data and monitoring the nodes. Gabriel