Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 42462 invoked from network); 28 Dec 2010 23:02:25 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 28 Dec 2010 23:02:25 -0000 Received: (qmail 57184 invoked by uid 500); 28 Dec 2010 23:02:19 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 57000 invoked by uid 500); 28 Dec 2010 23:02:19 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 56941 invoked by uid 99); 28 Dec 2010 23:02:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 23:02:19 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 23:02:13 +0000 Received: by vws18 with SMTP id 18so3735726vws.35 for ; Tue, 28 Dec 2010 15:01:52 -0800 (PST) Received: by 10.220.194.69 with SMTP id dx5mr4720514vcb.60.1293577311931; Tue, 28 Dec 2010 15:01:51 -0800 (PST) MIME-Version: 1.0 Received: by 10.220.190.75 with HTTP; Tue, 28 Dec 2010 15:01:31 -0800 (PST) X-Originating-IP: [67.160.196.149] In-Reply-To: References: <11DE171E-F448-45E4-9CC0-3901ECA0BECA@101tec.com> From: Ted Dunning Date: Tue, 28 Dec 2010 15:01:31 -0800 Message-ID: Subject: Re: Hadoop RPC call response post processing To: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=90e6ba53a9f42acd2c0498807097 --90e6ba53a9f42acd2c0498807097 Content-Type: text/plain; charset=ISO-8859-1 Knowing the tenuring distribution will tell a lot about that exact issue. Ephemeral collections take on average less than one instruction per allocation and the allocation itself is generally only a single instruction. For ephemeral garbage, it is extremely unlikely that you can beat that. So the real question is whether you are actually creating so much garbage that you are over-whelming the collector or whether the data is much longer lived than it should be. *That* can cause lots of collection costs. To tell how long data lives, you need to get the tenuring distribution: -XX:+PrintTenuringDistribution Prints details about the tenuring distribution to standard out. It can be used to show this threshold and the ages of objects in the new generation. It is also useful for observing the lifetime distribution of an application. On Tue, Dec 28, 2010 at 11:59 AM, Stefan Groschupf wrote: > I don't think the problem is allocation but garbage collection. > --90e6ba53a9f42acd2c0498807097--