Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 42304 invoked from network); 18 Mar 2009 05:21:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 18 Mar 2009 05:21:07 -0000 Received: (qmail 88569 invoked by uid 500); 18 Mar 2009 05:21:01 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 88524 invoked by uid 500); 18 Mar 2009 05:21:01 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 88505 invoked by uid 99); 18 Mar 2009 05:21:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Mar 2009 22:21:01 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [76.96.62.80] (HELO QMTA08.westchester.pa.mail.comcast.net) (76.96.62.80) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Mar 2009 05:20:52 +0000 Received: from OMTA11.westchester.pa.mail.comcast.net ([76.96.62.36]) by QMTA08.westchester.pa.mail.comcast.net with comcast id UVEQ1b0040mv7h058VLYxP; Wed, 18 Mar 2009 05:20:32 +0000 Received: from [10.0.0.13] ([209.131.62.115]) by OMTA11.westchester.pa.mail.comcast.net with comcast id UVLM1b0052VBGtd3XVLRUs; Wed, 18 Mar 2009 05:20:30 +0000 Message-Id: From: Owen O'Malley To: core-user@hadoop.apache.org In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Subject: Re: Massive discrepancies in job's bytes written/read Date: Tue, 17 Mar 2009 22:20:19 -0700 References: X-Mailer: Apple Mail (2.930.3) X-Virus-Checked: Checked by ClamAV on apache.org On Mar 17, 2009, at 7:44 PM, Bryan Duxbury wrote: > There is no compression in the mix for us, so that's not the culprit. > > I'd be sort of willing to believe that spilling and sorting play a > role in this, but, wow, over 10x read and write? That seems like a > big problem. It happened recently to me too. It was off by 6x. The strange thing was that the individual tasks looked right. It was just the aggregate that was wrong. -- Owen