Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2F755D74A for ; Wed, 19 Sep 2012 23:21:28 +0000 (UTC) Received: (qmail 34049 invoked by uid 500); 19 Sep 2012 23:21:23 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 33968 invoked by uid 500); 19 Sep 2012 23:21:23 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 33961 invoked by uid 99); 19 Sep 2012 23:21:23 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Sep 2012 23:21:23 +0000 Received: from localhost (HELO mail-ie0-f176.google.com) (127.0.0.1) (smtp-auth username cdouglas, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Sep 2012 23:21:23 +0000 Received: by ieak12 with SMTP id k12so659515iea.35 for ; Wed, 19 Sep 2012 16:21:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.60.13.41 with SMTP id e9mr20739oec.15.1348096882455; Wed, 19 Sep 2012 16:21:22 -0700 (PDT) Received: by 10.182.213.69 with HTTP; Wed, 19 Sep 2012 16:21:21 -0700 (PDT) In-Reply-To: References: Date: Wed, 19 Sep 2012 16:21:21 -0700 Message-ID: Subject: Re: How does map-merge work exactly? From: Chris Douglas To: user@hadoop.apache.org Content-Type: text/plain; charset=ISO-8859-1 On Tue, Sep 18, 2012 at 7:02 AM, Martin Dobmeier wrote: > Ah, alright. But why is Hadoop telling me that there are 117 segments given > that only 96 reducers have been configured? > (btw, I'm using Hadoop 1.0.0) There were 117 spills, so the merger starts with 117 files, does an intermediate merge of 54 segments (#reducers = 96 times), then a final merge of 64 segments (96 times). All of those layers produce log statements. > So the merger is called "number of reducers" times because it combines the > data for a particular reducer which is spread over all spill files, right? Yup, you have it. -C