Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4192695BA for ; Fri, 13 Jan 2012 13:33:59 +0000 (UTC) Received: (qmail 80205 invoked by uid 500); 13 Jan 2012 13:33:57 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 79897 invoked by uid 500); 13 Jan 2012 13:33:55 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 79802 invoked by uid 99); 13 Jan 2012 13:33:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2012 13:33:54 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of baishen.lists@gmail.com designates 74.125.82.48 as permitted sender) Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2012 13:33:47 +0000 Received: by wgbds1 with SMTP id ds1so3279879wgb.29 for ; Fri, 13 Jan 2012 05:33:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=TsKbs7l+7Gl11T9ORuatJBtqir6rQimI9goUu3UsKr0=; b=icnLkWCGGuqXyXTa9yly/jEPrarziwMjjntSBN/su49vJu9rFVkVTL2SXtilQdwXwT INW3dFjJg4EC1clhIjtfvXLFpUWlPrgoPDai+InSvb2K3vEVdcJYNpOIW9zpimkx5YS2 AHwmuterQI+6RMULwPhHRMx0UZ/CWa80XzJxk= MIME-Version: 1.0 Received: by 10.180.93.132 with SMTP id cu4mr8946985wib.9.1326461606926; Fri, 13 Jan 2012 05:33:26 -0800 (PST) Received: by 10.180.20.72 with HTTP; Fri, 13 Jan 2012 05:33:26 -0800 (PST) In-Reply-To: References: Date: Fri, 13 Jan 2012 08:33:26 -0500 Message-ID: Subject: Re: Hadoop map reduce merge algorithm From: Bai Shen To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d043c80b8e39a7c04b668e884 X-Virus-Checked: Checked by ClamAV on apache.org --f46d043c80b8e39a7c04b668e884 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable As far as I can tell, the amount of ram available has no effect on the merge. Regardless of what you set the io.sort.mb and io.sort.factor to, it will eventually end up attempting to bring the entire output into memory to merge. If the mb and factor are set low, it will simply require more passes. Is there a way to actually configure the amount of ram used by the merge? On Thu, Jan 12, 2012 at 11:14 PM, Ravi Gummadi wrote: > Yes. Spills of map output get merged to single file. The spills are > triggered by the buffer size set using the configuration property > io.sort.mb. Obviously bigger value for io.sort.mb is preferred for better > performance --- but the limit is to be set based on the amount of RAM > available. > Also, the bigger the value for the configuration property io.sort.factor > the better in terms of performance. Even in this case, smaller value may > have to be set for this config property based on the size of RAM availabl= e. > > -Ravi > > > On 1/13/12 Friday 3:12 AM, "Bai Shen" wrote: > > That's my understanding as well. I can't seem to find any settings that > govern the step where the output is merged into a single file. > io.sort.factor modifies the number of passes that is done, but it > eventually ends up doing the same thing no matter how many spill files > there are. They're simply combined incrementally instead of all at once. > > Is anybody more familiar with this step of the process? > > Thanks. > > On Thu, Jan 12, 2012 at 2:27 PM, Robert Evans wrote= : > > My understanding is that the mapper will cache the output in memory until > its memory buffer fills up, at which point it will sort the data and spil= l > it to disk. Once a given number of spill files are created they will be > merged together into a larger spill file. Once the mapper finishes then > the output is totally merged into a single file that can be served to the > Reducer through the TaskTracker, or NodeManger under YARN. The reducer > does a similar thing as it merges the output form all of the mappers. I > don=92t understand all of the reasons behind this, but I think much of it= is > to optimize the time it takes to sort the data. If you try to merge too > many files then you waste a lot of time doing seeks and less time reading > data. But I was not involved with developing it so I don=92t know for su= re. > > --Bobby Evans > > > On 1/12/12 10:27 AM, "Bai Shen" http://baishen.lists@gmail.com> > wrote: > > Can someone explain how the map reduce merge is done? As far as I can > tell, it appears to pull all of the spill files into one giant file to se= nd > to the reducer. Is this correct? Even if you set smaller spill files an= d > a lower sort factor, the eventual merge is still the same. It just takes > more passes to get there. > > Thanks. > > > > --f46d043c80b8e39a7c04b668e884 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable As far as I can tell, the amount of ram available has no effect on the merg= e.=A0 Regardless of what you set the io.sort.mb and io.sort.factor to, it w= ill eventually end up attempting to bring the entire output into memory to = merge.=A0 If the mb and factor are set low, it will simply require more pas= ses.

Is there a way to actually configure the amount of ram used by the merg= e?

On Thu, Jan 12, 2012 at 11:14 PM, Ravi= Gummadi <grav= i@yahoo-inc.com> wrote:
Yes. Spills of map output get merged to single file. The spills are t= riggered by the buffer size set using the configuration property io.sort.mb= . Obviously bigger value for io.sort.mb is preferred for better performance= --- but the limit is to be set based on the amount of RAM available.
Also, the bigger the value for the configuration property io.sort.factor th= e better in terms of performance. Even in this case, smaller value may have= to be set for this config property based on the size of RAM available.

-Ravi


On 1/13/12 =A0=A0Friday 3:12 AM, "Bai Shen" <baishen.lists@gmail.com>= wrote:

That's my under= standing as well.=A0 I can't seem to find any settings that govern the = step where the output is merged into a single file.=A0 io.sort.factor modif= ies the number of passes that is done, but it eventually ends up doing the = same thing no matter how many spill files there are.=A0 They're simply = combined incrementally instead of all at once.

Is anybody more familiar with this step of the process?

Thanks.

On Thu, Jan 12, 2012 at 2:27 PM, Robert Evans <evans@yahoo-inc.com> wrote:
My understanding is= that the mapper will cache the output in memory until its memory buffer fi= lls up, at which point it will sort the data and spill it to disk. =A0Once = a given number of spill files are created they will be merged together into= a larger spill file. =A0Once the mapper finishes then the output is totall= y merged into a single file that can be served to the Reducer through the T= askTracker, or NodeManger under YARN. =A0The reducer does a similar thing a= s it merges the output form all of the mappers. =A0I don=92t understand all= of the reasons behind this, but I think much of it is to optimize the time= it takes to sort the data. =A0If you try to merge too many files then you = waste a lot of time doing seeks and less time reading data. =A0But I was no= t involved with developing it so I don=92t know for sure.

--Bobby Evans


On 1/12/12 10:27 AM, "Bai Shen" <baishen.lists@gmail.com <http://baishen.lists@gmai= l.com> > wrote:

Can someone explain= how the map reduce merge is done?=A0 As far as I can tell, it appears to p= ull all of the spill files into one giant file to send to the reducer.=A0 I= s this correct?=A0 Even if you set smaller spill files and a lower sort fac= tor, the eventual merge is still the same.=A0 It just takes more passes to = get there.

Thanks.




--f46d043c80b8e39a7c04b668e884--