Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8C84810589 for ; Mon, 28 Oct 2013 14:16:15 +0000 (UTC) Received: (qmail 85053 invoked by uid 500); 28 Oct 2013 14:16:08 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 84592 invoked by uid 500); 28 Oct 2013 14:16:07 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 84498 invoked by uid 99); 28 Oct 2013 14:16:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Oct 2013 14:16:06 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [72.30.239.77] (HELO nm34-vm5.bullet.mail.bf1.yahoo.com) (72.30.239.77) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Oct 2013 14:15:58 +0000 Received: from [98.139.212.147] by nm34.bullet.mail.bf1.yahoo.com with NNFMP; 28 Oct 2013 14:15:36 -0000 Received: from [98.139.212.213] by tm4.bullet.mail.bf1.yahoo.com with NNFMP; 28 Oct 2013 14:15:36 -0000 Received: from [127.0.0.1] by omp1022.mail.bf1.yahoo.com with NNFMP; 28 Oct 2013 14:15:36 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 776792.4779.bm@omp1022.mail.bf1.yahoo.com Received: (qmail 94494 invoked by uid 60001); 28 Oct 2013 14:15:36 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ymail.com; s=s1024; t=1382969736; bh=6nsw+RPmdN35cKkiCqln0OOnVy8L3ZazehCR1XuKZco=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=z7VdGKRKsFXvmeJryLUMqmpqvU1wo1eSwAYO2K4opkq2tb6I+F8vess+Jh3+DBxDNQsxzUWacTajJrSzs5KkHYPK/V0GnQDERret6C17AF5L6mtD7UyA8tf4JkAMNTV+ljGhj/yPGWTKeeW6+AF3wVb8E5Jhh5hZXDMhJxrUVnU= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=ymail.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=fFCkQZ7hpZkRT9KJQDAkn20H6EqnJRJ3dDq3vpLuTiK2iJ8k7okyP7r6AuRS+Yb1z+gzgQjNWcXHayH/v2NVB8FRO/rmZJGh7fbdPnre8cF0vz0iXEE4/KJZyznLZ12v7JI40BAcZ44byJvJWG+sQMKsDnSyvZOBKe5kbFv/Qzc=; X-YMail-OSG: RDvANtQVM1lJ3u6B2.xOnNo.ZR2JpfrBpYBWreEQ8zgzNT7 NP4FU8Mogjc7E4XZ6fVTCCjFGHAX0VOyWexBJkqjHGIdldvXCGxOnxgsgt.y pfcuWnhVAhkP5lpwESJ97Nj6IMGJNbm.t6fUKFwJLoQgOUh9no.1Gr39Yyjr o5ut._i4M0j9uXQ8tQJhit9U5GgFp1Ki8bPuWX8WE9zTZBRVVN69VtkH8LYT XsFPLCDzkIjaH1Ko07_JwyGQeXFtVDDUzN1WA3K5CDWEUALeUEJfyMb2auYs zjD5cntgjmIVIl1dvrYzAWKC5l9NOzJg_phOMt97xoQjXPcYKt58pYsWgyq0 QhUM58xgZZ8i_qk..O30whHeCz.f2RWhlHMCpd4cX3uc70s_AMvEh8sR9yuV RX786IVzfRKWGXzCJQK0UbAjIwA26eR.oQVUa6fz6Lya_2IKce9ZpyIqTS2B tTlYPmhESCCOokOmUPKZwTj4EhiyT9cXlGqXH6IUJiztkHO3BO.x6w7crKOa TJEtG9SXVWs_H4C3i1fwnGM8CFobzGnRg8r7N5aRAivSYl5qX1RGmcpfA57I Fm9rYDnZrr38fzTGdjBZvfKeD3AeO0_WgCp_IZ6L13UYP0yaXcoT5MX0HVUk 61BhBs6wsRUEuowq9IQ_XvCl24fQB2MZh8Qj770d08B3Yr99sM4HY2O4koF2 IQ7tRqMJw_nxdQgpEPFyyxVmPo_TkTfzdG7ZulSGrdqp_bpWO.V9TYudjHiQ X2rHam_QN Received: from [216.145.54.50] by web141201.mail.bf1.yahoo.com via HTTP; Mon, 28 Oct 2013 07:15:36 PDT X-Rocket-MIMEInfo: 002.001,SGkhCgpUb20gV2hpdGUncyAiSGFkb29wOiBUaGUgRGVmaW5pdGl2ZSBHdWlkZSIgaXMgcHJvYmFibHkgdGhlIGJlc3Qgc291cmNlIGZvciBpbmZvcm1hdGlvbiBvbiB0aGlzIChhcGFydCBmcm9tIHRoZSBjb2RlIGl0c2VsZiA7LSkgTG9vayBhdCBNZXJnZU1hbmFnZXJJbXBsLmphdmEgYnR3IGluIGNhc2UgeW91IGFyZSBzbyBpbmNsaW5lZCkuCgpIVEgKUmF2acKgwqAgCgoKCk9uIEZyaWRheSwgT2N0b2JlciAyNSwgMjAxMyAyOjM2IFBNLCAtIDxjb21tb2RvcmU2NUB5bWFpbC5jb20.IHdyb3RlOgogCkhpIEEBMAEBAQE- X-Mailer: YahooMailWebService/0.8.160.587 References: <1382729740.96642.YahooMailNeo@web121502.mail.ne1.yahoo.com> Message-ID: <1382969736.90696.YahooMailNeo@web141201.mail.bf1.yahoo.com> Date: Mon, 28 Oct 2013 07:15:36 -0700 (PDT) From: Ravi Prakash Reply-To: Ravi Prakash Subject: Re: How / when does On-disk merge work? To: "user@hadoop.apache.org" , - In-Reply-To: <1382729740.96642.YahooMailNeo@web121502.mail.ne1.yahoo.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="1105474026-986835039-1382969736=:90696" X-Virus-Checked: Checked by ClamAV on apache.org --1105474026-986835039-1382969736=:90696 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hi!=0A=0ATom White's "Hadoop: The Definitive Guide" is probably the best so= urce for information on this (apart from the code itself ;-) Look at MergeM= anagerImpl.java btw in case you are so inclined).=0A=0AHTH=0ARavi=A0=A0 =0A= =0A=0A=0AOn Friday, October 25, 2013 2:36 PM, - wro= te:=0A =0AHi All,=0A=0ACan anyone provide documentation regarding how on-di= sk merge on reduce phase works in detail in Hadoop 2.2.0?=0AThere is an exp= lanation in this page but I am afraid it could be outdated since what I obs= erve in my log files is a bunch of "OnDiskMerger - Thread to merge on-disk = map-outputs" work at the end of merge phase.=0A=0AThanks,=0A- --1105474026-986835039-1382969736=:90696 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Hi!

= Tom White's "Hadoop: The Definitive Guide" is probably the bes= t source for information on this (apart from the code itself ;-) Look at Me= rgeManagerImpl.java btw in case you are so inclined).

HTH
Ravi  =  
=

On Friday, October 25, 2013 2:36 PM, - <commodore65@ymail.com> w= rote:
Hi All,

Can any= one provide documentation regarding how on-disk merge on reduce phase works= in detail in Hadoop 2.2.0?
Ther= e is an explanation in th= is page but I am afraid it could be outdated since what I observe in my= log files is a bunch of "OnDiskMerger - Thread to merge o= n-disk map-outputs" work at the end of merge phase.

Thanks,
-

--1105474026-986835039-1382969736=:90696--