Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4B78710C3D for ; Mon, 6 Jan 2014 19:07:41 +0000 (UTC) Received: (qmail 64418 invoked by uid 500); 6 Jan 2014 19:07:36 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 64256 invoked by uid 500); 6 Jan 2014 19:07:36 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 64247 invoked by uid 99); 6 Jan 2014 19:07:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jan 2014 19:07:36 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nagarjuna.kanamarlapudi@gmail.com designates 209.85.220.42 as permitted sender) Received: from [209.85.220.42] (HELO mail-pa0-f42.google.com) (209.85.220.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jan 2014 19:07:30 +0000 Received: by mail-pa0-f42.google.com with SMTP id lj1so19029168pab.29 for ; Mon, 06 Jan 2014 11:07:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=IlBz9eGpllMG4ePDbFh3tUUNrWdbOjnHpthW3RLjb/E=; b=zB2iXE0X9XdruaiywBQVMMBigbdzWlGvo4foytGX6jb+jUDJ/Wn1Cm8HbPG1bkVR/S A3eOx6L/07N5JVGrd+f1X9vUaBemL4GSj0LVBvDyyPFxzBy2AWNhvvQzJ0X9LK3oIND+ yvbV7W5o7WCPKQuduCjpiqMrDXvpn2DmLOfMu0SR8XHSbeTNi8A33tD3/KGKRYuC5qZa 4TNZaP27AqZD/12aScpW/VX8Jmopje3w0iWlH4QZv6lKfD5taBM45/huVg+MnNm5pf53 dVj3briI/T9bLvFVsWob+YrZc09FpT1kSRqg3z1OTLKBivoOJg3NfMuAMpiFvPvNt1gC Qk9w== MIME-Version: 1.0 X-Received: by 10.66.26.106 with SMTP id k10mr46618pag.136.1389035230687; Mon, 06 Jan 2014 11:07:10 -0800 (PST) Received: by 10.70.8.69 with HTTP; Mon, 6 Jan 2014 11:07:10 -0800 (PST) In-Reply-To: References: Date: Tue, 7 Jan 2014 00:37:10 +0530 Message-ID: Subject: Re: Understanding MapReduce source code : Flush operations From: nagarjuna kanamarlapudi To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=bcaec520f2e381793504ef51f74d X-Virus-Checked: Checked by ClamAV on apache.org --bcaec520f2e381793504ef51f74d Content-Type: text/plain; charset=ISO-8859-1 I want to have a look at the code where of flush operations that happens after the reduce phase. Reducer writes the output to OutputFormat which inturn pushes that to memory and once it reaches 90% of chunk size it starts to flush the reducer output. I essentially want to look at the code of that flushing operation. What is the class(es) I need to look into On Mon, Jan 6, 2014 at 11:23 PM, Hardik Pandya wrote: > Please do not tell me since last 2.5 years you have not used virtual > Hadoop environment to debug your Map Reduce application before deploying to > Production environment > > No one can stop you looking at the code , Hadoop and its ecosystem is > open-source > > > On Mon, Jan 6, 2014 at 9:35 AM, nagarjuna kanamarlapudi < > nagarjuna.kanamarlapudi@gmail.com> wrote: > >> >> >> ---------- Forwarded message ---------- >> From: nagarjuna kanamarlapudi >> Date: Mon, Jan 6, 2014 at 6:39 PM >> Subject: Understanding MapReduce source code : Flush operations >> To: mapreduce-user@hadoop.apache.org >> >> >> Hi, >> >> I am using hadoop/ map reduce for aout 2.5 years. I want to understand >> the internals of the hadoop source code. >> >> Let me put my requirement very clear. >> >> I want to have a look at the code where of flush operations that happens >> after the reduce phase. >> >> Reducer writes the output to OutputFormat which inturn pushes that to >> memory and once it reaches 90% of chunk size it starts to flush the reducer >> output. >> >> I essentially want to look at the code of that flushing operation. >> >> >> >> >> Regards, >> Nagarjuna K >> >> > --bcaec520f2e381793504ef51f74d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I want to have a look at the code where of flush oper= ations that happens after the reduce phase.

Reducer writes the output to OutputFormat which inturn pushes that to memory=20 and once it reaches 90% of chunk size it starts to flush the reducer=20 output.

I essentially want to look at the code of that flushing operation.
<= br>
What is the class(es) I need to look into


On Mon, Jan 6, 2014 at = 11:23 PM, Hardik Pandya <smarty.juice@gmail.com> wrote:=
Please do not tell me since= last 2.5 years you have not used virtual Hadoop environment to debug your = Map Reduce application before deploying to Production environment

No one can stop you looking at the code , Hadoop and its eco= system is open-source

On Mon, Jan 6, 2014 at 9:35 AM, nagarjuna = kanamarlapudi <nagarjuna.kanamarlapudi@gmail.com> wrote:


---------- Forwarded message ----------
From: nagarjuna kanamarlapudi <nagarjuna.k= anamarlapudi@gmail.com>
Date: Mon, Jan 6, 2014 at 6:39 PM
Subject: Understanding MapReduce sourc= e code : Flush operations
To: mapreduce-user@hadoop.apache.org

Hi,

I am using hadoop/ map reduc= e for aout 2.5 years. I want to understand the internals of the hadoop sour= ce code.

Let me put my requirement very clear.

I want to have a look at the code where of flush operations that = happens after the reduce phase.

Reducer writes the output to O= utputFormat which inturn pushes that to memory and once it reaches 90% of c= hunk size it starts to flush the reducer output.

I essentially want to look at the code of that flushing operation= .




Regards,
Nagarjuna K



--bcaec520f2e381793504ef51f74d--