Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 41D55CB29 for ; Mon, 22 Dec 2014 04:51:09 +0000 (UTC) Received: (qmail 89058 invoked by uid 500); 22 Dec 2014 04:51:02 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 88942 invoked by uid 500); 22 Dec 2014 04:51:02 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 88932 invoked by uid 99); 22 Dec 2014 04:51:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Dec 2014 04:51:00 +0000 X-ASF-Spam-Status: No, hits=0.3 required=5.0 tests=FREEMAIL_REPLY,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of skgadalay@gmail.com designates 74.125.82.43 as permitted sender) Received: from [74.125.82.43] (HELO mail-wg0-f43.google.com) (74.125.82.43) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Dec 2014 04:50:55 +0000 Received: by mail-wg0-f43.google.com with SMTP id l18so5793804wgh.16 for ; Sun, 21 Dec 2014 20:50:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=7aifnSXHnS/DJrDJzJVrlNki5rNGBDPVALrKBWwdjmM=; b=FBG/RLgg/wznm22gIaeViYx4uhzU3crIYAbeF+QgXs6QwWn+5KicT/G0p8/E1UNwGT j/p8MN43B3mGTQZdMoGQAXV064OG8o+irIwXBZps+pC1iapHpwOxP6QHvhrnk1ceE5L/ 0VQ/rB4MFSOMQrN2Sbpg+2h9mDk7AYT1ORm9L8s9xzJeaLm1XAO+Ebxk990k875XMhqk NlfmfMd0ojR4WSKs51WZuSr8DLt79sI6pW61eUZpSpV5py8+NYEGdHQ4V+D0EoD6kDPo GStbg2i286d63Hfoc5sYivaUVDOWhSsWVxU1OngXJ9LWYqeUCcv3loSOj4OWng+IIH2x Z6kQ== MIME-Version: 1.0 X-Received: by 10.194.191.227 with SMTP id hb3mr38255985wjc.79.1419223834549; Sun, 21 Dec 2014 20:50:34 -0800 (PST) Received: by 10.217.191.202 with HTTP; Sun, 21 Dec 2014 20:50:34 -0800 (PST) In-Reply-To: References: <43cab251.4652.14a6d24f932.Coremail.bit1129@163.com> <201412221148321728172@163.com> Date: Mon, 22 Dec 2014 10:20:34 +0530 Message-ID: Subject: Re: Question about shuffle/merge/sort phrase From: Susheel Kumar Gadalay To: user@hadoop.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Sorry, typo It is the reducer which will pull the mapper o/p as soon as it completes. On 12/22/14, Susheel Kumar Gadalay wrote: > It is the mapper which will push the o/p to the respective reducer as > soon as it completes. > > The no of reducers are known at the beginning itself. > The mapper as it process the input split, generate the o/p of for each > reducer (if the mapper o/p key is eligible for the reducer). > The reducer will wait till the completion of all map tasks to start it > processing. > > > On 12/22/14, bit1129@163.com wrote: >> Could some one help me on this question? thanks. >> >> >> >> bit1129@163.com >> >> =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9A Todd >> =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A 2014-12-21 21:59 >> =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9A user@hadoop.apache.org >> =E4=B8=BB=E9=A2=98=EF=BC=9A Question about shuffle/merge/sort phrase >> Hi, Hadoopers, >> I got a question about shuffle/sort/merge phrase related.. >> My understanding is that shuffle is used to transfer the mapper >> output(key/value pairs) from mapper node to reducer node, and merge >> phrase >> is used to merge all the mapper output from all mapper nodes, and sort >> phrase is used to sort the key/value pair by key, >> Then my question, whose responsibility is it that brings each key with >> all >> its values together (The reducer's input is a key and an iterative >> values). >> >> >> Thanks. >> >