Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5254018E63 for ; Mon, 29 Jun 2015 03:52:59 +0000 (UTC) Received: (qmail 19486 invoked by uid 500); 29 Jun 2015 03:52:54 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 19377 invoked by uid 500); 29 Jun 2015 03:52:54 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 19366 invoked by uid 99); 29 Jun 2015 03:52:54 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jun 2015 03:52:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A0931D0DBE for ; Mon, 29 Jun 2015 03:52:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.879 X-Spam-Level: *** X-Spam-Status: No, score=3.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id SBmumpUD2mL6 for ; Mon, 29 Jun 2015 03:52:52 +0000 (UTC) Received: from mail-la0-f46.google.com (mail-la0-f46.google.com [209.85.215.46]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 47C22214D8 for ; Mon, 29 Jun 2015 03:52:52 +0000 (UTC) Received: by laar3 with SMTP id r3so48216247laa.0 for ; Sun, 28 Jun 2015 20:52:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=1hAtG6pNl4AP0keB63otMBCH6c4GURyH5/Xvs7GmpJE=; b=qy8Pa6elUllupzX6Sce8GEGuFmYXADwb4u7J/Ge2KGosd1//3UseviTmCZKfIXw6dZ TcCx/9KNv/J9Xb0FGQbLvlMakzUnBP4QLayxJmytZM6+DprkGBVdmqED6L3X+iXl9UPz xFO+G/aT7GtAe1bzZEVNs8wrp/F93MezGQrKC1aCNco7W2Lud28N7+/4lvr7evF+xzBF +W+BJMXsZ8FGOWoBXxIWY41HgQ4qsZj7n+tyquIuaAhYBreSftvbcXtoZnyDiOB2kC3P v3Gf6hWoVLTS5EZXlHTTK5CxUfjJ/69bW5l0JHhmokvXWNbuSZ04UM8NaJ+S6Vh1R92D DKbQ== MIME-Version: 1.0 X-Received: by 10.152.115.199 with SMTP id jq7mr12088798lab.113.1435549920325; Sun, 28 Jun 2015 20:52:00 -0700 (PDT) Received: by 10.112.158.167 with HTTP; Sun, 28 Jun 2015 20:52:00 -0700 (PDT) In-Reply-To: References: Date: Mon, 29 Jun 2015 09:22:00 +0530 Message-ID: Subject: Re: Reducer called twice for same key From: Harshit Mathur To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11c2588e0ef76f0519a00367 --001a11c2588e0ef76f0519a00367 Content-Type: text/plain; charset=UTF-8 As per Map Reduce, it is not possible that two different reducers will get same keys. I think you have created some custom key type? If that is the case then there should be some issue with the comparator. On Mon, Jun 29, 2015 at 12:40 AM, Ravikant Dindokar wrote: > Hi Hadoop user, > > I have two map classes processing two different input files. Both map > functions have same key,value format to emit. > > But Reducer called twice for same key , one for value from first map while > one for value from other map. > > I am printing (key ,value) pairs in reducer : > ./container_1435326857837_0036_01_000102/stdout:Reduce:391:-1#11 > > ./container_1435326857837_0036_01_000102/stdout:Reduce:391:3278620528725786624:5352454#-1 > > ./container_1435326857837_0036_01_000102/stdout:Reduce:591:3278620528725852160:4194699#-1 > ./container_1435326857837_0036_01_000102/stdout:Reduce:591:-1#13 > ./container_1435326857837_0036_01_000102/stdout:Reduce:2391:-1#19 > > ./container_1435326857837_0036_01_000102/stdout:Reduce:2391:3278620528725917696:5283986#-1 > > ./container_1435326857837_0036_01_000102/stdout:Reduce:3291:3278620528725983232:4973087#-1 > > both maps emit Longwritable key and Text value. > > > Any idea why this is happening? > Is there any way to get hash values generated by hadoop for keys emitted > by mapper? > > Thanks > Ravikant > -- Harshit Mathur --001a11c2588e0ef76f0519a00367 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
As per Map Reduce, it is not possible that two different r= educers will get same keys.
I think you have created some custom key ty= pe? If that is the case then there should be some issue with the comparator= .

On Mon= , Jun 29, 2015 at 12:40 AM, Ravikant Dindokar <ravikant.iisc@gmail.c= om> wrote:
Hi Hadoop user,

I have two map class= es processing two different input files. Both map functions have same key,v= alue format to emit.

But Reducer called twice for same key , o= ne for value from first map while one for value from other map.

I am printing (key ,value) pairs in reducer=C2=A0 :
./container_14353= 26857837_0036_01_000102/stdout:Reduce:391:-1#11
./container_143532685783= 7_0036_01_000102/stdout:Reduce:391:3278620528725786624:5352454#-1
./cont= ainer_1435326857837_0036_01_000102/stdout:Reduce:591:3278620528725852160:41= 94699#-1
./container_1435326857837_0036_01_000102/stdout:Reduce:591:-1#1= 3
./container_1435326857837_0036_01_000102/stdout:Reduce:2391:-1#19
.= /container_1435326857837_0036_01_000102/stdout:Reduce:2391:3278620528725917= 696:5283986#-1
./container_1435326857837_0036_01_000102/stdout:Reduce:32= 91:3278620528725983232:4973087#-1

both maps emit Longwritable = key and Text value.


Any idea why this is happening?<= br>
Is there any way to get hash values generated by hadoop for k= eys emitted by mapper?

Thanks
Ravikant



--
Harshit Mathur
--001a11c2588e0ef76f0519a00367--