Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jssarma@facebook.com
 designates 204.15.23.140 as permitted sender)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: missing combiner output
Date: Tue, 21 Aug 2007 18:06:10 -0700
Message-ID: <FCDAD498A048A6428B2255D5D4587D3755833D@sf2pmxb01.thefacebook.com>
In-Reply-To: 
 <FCDAD498A048A6428B2255D5D4587D37558326@sf2pmxb01.thefacebook.com>
Thread-Topic: missing combiner output
Thread-Index: AcfkU5mUxz3MSVjrTAGGlx4j336JXQABOblg
From: "Joydeep Sen Sarma" <jssarma@facebook.com>
To: <hadoop-user@lucene.apache.org>

Ah - never mind - the 'combiner output record' metric reported by mapred
is lying. The reduce job does see all the records.

(I guess this is a bug)

-----Original Message-----
From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]=20
Sent: Tuesday, August 21, 2007 5:30 PM
To: hadoop-user@lucene.apache.org
Subject: missing combiner output

Hi folks,

=20

I am a little puzzled by (what looks to me) is like records that I am
emitting from my combiner - but that are not showing up under 'combine
output records' (and seem to be disappearing). Here's some evidence:

=20

Mapred says:

=20

Combine input records 230,803,567=20

Combine output records 112,533,683

=20

i am maintaining three counters and bump one of them when emitting
records from the combiner (ie. The combiner emits three types of key-val
pairs):

=20

COMBINERJOIN 28,264,088

COMBINERPASS 199,193,336

COMBINERKEYS 3,346,143

=20

as can be seen - the total number of combiner outputs (sum of above
three counters) is the same as the combine input records - and that is
exactly what I expect from my program. However, something is going wrong
somewhere and all the emitted records don't show up in the combiner
output. There are no exceptions in the logs. And the output.collect()
interface does not return an error code.

=20

Any ideas what's going on? Is this a pathogenic case (combiner emitting
same number of output records as input records)

=20

Thanks,

=20

Joydeep