Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 204.16.106.233 is neither permitted
 nor denied by domain of jeastman@collab.net)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Best Practice?
Date: Mon, 11 Feb 2008 12:24:50 -0800
Message-ID: 
 <A5A01492DDFB8040B5E0D81EDED565825F9414@SP-EXCHMBC.sp.corp.collab.net>
In-Reply-To: <653CA365-A5C8-4DF0-A0BE-297DB16BE69A@yahoo-inc.com>
Thread-Topic: Best Practice?
Thread-Index: Achs3aSAXYPdyRTGThqr1kx5jr5wKwADhAfQ
References: 
 <A5A01492DDFB8040B5E0D81EDED565825F940D@SP-EXCHMBC.sp.corp.collab.net>
 <47AE3A85.7010801@cs.washington.edu>
 <A5A01492DDFB8040B5E0D81EDED565825F940E@SP-EXCHMBC.sp.corp.collab.net>
 <653CA365-A5C8-4DF0-A0BE-297DB16BE69A@yahoo-inc.com>
From: "Jeff Eastman" <jeastman@collab.net>
To: <core-user@hadoop.apache.org>

Hi Owen,

Thanks for the information. I took Ted's advice and refactored my mapper
so as to use a combiner and that solved my front-end canopy generation
problem, but I still have to output the final canopies in the reducer
during close() since there is no similar combiner mechanism. I was
worried about this, but now I won't.

Thanks,
Jeff

-----Original Message-----
From: Owen O'Malley [mailto:oom@yahoo-inc.com]=20
Sent: Monday, February 11, 2008 10:40 AM
To: core-user@hadoop.apache.org
Subject: Re: Best Practice?


On Feb 9, 2008, at 4:21 PM, Jeff Eastman wrote:

> I'm trying to wait until close() to output the cluster centroids to =20
> the
> reducer, but the OutputCollector is not available.

You hit on exactly the right solution. Actually, because of Pipes and =20
Streaming, you have a lot more guarantees than you would expect. In =20
particular, you can call output.collect when the framework is between =20
calls to map or reduce up until the close finishes.

-- Owen