hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball" <aa...@cloudera.com>
Subject Re: End of block/file for Map
Date Wed, 10 Dec 2008 03:34:41 GMT
That's true, but you should be aware that you no longer have an
OutputCollector available in the close() method.  So if you are planning to
have each mapper emit some sort of "end" record along to the reducer, you
can't do so there. In general, there is not a good solution to that; you
should rethink your algorithm if possible so that you don't need to do that.

(I am not sure what happens if you memoize the OutputCollector you got as a
parameter to your map() method and try to use it. Probably nothing good.)

- Aaron

On Tue, Dec 9, 2008 at 11:42 AM, Owen O'Malley <omalley@apache.org> wrote:

> On Dec 9, 2008, at 11:35 AM, Songting Chen wrote:
>  Is there a way for the Map process to know it's the end of records?
>> I need to flush some additional data at the end of the Map process, but
>> wondering where I should put that code.
> The close() method is called at the end of the map.
> -- Owen

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message