hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thibaut_ <tbr...@blue.lu>
Subject Running procedure after last row has been processed in reducer
Date Fri, 14 Aug 2009 15:54:03 GMT


As my reducer class is processing data in threads, I need to run a teardown
procedure when the last element was reached (to wait until the threads have
finished writing to the output collector).

The close function provided by the interface doesn't work, as when called,
all output files are allready closed and collecting data will fail. (Already
too late in code)

For Map jobs, there is a runner class which i can set
(conf.setMapRunnerClass()) which I can modifiy then to execute the teardown.
However there is no such setter for the Reducer.

Is there a possibility to check when the reduce function is called that this
is indeed the last time the reduce function is called? (I run hadoop in non
streaming mode).


View this message in context: http://www.nabble.com/Running-procedure-after-last-row-has-been-processed-in-reducer-tp24974027p24974027.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

View raw message