hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rasit OZDAS <rasitoz...@gmail.com>
Subject Re: Strange Reduce Bahavior
Date Thu, 02 Apr 2009 13:29:01 GMT
Yes, we've constructed a local version of a hadoop process,
We needed 500 input files in hadoop to reach the speed of local process,
total time was 82 seconds in a cluster of 6 machines.
And I think it's a good performance among other distributed processing

2009/4/2 jason hadoop <jason.hadoop@gmail.com>

> 3) The framework is designed for working on large clusters of machines
> where
> there needs to be a little delay between operations to avoid massive
> network
> loading spikes, and the initial setup of the map task execution environment
> on a machine, and the initial setup of the reduce task execution
> environment
> take a bit of time.
> In production jobs, these delays and setup times are lost in the overall
> task run time.
> In the small test job case the delays and setup times will be the bulk of
> the time spent executing the test.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message