hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Hadoop Mapper Intermediate Result Storage with No Reducer
Date Thu, 26 Apr 2012 06:12:48 GMT
A map-only job does not write to local disk, but rather directly to
the configured FileSystem the job's running on (HDFS usually).

It has always been so as far as I know, as map-only jobs do not do any
form of sorting/partitioning and hence do not require transient

On Thu, Apr 26, 2012 at 2:19 AM, Xun TANG <tangxun.alice@gmail.com> wrote:
> Hi,
> We are running Hadoop jobs with  mappers only, no reducers.
> After each map computation, it writes its result (i.e. intermediate
> results) to local disk. How fast is this speed, compared with write-speed
> to HDFS?
> We used TestFDSIO to do the benchmark to get write-speed to HDFS, but we
> wonder how much the gap is.
> Another question is, when did Hadoop move this chuck to HDFS?
> Any thoughts, guys? Thanks ahead.
> Xun

Harsh J

View raw message