hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Costa <psdc1...@gmail.com>
Subject Re: FILE_BYTES_WRITTEN and HDFS_BYTES_WRITTEN
Date Wed, 01 Dec 2010 08:49:48 GMT
This is not what I ask. Maybe I haven't expressed correctly.

1 - I suppose that the map tasks are the only tasks that write to
local disks. This happens during the creation of the map output. The
FILE_BYTES_WRITTEN corresponds only to the total of number of bytes
produced by all map tasks during the creation of the map outputs or
exists another phase beside the creation of the map outputs that
contributes to the sum of the field (for example, the creation of the
directories during the setup phase)?

2 - The HDFS_BYTES_WRITTEN is the sum of bytes of the output of all
Reduce Tasks saved in HDFS?

On Tue, Nov 30, 2010 at 8:43 PM, Niels Basjes <Niels@basjes.nl> wrote:
> For some parts of a task the system stores information on the local
> (non-HDFS) file system of the node that is actually running the job.
> That is the FILE_.. Stuff written to HDFS is the HDFS_...
>
> HTH
>
> 2010/11/30  <psdc1978@gmail.com>:
>> When an hadoop MapReduce example is executed, at the end of the example it's
>> showed a table with all the information about the execution, like the number
>> of Map and Reduce tasks executed, the number of bytes read and written.
>>
>> In this information it exists 2 fields FILE_BYTES_WRITTEN and
>> HDFS_BYTES_WRITTEN. What's the difference between these 2 fields?
>>
>> Thanks,
>> Pedro
>
>
>
> --
> Met vriendelijke groeten,
>
> Niels Basjes
>



-- 
Pedro

Mime
View raw message