hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hailong.yang1115" <hailong.yang1...@gmail.com>
Subject Fw: Problems about the job counters
Date Wed, 15 Jun 2011 03:00:08 GMT

Sorry for sending this email again but I got no answers from the first one. Anyone please
help or forward it to mail-list that would help.

2011-06-15



***********************************************
* Hailong Yang, PhD. Candidate 
* Sino-German Joint Software Institute, 
* School of Computer Science&Engineering, Beihang University
* Phone: (86-010)82315908
* Email: hailong.yang1115@gmail.com
* Address: G413, New Main Building in Beihang University, 
*              No.37 XueYuan Road,HaiDian District, 
*              Beijing,P.R.China,100191
***********************************************



发件人: hailong.yang1115
发送时间: 2011-06-10 13:28:46
收件人: general
抄送: 
主题: Problems about the job counters

Dear all,

I am trying to the built-in example wordcount with nearly 15GB input. When the Hadoop job
finished, I got the following counters.


CounterMapReduceTotal
Job CountersLaunched reduce tasks001
Rack-local map tasks0035
Launched map tasks002,318
Data-local map tasks002,283
FileSystemCountersFILE_BYTES_READ22,863,580,65617,654,943,34140,518,523,997
HDFS_BYTES_READ154,400,997,4590154,400,997,459
FILE_BYTES_WRITTEN33,490,829,40317,654,943,34151,145,772,744
HDFS_BYTES_WRITTEN02,747,356,7042,747,356,704


My question is what does the FILE_BYTES_READ counter mean? And what is the difference between
FILE_BYTES_READ and HDFS_BYTES_READ? In my opinion, all the input is located in HDFS, so where
does FILE_BYTES_READ come from during the map phase?


Any help will be appreciated!

Hailong

2011-06-10 



***********************************************
* Hailong Yang, PhD. Candidate 
* Sino-German Joint Software Institute, 
* School of Computer Science&Engineering, Beihang University
* Phone: (86-010)82315908
* Email: hailong.yang1115@gmail.com
* Address: G413, New Main Building in Beihang University, 
*              No.37 XueYuan Road,HaiDian District, 
*              Beijing,P.R.China,100191
***********************************************
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message