hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Baldeschwieler <eri...@yahoo-inc.com>
Subject Re: use counter to statistics file row number
Date Fri, 21 Jan 2011 07:47:27 GMT
Hi ,

mapreduce-user seems like a great place for this discussion.  general does not.

Thanks!
E14

On Jan 19, 2011, at 7:33 PM, venkatesh kavuluri wrote:

> (Bcc general@. This is for Hadoop project level discussions. Includingmapreduce
> -user@)
> 
> Liu,
> 
> If you want the count of number of records in your input data set, the
> map/reduce framework provides a default counter "Map input records". The
> only caution to follow regarding the custom counters is to not exceed 20 per
> application as they are very expensive. Generally counters are used to track
> few important pieces of information.
> 
> Thanks,
> Venkatesh Kavuluri
> 
> 2011/1/19 lei liu <liulei412@gmail.com>
> 
>> I use counter to statistics  file row number in Mapper, example below code:
>> 
>> public void map(LongWritable key, Text value,
>>           OutputCollector<Text, IntWritable> output, Reporter reporter)
>>           throws IOException {
>> 
>>       try {
>> 
>>           reporter.incrCounter("row", "num", 1);
>> 
>>       } catch (Throwable e) {
>>           e.printStackTrace();
>>           throw new RuntimeException(e);
>>       }
>>   }
>> 
>> Could everyone tell me whether there are any risks do so?
>> 
>> 
>> Thanks,
>> 
>> LiuLei
>> 


Mime
View raw message