hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From venkatesh kavuluri <venkatesh.kowl...@gmail.com>
Subject Re: use counter to statistics file row number
Date Thu, 20 Jan 2011 03:33:02 GMT
(Bcc general@. This is for Hadoop project level discussions. Includingmapreduce
-user@)

Liu,

If you want the count of number of records in your input data set, the
map/reduce framework provides a default counter "Map input records". The
only caution to follow regarding the custom counters is to not exceed 20 per
application as they are very expensive. Generally counters are used to track
few important pieces of information.

Thanks,
Venkatesh Kavuluri

2011/1/19 lei liu <liulei412@gmail.com>

> I use counter to statistics  file row number in Mapper, example below code:
>
> public void map(LongWritable key, Text value,
>            OutputCollector<Text, IntWritable> output, Reporter reporter)
>            throws IOException {
>
>        try {
>
>            reporter.incrCounter("row", "num", 1);
>
>        } catch (Throwable e) {
>            e.printStackTrace();
>            throw new RuntimeException(e);
>        }
>    }
>
> Could everyone tell me whether there are any risks do so?
>
>
> Thanks,
>
> LiuLei
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message