hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akira AJISAKA <ajisa...@oss.nttdata.co.jp>
Subject Re: Update interval of default counters
Date Wed, 16 Apr 2014 09:44:00 GMT
Moved mapreduce-dev@ to Bcc.

Hi Dharmesh,

The parameter is to set the interval of polling the progress
of the MRAppMaster, not the Map/Reduce tasks. The tasks send
the progress (includes the counter information) to MRAppMaster
every 3000 milliseconds, which is hard-coded.

That's why a sudden big change in counter values happens
even if the parameter is set to a small value.

Regards,
Akira

(2014/04/16 15:42), Dharmesh Kakadia wrote:
> Hi Akira,
>
> Thanks for the reply, but as I understand this is the interval of console
> counter printing. What I am trying to get
>
> while(!job.isComplete()){
>   getcounters() and do some processing on that.
> }
>
> Now this is running fine, but the status I get the same counter values
> repeatedly and then suddenly a big change in counter values.
> For example, getcounters for REDUCE_INPUT_RECORDS returns values like
>
> 0
> 0
> ..
> 0
> 280
> 280
> ...
> 280
> 516
> 516
> ...
> 516
>
> etc.
>
> I want to get more finer values, instead of directly jumping from 280 to
> 516.
> Did that make sense? mapreduce.client.progressmonitor.pollinterval does not
> seem to effect it. Any workaround ?
>
> Thanks,
> Dharmesh
>
>
>
>
> On Tue, Apr 15, 2014 at 7:51 PM, Akira AJISAKA
> <ajisakaa@oss.nttdata.co.jp>wrote:
>
>> Moved to user@hadoop.apache.org.
>>
>> You can configure the interval by setting
>> "mapreduce.client.progressmonitor.pollinterval" parameter.
>> The default value is 1000 ms.
>>
>> For more details, please see http://hadoop.apache.org/docs/
>> stable/hadoop-mapreduce-client/hadoop-mapreduce-
>> client-core/mapred-default.xml.
>>
>> Regards,
>> Akira
>>
>>
>> (2014/04/15 15:29), Dharmesh Kakadia wrote:
>>
>>> Hi,
>>>
>>> What is the update interval of inbuilt framework counters? Is that
>>> configurable?
>>> I am trying to collect very fine grained information about the job
>>> execution and using counters for that. It would be great if someone can
>>> point me to documentation/code for it. Thanks in advance.
>>>
>>> Thanks,
>>> Dharmesh
>>>
>>>
>>
>


Mime
View raw message