hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris MacKenzie <stu...@chrismackenziephotography.co.uk>
Subject Re: Hadoop 2.2 Built-in Counters
Date Thu, 14 Aug 2014 09:39:26 GMT

This is the content of my shell script for running the job history server:

hadoop fs -mkdir -p /mr-history/tmp
hadoop fs -chmod -R 1777 /mr-history/tmp
hadoop fs -mkdir -p /mr-history/done
hadoop fs -chmod -R 1777 /mr-history/done

sbin/mr-jobhistory-daemon.sh start historyserver

These configurable variables are in mapred-site.xml

        <description>Default port is 10020.</description>
        <description>Default port is 19888.</description>

I start the history server on the same node as my resource manager

The counters are available from when the job is running from:


Drill down through the application master to the job.

If you donĀ¹t have the history server running the job data is not

Hope this helps.


Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk

From:  ou senshaw <senshaw@gmail.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Thursday, 14 August 2014 07:14
To:  <user@hadoop.apache.org>
Subject:  Hadoop 2.2 Built-in Counters

Hi all,
I'm trying to analyze my mapreduce job performance via built-in counters
such as physical memory usage, heap memory usage...
When the job is running, I can watch these counters via Resource manager
website(namenode:8088). However, when the job is done, counter information
is not available in resource manager website anymore. I know I can get
them from client output. I was wondering if there is other place in name
node or data node to get the final counter measures regarding job id?

View raw message