hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: How to share Same Counter in Multiple Jobs?
Date Fri, 10 Dec 2010 05:51:15 GMT
I wrote the following code today. We have our own flow execution logic which
calls the following to collect counters.

    enum COUNT_COLLECTION {
        LOG,            // log the counters
        ADD_TO_CONF        // add counters to JobConf
    }

    protected static void collectCounters(RunningJob running, JobConf
jobConf, EnumSet<COUNT_COLLECTION> collFlags) {
        try {
            Counters counters = running.getCounters();
            Collection<String> counterGroupNames = counters.getGroupNames();
            if (counterGroupNames == null) {
                LOG.warn("No counters returned from job " +
running.getJobName());
            } else {
                String[] groupsToCollect = { "Map-Reduce Framework",
"FileSystemCounters" };
                for (String counterGroupName : groupsToCollect) {
                    for (Iterator<Counter> iterator =
counters.getGroup(counterGroupName).iterator(); iterator.hasNext();) {
                        Counter counter = iterator.next();
                        String counterName =
counters.getGroup(counterGroupName).getDisplayName()+"."+
                        counter.getDisplayName();
                        if (collFlags.contains(COUNT_COLLECTION.LOG)) {
                            LOG.info(counterName + ": " +
counter.getCounter());
                        }
                    }
                }
            }
        } catch (IOException e) {
            LOG.error("unable to retrieve counters", e);
        }
    }

You can pass the counter from Job 1 to Job 3 via JobConf.

On Thu, Dec 9, 2010 at 9:45 PM, Savannah Beckett <
savannah_beckett30@yahoo.com> wrote:

> Hi,
>   I chain multiple jobs in my program.  Job 1's reduce function has a
> counter.  I want job 3's reduce function to read this Job 1's counter.
> How?
> Thanks.
>
>

Mime
View raw message