hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom White <...@cloudera.com>
Subject Re: Problem with Counters
Date Thu, 05 Feb 2009 14:49:53 GMT
Try moving the enum to inside the top level class (as you already did)
and then use getCounter() passing the enum value:

public class MyJob {

  static enum MyCounter{ct_key1};

  // Mapper and Reducer defined here

  public static void main(String[] args) throws IOException {
    // ...
    RunningJob running =JobClient.runJob(conf);
    Counters ct = running.getCounters();
    long res = ct.getCounter(MyCounter.ct_key1);
    // ...
  }

}

BTW org.apache.hadoop.mapred.Task$Counter is a built-in MapReduce
counter, so that won't help you retrieve your custom counter.

Cheers,

Tom

On Thu, Feb 5, 2009 at 2:22 PM, Rasit OZDAS <rasitozdas@gmail.com> wrote:
> Sharath,
>
> You're using  reporter.incrCounter(enumVal, intVal);  to increment counter,
> I think method to get should also be similar.
>
> Try to use findCounter(enumVal).getCounter() or  getCounter(enumVal).
>
> Hope this helps,
> Rasit
>
> 2009/2/5 some speed <speed.some@gmail.com>:
>> In fact I put the enum in my Reduce method as the following link (from
>> Yahoo) says so:
>>
>> http://public.yahoo.com/gogate/hadoop-tutorial/html/module5.html#metrics
>> --->Look at the section under Reporting Custom Metrics.
>>
>> 2009/2/5 some speed <speed.some@gmail.com>
>>
>>> Thanks Rasit.
>>>
>>> I did as you said.
>>>
>>> 1) Put the static enum MyCounter{ct_key1} just above main()
>>>
>>> 2) Changed  result =
>>> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 1,
>>> "Reduce.MyCounter").getCounter();
>>>
>>> Still is doesnt seem to help. It throws a null pointer exception.Its not
>>> able to find the Counter.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Sharath
>>>
>>>
>>>
>>>
>>> On Thu, Feb 5, 2009 at 8:04 AM, Rasit OZDAS <rasitozdas@gmail.com> wrote:
>>>
>>>> Forgot to say, value "0" means that the requested counter does not exist.
>>>>
>>>> 2009/2/5 Rasit OZDAS <rasitozdas@gmail.com>:
>>>> > Sharath,
>>>> >  I think the static enum definition should be out of Reduce class.
>>>> > Hadoop probably tries to find it elsewhere with "MyCounter", but it's
>>>> > actually "Reduce.MyCounter" in your example.
>>>> >
>>>> > Hope this helps,
>>>> > Rasit
>>>> >
>>>> > 2009/2/5 some speed <speed.some@gmail.com>:
>>>> >> I Tried the following...It gets compiled but the value of result
seems
>>>> to be
>>>> >> 0 always.
>>>> >>
>>>> >>        RunningJob running = JobClient.runJob(conf);
>>>> >>
>>>> >>         Counters ct = new Counters();
>>>> >>             ct = running.getCounters();
>>>> >>
>>>> >>        long     result =
>>>> >> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
>>>> >> "*MyCounter*").getCounter();
>>>> >> //even tried MyCounter.Key1
>>>> >>
>>>> >>
>>>> >>
>>>> >> Does anyone know whay that is happening?
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >> Sharath
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Thu, Feb 5, 2009 at 5:59 AM, some speed <speed.some@gmail.com>
>>>> wrote:
>>>> >>
>>>> >>> Hi Tom,
>>>> >>>
>>>> >>> I get the error :
>>>> >>>
>>>> >>> Cannot find Symbol* "**MyCounter.ct_key1 " *
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <tom@cloudera.com>
wrote:
>>>> >>>
>>>> >>>> Hi Sharath,
>>>> >>>>
>>>> >>>> The code you posted looks right to me. Counters#getCounter()
will
>>>> >>>> return the counter's value. What error are you getting?
>>>> >>>>
>>>> >>>> Tom
>>>> >>>>
>>>> >>>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <speed.some@gmail.com>
>>>> wrote:
>>>> >>>> > Hi,
>>>> >>>> >
>>>> >>>> > Can someone help me with the usage of counters please?
I am
>>>> incrementing
>>>> >>>> a
>>>> >>>> > counter in Reduce method but I am unable to collect
the counter
>>>> value
>>>> >>>> after
>>>> >>>> > the job is completed.
>>>> >>>> >
>>>> >>>> > Its something like this:
>>>> >>>> >
>>>> >>>> > public static class Reduce extends MapReduceBase implements
>>>> >>>> Reducer<Text,
>>>> >>>> > FloatWritable, Text, FloatWritable>
>>>> >>>> >    {
>>>> >>>> >        static enum MyCounter{ct_key1};
>>>> >>>> >
>>>> >>>> >         public void reduce(..............) throws IOException
>>>> >>>> >        {
>>>> >>>> >
>>>> >>>> >            reporter.incrCounter(MyCounter.ct_key1,
1);
>>>> >>>> >
>>>> >>>> >            output.collect(......);
>>>> >>>> >
>>>> >>>> >        }
>>>> >>>> > }
>>>> >>>> >
>>>> >>>> > -----main method----
>>>> >>>> > {
>>>> >>>> >            RunningJob running = null;
>>>> >>>> >            running=JobClient.runJob(conf);
>>>> >>>> >
>>>> >>>> >            Counters ct = running.getCounters();
>>>> >>>> > /*********  How do I Collect the ct_key1 value ????***********/
>>>> >>>> >            long res = ct.getCounter(MyCounter.ct_key1);
>>>> >>>> >
>>>> >>>> > }
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > Thanks,
>>>> >>>> >
>>>> >>>> > Sharath
>>>> >>>> >
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > M. Raşit ÖZDAŞ
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> M. Raşit ÖZDAŞ
>>>>
>>>
>>>
>>
>
>
>
> --
> M. Raşit ÖZDAŞ
>

Mime
View raw message