hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Job Tracker questions
Date Wed, 03 Feb 2010 14:48:24 GMT
I think you can use JobClient to get the counters in your web service.
If you look at the shell script bin/hadoop, you will find that actually this
shell use the JobClient to get the counters.



On Wed, Feb 3, 2010 at 4:34 AM, Mark N <nipen.mark@gmail.com> wrote:

> We have a hadoop job running and have used custom counters to track  few
> counters ( like no of successfully processed documents matching certain
> conditions)
>
>
> Since we need to get this counters even while the Hadoop job is running ,
> we
> wrote another Java program to read these counters
>
>
> *Counter reader  program *will do the following :
>
>
> 1)      List all the running jobs.
>
> 2)   Get the running job using Job name
>
> 2)     Get all the counter for individual running jobs
>
> 3)      Set this counters in variables.
>        We could successfully read these counters  , but since we need to
> show these counters to custom UI , how can we show these counters?
>
>        we looked into various options to read these counters to show in UI
> as following :
>
>      1. Dump these counters to database , however this may be overhead
>      2. Write web service   and UI will invoke the functions from these
> service to show in UI ( However since we need to run "*Counter reader
> program "  *with Hadoop command it might not be feasible to write web
> service ?   )
>
>      so the question is can we achive to read the counters using simple
> Java APIs ? Does anyone have idea how does the default jobtracker JSP works
> ? we wanted to built something similar to this
>
> thanks
>
>
>
> --
> Nipen Mark
>



-- 
Best Regards

Jeff Zhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message