hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark N <nipen.m...@gmail.com>
Subject Re: Job Tracker questions
Date Thu, 04 Feb 2010 10:59:39 GMT
Ye currently am using jobclient to read these counters.

But We are not able to use *webservices *because the jar which is used to
read the counters from  running hadoop job  is itself a Hadoop program

If we could have pure Java Api which is run without hadoop command then we
could return the counter variable into webservices and show in UI.

Any help  or technique to show thsese counters in the UI would be
appreciated  ( not necessarily using web service )


I am using webservices because I am having .net VB client

thanks



On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> I think you can use JobClient to get the counters in your web service.
> If you look at the shell script bin/hadoop, you will find that actually
> this
> shell use the JobClient to get the counters.
>
>
>
> On Wed, Feb 3, 2010 at 4:34 AM, Mark N <nipen.mark@gmail.com> wrote:
>
> > We have a hadoop job running and have used custom counters to track  few
> > counters ( like no of successfully processed documents matching certain
> > conditions)
> >
> >
> > Since we need to get this counters even while the Hadoop job is running ,
> > we
> > wrote another Java program to read these counters
> >
> >
> > *Counter reader  program *will do the following :
> >
> >
> > 1)      List all the running jobs.
> >
> > 2)   Get the running job using Job name
> >
> > 2)     Get all the counter for individual running jobs
> >
> > 3)      Set this counters in variables.
> >        We could successfully read these counters  , but since we need to
> > show these counters to custom UI , how can we show these counters?
> >
> >        we looked into various options to read these counters to show in
> UI
> > as following :
> >
> >      1. Dump these counters to database , however this may be overhead
> >      2. Write web service   and UI will invoke the functions from these
> > service to show in UI ( However since we need to run "*Counter reader
> > program "  *with Hadoop command it might not be feasible to write web
> > service ?   )
> >
> >      so the question is can we achive to read the counters using simple
> > Java APIs ? Does anyone have idea how does the default jobtracker JSP
> works
> > ? we wanted to built something similar to this
> >
> > thanks
> >
> >
> >
> > --
> > Nipen Mark
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



-- 
Nipen Mark

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message