hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark N <nipen.m...@gmail.com>
Subject Re: Job Tracker questions
Date Thu, 04 Feb 2010 11:18:26 GMT
could you please elaborate on this  ( * hint to get started  as am very new
to hadoop? )
So far I could succesfully read all the default and custom counters.

Currently we are having a .net client.

thanks in advance.


On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> Well, you can create a proxy of JobTracker in client side, and then you can
> use the API of JobTracker to get the information of jobs. The Proxy take
> the
> responsibility of  communication with the Master Node.  Read the source
> code
> of JobClient can help you.
>
>
> On Thu, Feb 4, 2010 at 6:59 PM, Mark N <nipen.mark@gmail.com> wrote:
>
> > Ye currently am using jobclient to read these counters.
> >
> > But We are not able to use *webservices *because the jar which is used to
> > read the counters from  running hadoop job  is itself a Hadoop program
> >
> > If we could have pure Java Api which is run without hadoop command then
> we
> > could return the counter variable into webservices and show in UI.
> >
> > Any help  or technique to show thsese counters in the UI would be
> > appreciated  ( not necessarily using web service )
> >
> >
> > I am using webservices because I am having .net VB client
> >
> > thanks
> >
> >
> >
> > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >
> > > I think you can use JobClient to get the counters in your web service.
> > > If you look at the shell script bin/hadoop, you will find that actually
> > > this
> > > shell use the JobClient to get the counters.
> > >
> > >
> > >
> > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <nipen.mark@gmail.com> wrote:
> > >
> > > > We have a hadoop job running and have used custom counters to track
> >  few
> > > > counters ( like no of successfully processed documents matching
> certain
> > > > conditions)
> > > >
> > > >
> > > > Since we need to get this counters even while the Hadoop job is
> running
> > ,
> > > > we
> > > > wrote another Java program to read these counters
> > > >
> > > >
> > > > *Counter reader  program *will do the following :
> > > >
> > > >
> > > > 1)      List all the running jobs.
> > > >
> > > > 2)   Get the running job using Job name
> > > >
> > > > 2)     Get all the counter for individual running jobs
> > > >
> > > > 3)      Set this counters in variables.
> > > >        We could successfully read these counters  , but since we need
> > to
> > > > show these counters to custom UI , how can we show these counters?
> > > >
> > > >        we looked into various options to read these counters to show
> in
> > > UI
> > > > as following :
> > > >
> > > >      1. Dump these counters to database , however this may be
> overhead
> > > >      2. Write web service   and UI will invoke the functions from
> these
> > > > service to show in UI ( However since we need to run "*Counter reader
> > > > program "  *with Hadoop command it might not be feasible to write web
> > > > service ?   )
> > > >
> > > >      so the question is can we achive to read the counters using
> simple
> > > > Java APIs ? Does anyone have idea how does the default jobtracker JSP
> > > works
> > > > ? we wanted to built something similar to this
> > > >
> > > > thanks
> > > >
> > > >
> > > >
> > > > --
> > > > Nipen Mark
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> >
> >
> >
> > --
> > Nipen Mark
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



-- 
Nipen Mark

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message