hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Job Tracker questions
Date Thu, 04 Feb 2010 11:36:13 GMT
I look at the source code, it seems the job tracker web ui also use the
proxy of JobTracker to get the counter information rather the xml file.


On Thu, Feb 4, 2010 at 7:29 PM, Mark N <nipen.mark@gmail.com> wrote:

> yes we can create a webservice in java which would be called by .net to
> display these counters.
>
> But since the java code to read these counters needs use hadoop APIs  ( job
> client  ) ,  am not sure we can create a webservice to read the counters
>
> Question is how does the default hadoop task tracker display counter
> information in JSP pages ? does it read from the XML files ?
>
> thanks,
>
> On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > I think you can create web service using Java, and then in .net using the
> > web service to display the result.
> >
> >
> > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >
> > > Do you mean want to connect the JobTracker using .Net ? If so, I'm
> afraid
> > I
> > > have no idea how to this. The rpc of hadoop is language dependent.
> > >
> > >
> > >
> > >
> > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <nipen.mark@gmail.com> wrote:
> > >
> > >> could you please elaborate on this  ( * hint to get started  as am
> very
> > >> new
> > >> to hadoop? )
> > >> So far I could succesfully read all the default and custom counters.
> > >>
> > >> Currently we are having a .net client.
> > >>
> > >> thanks in advance.
> > >>
> > >>
> > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> > >>
> > >> > Well, you can create a proxy of JobTracker in client side, and then
> > you
> > >> can
> > >> > use the API of JobTracker to get the information of jobs. The Proxy
> > take
> > >> > the
> > >> > responsibility of  communication with the Master Node.  Read the
> > source
> > >> > code
> > >> > of JobClient can help you.
> > >> >
> > >> >
> > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <nipen.mark@gmail.com>
> wrote:
> > >> >
> > >> > > Ye currently am using jobclient to read these counters.
> > >> > >
> > >> > > But We are not able to use *webservices *because the jar which
is
> > used
> > >> to
> > >> > > read the counters from  running hadoop job  is itself a Hadoop
> > program
> > >> > >
> > >> > > If we could have pure Java Api which is run without hadoop command
> > >> then
> > >> > we
> > >> > > could return the counter variable into webservices and show in
UI.
> > >> > >
> > >> > > Any help  or technique to show thsese counters in the UI would
be
> > >> > > appreciated  ( not necessarily using web service )
> > >> > >
> > >> > >
> > >> > > I am using webservices because I am having .net VB client
> > >> > >
> > >> > > thanks
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <zjffdu@gmail.com>
> > wrote:
> > >> > >
> > >> > > > I think you can use JobClient to get the counters in your
web
> > >> service.
> > >> > > > If you look at the shell script bin/hadoop, you will find
that
> > >> actually
> > >> > > > this
> > >> > > > shell use the JobClient to get the counters.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <nipen.mark@gmail.com>
> > >> wrote:
> > >> > > >
> > >> > > > > We have a hadoop job running and have used custom counters
to
> > >> track
> > >> > >  few
> > >> > > > > counters ( like no of successfully processed documents
> matching
> > >> > certain
> > >> > > > > conditions)
> > >> > > > >
> > >> > > > >
> > >> > > > > Since we need to get this counters even while the Hadoop
job
> is
> > >> > running
> > >> > > ,
> > >> > > > > we
> > >> > > > > wrote another Java program to read these counters
> > >> > > > >
> > >> > > > >
> > >> > > > > *Counter reader  program *will do the following :
> > >> > > > >
> > >> > > > >
> > >> > > > > 1)      List all the running jobs.
> > >> > > > >
> > >> > > > > 2)   Get the running job using Job name
> > >> > > > >
> > >> > > > > 2)     Get all the counter for individual running jobs
> > >> > > > >
> > >> > > > > 3)      Set this counters in variables.
> > >> > > > >        We could successfully read these counters  ,
but since
> we
> > >> need
> > >> > > to
> > >> > > > > show these counters to custom UI , how can we show
these
> > counters?
> > >> > > > >
> > >> > > > >        we looked into various options to read these
counters
> to
> > >> show
> > >> > in
> > >> > > > UI
> > >> > > > > as following :
> > >> > > > >
> > >> > > > >      1. Dump these counters to database , however this
may be
> > >> > overhead
> > >> > > > >      2. Write web service   and UI will invoke the
functions
> > from
> > >> > these
> > >> > > > > service to show in UI ( However since we need to run
"*Counter
> > >> reader
> > >> > > > > program "  *with Hadoop command it might not be feasible
to
> > write
> > >> web
> > >> > > > > service ?   )
> > >> > > > >
> > >> > > > >      so the question is can we achive to read the counters
> using
> > >> > simple
> > >> > > > > Java APIs ? Does anyone have idea how does the default
> > jobtracker
> > >> JSP
> > >> > > > works
> > >> > > > > ? we wanted to built something similar to this
> > >> > > > >
> > >> > > > > thanks
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Nipen Mark
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Best Regards
> > >> > > >
> > >> > > > Jeff Zhang
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Nipen Mark
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Best Regards
> > >> >
> > >> > Jeff Zhang
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Nipen Mark
> > >>
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>
>
>
> --
> Nipen Mark
>



-- 
Best Regards

Jeff Zhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message