hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark N <nipen.m...@gmail.com>
Subject Re: Job Tracker questions
Date Thu, 04 Feb 2010 11:29:31 GMT
yes we can create a webservice in java which would be called by .net to
display these counters.

But since the java code to read these counters needs use hadoop APIs  ( job
client  ) ,  am not sure we can create a webservice to read the counters

Question is how does the default hadoop task tracker display counter
information in JSP pages ? does it read from the XML files ?

thanks,

On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> I think you can create web service using Java, and then in .net using the
> web service to display the result.
>
>
> On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid
> I
> > have no idea how to this. The rpc of hadoop is language dependent.
> >
> >
> >
> >
> > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <nipen.mark@gmail.com> wrote:
> >
> >> could you please elaborate on this  ( * hint to get started  as am very
> >> new
> >> to hadoop? )
> >> So far I could succesfully read all the default and custom counters.
> >>
> >> Currently we are having a .net client.
> >>
> >> thanks in advance.
> >>
> >>
> >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >>
> >> > Well, you can create a proxy of JobTracker in client side, and then
> you
> >> can
> >> > use the API of JobTracker to get the information of jobs. The Proxy
> take
> >> > the
> >> > responsibility of  communication with the Master Node.  Read the
> source
> >> > code
> >> > of JobClient can help you.
> >> >
> >> >
> >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <nipen.mark@gmail.com> wrote:
> >> >
> >> > > Ye currently am using jobclient to read these counters.
> >> > >
> >> > > But We are not able to use *webservices *because the jar which is
> used
> >> to
> >> > > read the counters from  running hadoop job  is itself a Hadoop
> program
> >> > >
> >> > > If we could have pure Java Api which is run without hadoop command
> >> then
> >> > we
> >> > > could return the counter variable into webservices and show in UI.
> >> > >
> >> > > Any help  or technique to show thsese counters in the UI would be
> >> > > appreciated  ( not necessarily using web service )
> >> > >
> >> > >
> >> > > I am using webservices because I am having .net VB client
> >> > >
> >> > > thanks
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <zjffdu@gmail.com>
> wrote:
> >> > >
> >> > > > I think you can use JobClient to get the counters in your web
> >> service.
> >> > > > If you look at the shell script bin/hadoop, you will find that
> >> actually
> >> > > > this
> >> > > > shell use the JobClient to get the counters.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <nipen.mark@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > We have a hadoop job running and have used custom counters
to
> >> track
> >> > >  few
> >> > > > > counters ( like no of successfully processed documents matching
> >> > certain
> >> > > > > conditions)
> >> > > > >
> >> > > > >
> >> > > > > Since we need to get this counters even while the Hadoop
job is
> >> > running
> >> > > ,
> >> > > > > we
> >> > > > > wrote another Java program to read these counters
> >> > > > >
> >> > > > >
> >> > > > > *Counter reader  program *will do the following :
> >> > > > >
> >> > > > >
> >> > > > > 1)      List all the running jobs.
> >> > > > >
> >> > > > > 2)   Get the running job using Job name
> >> > > > >
> >> > > > > 2)     Get all the counter for individual running jobs
> >> > > > >
> >> > > > > 3)      Set this counters in variables.
> >> > > > >        We could successfully read these counters  , but
since we
> >> need
> >> > > to
> >> > > > > show these counters to custom UI , how can we show these
> counters?
> >> > > > >
> >> > > > >        we looked into various options to read these counters
to
> >> show
> >> > in
> >> > > > UI
> >> > > > > as following :
> >> > > > >
> >> > > > >      1. Dump these counters to database , however this may
be
> >> > overhead
> >> > > > >      2. Write web service   and UI will invoke the functions
> from
> >> > these
> >> > > > > service to show in UI ( However since we need to run "*Counter
> >> reader
> >> > > > > program "  *with Hadoop command it might not be feasible
to
> write
> >> web
> >> > > > > service ?   )
> >> > > > >
> >> > > > >      so the question is can we achive to read the counters
using
> >> > simple
> >> > > > > Java APIs ? Does anyone have idea how does the default
> jobtracker
> >> JSP
> >> > > > works
> >> > > > > ? we wanted to built something similar to this
> >> > > > >
> >> > > > > thanks
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Nipen Mark
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Best Regards
> >> > > >
> >> > > > Jeff Zhang
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Nipen Mark
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Best Regards
> >> >
> >> > Jeff Zhang
> >> >
> >>
> >>
> >>
> >> --
> >> Nipen Mark
> >>
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



-- 
Nipen Mark

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message