hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark N <nipen.m...@gmail.com>
Subject Re: Job Tracker questions
Date Tue, 23 Feb 2010 13:38:01 GMT
i have to find Counter and i am using inetsocketAddress and connecting
into the jobtracker jsp
with help of it am able to find the counter when it is running on pseudo mode

but as i am trying to run this in cluster . i am not able to get any
counters . SI there any parameters that I need to change in
configuration to read the counter information ?


Thanks in advance


On Tue, Feb 9, 2010 at 8:49 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> JobClient also use proxy of JobTracker.
>
>
> On Mon, Feb 8, 2010 at 11:19 PM, Mark N <nipen.mark@gmail.com> wrote:
>
> > Did you check the jobClient source code?
> >
> >
> > On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >
> > > I look at the source code, it seems the job tracker web ui also use the
> > > proxy of JobTracker to get the counter information rather the xml file.
> > >
> > >
> > > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <nipen.mark@gmail.com> wrote:
> > >
> > > > yes we can create a webservice in java which would be called by .net
> to
> > > > display these counters.
> > > >
> > > > But since the java code to read these counters needs use hadoop APIs
>  (
> > > job
> > > > client  ) ,  am not sure we can create a webservice to read the
> > counters
> > > >
> > > > Question is how does the default hadoop task tracker display counter
> > > > information in JSP pages ? does it read from the XML files ?
> > > >
> > > > thanks,
> > > >
> > > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> > > >
> > > > > I think you can create web service using Java, and then in .net
> using
> > > the
> > > > > web service to display the result.
> > > > >
> > > > >
> > > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <zjffdu@gmail.com>
> wrote:
> > > > >
> > > > > > Do you mean want to connect the JobTracker using .Net ? If so,
> I'm
> > > > afraid
> > > > > I
> > > > > > have no idea how to this. The rpc of hadoop is language
> dependent.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <nipen.mark@gmail.com>
> > wrote:
> > > > > >
> > > > > >> could you please elaborate on this  ( * hint to get started
 as
> am
> > > > very
> > > > > >> new
> > > > > >> to hadoop? )
> > > > > >> So far I could succesfully read all the default and custom
> > counters.
> > > > > >>
> > > > > >> Currently we are having a .net client.
> > > > > >>
> > > > > >> thanks in advance.
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <zjffdu@gmail.com>
> > > wrote:
> > > > > >>
> > > > > >> > Well, you can create a proxy of JobTracker in client
side, and
> > > then
> > > > > you
> > > > > >> can
> > > > > >> > use the API of JobTracker to get the information of
jobs. The
> > > Proxy
> > > > > take
> > > > > >> > the
> > > > > >> > responsibility of  communication with the Master Node.
 Read
> the
> > > > > source
> > > > > >> > code
> > > > > >> > of JobClient can help you.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <nipen.mark@gmail.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > > Ye currently am using jobclient to read these
counters.
> > > > > >> > >
> > > > > >> > > But We are not able to use *webservices *because
the jar
> which
> > > is
> > > > > used
> > > > > >> to
> > > > > >> > > read the counters from  running hadoop job  is
itself a
> Hadoop
> > > > > program
> > > > > >> > >
> > > > > >> > > If we could have pure Java Api which is run without
hadoop
> > > command
> > > > > >> then
> > > > > >> > we
> > > > > >> > > could return the counter variable into webservices
and show
> in
> > > UI.
> > > > > >> > >
> > > > > >> > > Any help  or technique to show thsese counters
in the UI
> would
> > > be
> > > > > >> > > appreciated  ( not necessarily using web service
)
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > I am using webservices because I am having .net
VB client
> > > > > >> > >
> > > > > >> > > thanks
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <
> zjffdu@gmail.com>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > I think you can use JobClient to get the
counters in your
> > web
> > > > > >> service.
> > > > > >> > > > If you look at the shell script bin/hadoop,
you will find
> > that
> > > > > >> actually
> > > > > >> > > > this
> > > > > >> > > > shell use the JobClient to get the counters.
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <
> > nipen.mark@gmail.com>
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > > > We have a hadoop job running and have
used custom
> counters
> > > to
> > > > > >> track
> > > > > >> > >  few
> > > > > >> > > > > counters ( like no of successfully processed
documents
> > > > matching
> > > > > >> > certain
> > > > > >> > > > > conditions)
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > Since we need to get this counters even
while the Hadoop
> > job
> > > > is
> > > > > >> > running
> > > > > >> > > ,
> > > > > >> > > > > we
> > > > > >> > > > > wrote another Java program to read these
counters
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > *Counter reader  program *will do the
following :
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > 1)      List all the running jobs.
> > > > > >> > > > >
> > > > > >> > > > > 2)   Get the running job using Job name
> > > > > >> > > > >
> > > > > >> > > > > 2)     Get all the counter for individual
running jobs
> > > > > >> > > > >
> > > > > >> > > > > 3)      Set this counters in variables.
> > > > > >> > > > >        We could successfully read these
counters  , but
> > > since
> > > > we
> > > > > >> need
> > > > > >> > > to
> > > > > >> > > > > show these counters to custom UI , how
can we show these
> > > > > counters?
> > > > > >> > > > >
> > > > > >> > > > >        we looked into various options
to read these
> > counters
> > > > to
> > > > > >> show
> > > > > >> > in
> > > > > >> > > > UI
> > > > > >> > > > > as following :
> > > > > >> > > > >
> > > > > >> > > > >      1. Dump these counters to database
, however this
> may
> > > be
> > > > > >> > overhead
> > > > > >> > > > >      2. Write web service   and UI will
invoke the
> > functions
> > > > > from
> > > > > >> > these
> > > > > >> > > > > service to show in UI ( However since
we need to run
> > > "*Counter
> > > > > >> reader
> > > > > >> > > > > program "  *with Hadoop command it might
not be feasible
> > to
> > > > > write
> > > > > >> web
> > > > > >> > > > > service ?   )
> > > > > >> > > > >
> > > > > >> > > > >      so the question is can we achive
to read the
> counters
> > > > using
> > > > > >> > simple
> > > > > >> > > > > Java APIs ? Does anyone have idea how
does the default
> > > > > jobtracker
> > > > > >> JSP
> > > > > >> > > > works
> > > > > >> > > > > ? we wanted to built something similar
to this
> > > > > >> > > > >
> > > > > >> > > > > thanks
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > --
> > > > > >> > > > > Nipen Mark
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > --
> > > > > >> > > > Best Regards
> > > > > >> > > >
> > > > > >> > > > Jeff Zhang
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > Nipen Mark
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Best Regards
> > > > > >> >
> > > > > >> > Jeff Zhang
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Nipen Mark
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards
> > > > > >
> > > > > > Jeff Zhang
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Regards
> > > > >
> > > > > Jeff Zhang
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Nipen Mark
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> >
> >
> >
> > --
> > Nipen Mark
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



-- 
Nipen Mark

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message