hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Job Tracker questions
Date Wed, 24 Feb 2010 02:49:26 GMT
This is the sample code for get the counters for one specified job ( I have
tested it on my cluster).  What you need to change is the jobtracker address
and jobID. Remember to put this class in package org.apache.hadoop.mapred,
becuase the JobSubmissionProtocol is not public.

package org.apache.hadoop.mapred;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.Iterator;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.ipc.RPC;
import org.apache.hadoop.mapred.Counters.Counter;
import org.apache.hadoop.mapred.Counters.Group;

public class MyServer {

    public static void main(String[] args) throws IOException {
        InetSocketAddress address = new InetSocketAddress("sha-cs-04",
9001);
        JobSubmissionProtocol jobClient = (JobSubmissionProtocol)
RPC.getProxy(
                JobSubmissionProtocol.class,
JobSubmissionProtocol.versionID,
                address, new Configuration());
        JobID jobID=new JobID("201002111947",460);
        Counters counters= jobClient.getJobCounters(jobID);
        Iterator<Group> iter=counters.iterator();
        while(iter.hasNext()){
            Group group=iter.next();
            System.out.println(group.getDisplayName());
            Iterator<Counter> cIter=group.iterator();
            while(cIter.hasNext()){
                Counter counter=cIter.next();

System.out.println("\t"+counter.getName()+":"+counter.getValue());
            }
        }
    }
}

On Tue, Feb 23, 2010 at 9:38 PM, Mark N <nipen.mark@gmail.com> wrote:

> i have to find Counter and i am using inetsocketAddress and connecting
> into the jobtracker jsp
> with help of it am able to find the counter when it is running on pseudo
> mode
>
> but as i am trying to run this in cluster . i am not able to get any
> counters . SI there any parameters that I need to change in
> configuration to read the counter information ?
>
>
> Thanks in advance
>
>
> On Tue, Feb 9, 2010 at 8:49 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > JobClient also use proxy of JobTracker.
> >
> >
> > On Mon, Feb 8, 2010 at 11:19 PM, Mark N <nipen.mark@gmail.com> wrote:
> >
> > > Did you check the jobClient source code?
> > >
> > >
> > > On Thu, Feb 4, 2010 at 5:21 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> > >
> > > > I look at the source code, it seems the job tracker web ui also use
> the
> > > > proxy of JobTracker to get the counter information rather the xml
> file.
> > > >
> > > >
> > > > On Thu, Feb 4, 2010 at 7:29 PM, Mark N <nipen.mark@gmail.com> wrote:
> > > >
> > > > > yes we can create a webservice in java which would be called by
> .net
> > to
> > > > > display these counters.
> > > > >
> > > > > But since the java code to read these counters needs use hadoop
> APIs
> >  (
> > > > job
> > > > > client  ) ,  am not sure we can create a webservice to read the
> > > counters
> > > > >
> > > > > Question is how does the default hadoop task tracker display
> counter
> > > > > information in JSP pages ? does it read from the XML files ?
> > > > >
> > > > > thanks,
> > > > >
> > > > > On Thu, Feb 4, 2010 at 5:08 PM, Jeff Zhang <zjffdu@gmail.com>
> wrote:
> > > > >
> > > > > > I think you can create web service using Java, and then in .net
> > using
> > > > the
> > > > > > web service to display the result.
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <zjffdu@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Do you mean want to connect the JobTracker using .Net ?
If so,
> > I'm
> > > > > afraid
> > > > > > I
> > > > > > > have no idea how to this. The rpc of hadoop is language
> > dependent.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Feb 4, 2010 at 7:18 PM, Mark N <nipen.mark@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > >> could you please elaborate on this  ( * hint to get
started
>  as
> > am
> > > > > very
> > > > > > >> new
> > > > > > >> to hadoop? )
> > > > > > >> So far I could succesfully read all the default and
custom
> > > counters.
> > > > > > >>
> > > > > > >> Currently we are having a .net client.
> > > > > > >>
> > > > > > >> thanks in advance.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <zjffdu@gmail.com>
> > > > wrote:
> > > > > > >>
> > > > > > >> > Well, you can create a proxy of JobTracker in
client side,
> and
> > > > then
> > > > > > you
> > > > > > >> can
> > > > > > >> > use the API of JobTracker to get the information
of jobs.
> The
> > > > Proxy
> > > > > > take
> > > > > > >> > the
> > > > > > >> > responsibility of  communication with the Master
Node.  Read
> > the
> > > > > > source
> > > > > > >> > code
> > > > > > >> > of JobClient can help you.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <
> nipen.mark@gmail.com>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Ye currently am using jobclient to read these
counters.
> > > > > > >> > >
> > > > > > >> > > But We are not able to use *webservices *because
the jar
> > which
> > > > is
> > > > > > used
> > > > > > >> to
> > > > > > >> > > read the counters from  running hadoop job
 is itself a
> > Hadoop
> > > > > > program
> > > > > > >> > >
> > > > > > >> > > If we could have pure Java Api which is run
without hadoop
> > > > command
> > > > > > >> then
> > > > > > >> > we
> > > > > > >> > > could return the counter variable into webservices
and
> show
> > in
> > > > UI.
> > > > > > >> > >
> > > > > > >> > > Any help  or technique to show thsese counters
in the UI
> > would
> > > > be
> > > > > > >> > > appreciated  ( not necessarily using web
service )
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > I am using webservices because I am having
.net VB client
> > > > > > >> > >
> > > > > > >> > > thanks
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang
<
> > zjffdu@gmail.com>
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > I think you can use JobClient to get
the counters in
> your
> > > web
> > > > > > >> service.
> > > > > > >> > > > If you look at the shell script bin/hadoop,
you will
> find
> > > that
> > > > > > >> actually
> > > > > > >> > > > this
> > > > > > >> > > > shell use the JobClient to get the counters.
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark
N <
> > > nipen.mark@gmail.com>
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > > > We have a hadoop job running and
have used custom
> > counters
> > > > to
> > > > > > >> track
> > > > > > >> > >  few
> > > > > > >> > > > > counters ( like no of successfully
processed documents
> > > > > matching
> > > > > > >> > certain
> > > > > > >> > > > > conditions)
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > Since we need to get this counters
even while the
> Hadoop
> > > job
> > > > > is
> > > > > > >> > running
> > > > > > >> > > ,
> > > > > > >> > > > > we
> > > > > > >> > > > > wrote another Java program to read
these counters
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > *Counter reader  program *will
do the following :
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > 1)      List all the running jobs.
> > > > > > >> > > > >
> > > > > > >> > > > > 2)   Get the running job using
Job name
> > > > > > >> > > > >
> > > > > > >> > > > > 2)     Get all the counter for
individual running jobs
> > > > > > >> > > > >
> > > > > > >> > > > > 3)      Set this counters in variables.
> > > > > > >> > > > >        We could successfully read
these counters  ,
> but
> > > > since
> > > > > we
> > > > > > >> need
> > > > > > >> > > to
> > > > > > >> > > > > show these counters to custom UI
, how can we show
> these
> > > > > > counters?
> > > > > > >> > > > >
> > > > > > >> > > > >        we looked into various options
to read these
> > > counters
> > > > > to
> > > > > > >> show
> > > > > > >> > in
> > > > > > >> > > > UI
> > > > > > >> > > > > as following :
> > > > > > >> > > > >
> > > > > > >> > > > >      1. Dump these counters to
database , however this
> > may
> > > > be
> > > > > > >> > overhead
> > > > > > >> > > > >      2. Write web service   and
UI will invoke the
> > > functions
> > > > > > from
> > > > > > >> > these
> > > > > > >> > > > > service to show in UI ( However
since we need to run
> > > > "*Counter
> > > > > > >> reader
> > > > > > >> > > > > program "  *with Hadoop command
it might not be
> feasible
> > > to
> > > > > > write
> > > > > > >> web
> > > > > > >> > > > > service ?   )
> > > > > > >> > > > >
> > > > > > >> > > > >      so the question is can we
achive to read the
> > counters
> > > > > using
> > > > > > >> > simple
> > > > > > >> > > > > Java APIs ? Does anyone have idea
how does the default
> > > > > > jobtracker
> > > > > > >> JSP
> > > > > > >> > > > works
> > > > > > >> > > > > ? we wanted to built something
similar to this
> > > > > > >> > > > >
> > > > > > >> > > > > thanks
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > --
> > > > > > >> > > > > Nipen Mark
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > --
> > > > > > >> > > > Best Regards
> > > > > > >> > > >
> > > > > > >> > > > Jeff Zhang
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > --
> > > > > > >> > > Nipen Mark
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > Best Regards
> > > > > > >> >
> > > > > > >> > Jeff Zhang
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Nipen Mark
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards
> > > > > > >
> > > > > > > Jeff Zhang
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards
> > > > > >
> > > > > > Jeff Zhang
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Nipen Mark
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards
> > > >
> > > > Jeff Zhang
> > > >
> > >
> > >
> > >
> > > --
> > > Nipen Mark
> > >
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>
>
>
> --
> Nipen Mark
>



-- 
Best Regards

Jeff Zhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message