hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Job Tracker questions
Date Thu, 04 Feb 2010 11:27:28 GMT
You can use org.apache.hadoop.ipc.RPC.getProxy(....) to initialize the proxy
of JobTracker



On Thu, Feb 4, 2010 at 7:23 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> I think you can create web service using Java, and then in .net using the
> web service to display the result.
>
>
>
> On Thu, Feb 4, 2010 at 7:21 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
>> Do you mean want to connect the JobTracker using .Net ? If so, I'm afraid
>> I have no idea how to this. The rpc of hadoop is language dependent.
>>
>>
>>
>>
>> On Thu, Feb 4, 2010 at 7:18 PM, Mark N <nipen.mark@gmail.com> wrote:
>>
>>> could you please elaborate on this  ( * hint to get started  as am very
>>> new
>>> to hadoop? )
>>> So far I could succesfully read all the default and custom counters.
>>>
>>> Currently we are having a .net client.
>>>
>>> thanks in advance.
>>>
>>>
>>> On Thu, Feb 4, 2010 at 4:53 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>>>
>>> > Well, you can create a proxy of JobTracker in client side, and then you
>>> can
>>> > use the API of JobTracker to get the information of jobs. The Proxy
>>> take
>>> > the
>>> > responsibility of  communication with the Master Node.  Read the source
>>> > code
>>> > of JobClient can help you.
>>> >
>>> >
>>> > On Thu, Feb 4, 2010 at 6:59 PM, Mark N <nipen.mark@gmail.com> wrote:
>>> >
>>> > > Ye currently am using jobclient to read these counters.
>>> > >
>>> > > But We are not able to use *webservices *because the jar which is
>>> used to
>>> > > read the counters from  running hadoop job  is itself a Hadoop
>>> program
>>> > >
>>> > > If we could have pure Java Api which is run without hadoop command
>>> then
>>> > we
>>> > > could return the counter variable into webservices and show in UI.
>>> > >
>>> > > Any help  or technique to show thsese counters in the UI would be
>>> > > appreciated  ( not necessarily using web service )
>>> > >
>>> > >
>>> > > I am using webservices because I am having .net VB client
>>> > >
>>> > > thanks
>>> > >
>>> > >
>>> > >
>>> > > On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <zjffdu@gmail.com>
wrote:
>>> > >
>>> > > > I think you can use JobClient to get the counters in your web
>>> service.
>>> > > > If you look at the shell script bin/hadoop, you will find that
>>> actually
>>> > > > this
>>> > > > shell use the JobClient to get the counters.
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N <nipen.mark@gmail.com>
>>> wrote:
>>> > > >
>>> > > > > We have a hadoop job running and have used custom counters
to
>>> track
>>> > >  few
>>> > > > > counters ( like no of successfully processed documents matching
>>> > certain
>>> > > > > conditions)
>>> > > > >
>>> > > > >
>>> > > > > Since we need to get this counters even while the Hadoop
job is
>>> > running
>>> > > ,
>>> > > > > we
>>> > > > > wrote another Java program to read these counters
>>> > > > >
>>> > > > >
>>> > > > > *Counter reader  program *will do the following :
>>> > > > >
>>> > > > >
>>> > > > > 1)      List all the running jobs.
>>> > > > >
>>> > > > > 2)   Get the running job using Job name
>>> > > > >
>>> > > > > 2)     Get all the counter for individual running jobs
>>> > > > >
>>> > > > > 3)      Set this counters in variables.
>>> > > > >        We could successfully read these counters  , but since
we
>>> need
>>> > > to
>>> > > > > show these counters to custom UI , how can we show these
>>> counters?
>>> > > > >
>>> > > > >        we looked into various options to read these counters
to
>>> show
>>> > in
>>> > > > UI
>>> > > > > as following :
>>> > > > >
>>> > > > >      1. Dump these counters to database , however this may
be
>>> > overhead
>>> > > > >      2. Write web service   and UI will invoke the functions
from
>>> > these
>>> > > > > service to show in UI ( However since we need to run "*Counter
>>> reader
>>> > > > > program "  *with Hadoop command it might not be feasible
to write
>>> web
>>> > > > > service ?   )
>>> > > > >
>>> > > > >      so the question is can we achive to read the counters
using
>>> > simple
>>> > > > > Java APIs ? Does anyone have idea how does the default jobtracker
>>> JSP
>>> > > > works
>>> > > > > ? we wanted to built something similar to this
>>> > > > >
>>> > > > > thanks
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > Nipen Mark
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Best Regards
>>> > > >
>>> > > > Jeff Zhang
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Nipen Mark
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards
>>> >
>>> > Jeff Zhang
>>> >
>>>
>>>
>>>
>>> --
>>> Nipen Mark
>>>
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



-- 
Best Regards

Jeff Zhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message