From general-return-1019-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 04 11:00:13 2010 Return-Path: Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: (qmail 49345 invoked from network); 4 Feb 2010 11:00:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 4 Feb 2010 11:00:12 -0000 Received: (qmail 72771 invoked by uid 500); 4 Feb 2010 11:00:09 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 72210 invoked by uid 500); 4 Feb 2010 11:00:09 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Received: (qmail 72149 invoked by uid 99); 4 Feb 2010 11:00:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 11:00:09 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nipen.mark@gmail.com designates 209.85.217.209 as permitted sender) Received: from [209.85.217.209] (HELO mail-gx0-f209.google.com) (209.85.217.209) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 11:00:02 +0000 Received: by gxk1 with SMTP id 1so2312276gxk.11 for ; Thu, 04 Feb 2010 02:59:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=cXwWGokMwVbpBJSvrXSy0AaMMmcxz8S1KhUZcnOiG/I=; b=oafuYCINu2TqOV79fXTs/y9JTzmCGJZlzuJtVNFSO3SdAtd06HMYwgHRv+3zTF5LRf wyV/D2qegMJfC6t816O/0P9IAIivgC/Bc/FsPmReYtOlTBhoeRRAloA3Z/Kyw9X+SIU9 4KIHP9wu23YD3D/bPbPWDYDm6EB1ve47Yvv9c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=ndyQGNZYAuUmWy+DMTWked5a6tk8PyUeIgzOtpjCp2YujLJyjpwFd7oHypWHXrGGce yZtlg9CEI0D9cfPLhuFVUtMkqHsWCTyyhVgofJP6VRSF+SNdm5eU4U3B0kCUTMctx57l qA56i0eKa3NRu5MkdvLr5F3c1HzrLiCRxv3JA= MIME-Version: 1.0 Received: by 10.91.164.39 with SMTP id r39mr1007306ago.113.1265281179432; Thu, 04 Feb 2010 02:59:39 -0800 (PST) In-Reply-To: <8211a1321002030648h565edc02yf8e41bdc2ca51cc@mail.gmail.com> References: <8211a1321002030648h565edc02yf8e41bdc2ca51cc@mail.gmail.com> Date: Thu, 4 Feb 2010 16:44:39 +0545 Message-ID: Subject: Re: Job Tracker questions From: Mark N To: general@hadoop.apache.org Cc: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001485f5ea343db781047ec43b19 --001485f5ea343db781047ec43b19 Content-Type: text/plain; charset=ISO-8859-1 Ye currently am using jobclient to read these counters. But We are not able to use *webservices *because the jar which is used to read the counters from running hadoop job is itself a Hadoop program If we could have pure Java Api which is run without hadoop command then we could return the counter variable into webservices and show in UI. Any help or technique to show thsese counters in the UI would be appreciated ( not necessarily using web service ) I am using webservices because I am having .net VB client thanks On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang wrote: > I think you can use JobClient to get the counters in your web service. > If you look at the shell script bin/hadoop, you will find that actually > this > shell use the JobClient to get the counters. > > > > On Wed, Feb 3, 2010 at 4:34 AM, Mark N wrote: > > > We have a hadoop job running and have used custom counters to track few > > counters ( like no of successfully processed documents matching certain > > conditions) > > > > > > Since we need to get this counters even while the Hadoop job is running , > > we > > wrote another Java program to read these counters > > > > > > *Counter reader program *will do the following : > > > > > > 1) List all the running jobs. > > > > 2) Get the running job using Job name > > > > 2) Get all the counter for individual running jobs > > > > 3) Set this counters in variables. > > We could successfully read these counters , but since we need to > > show these counters to custom UI , how can we show these counters? > > > > we looked into various options to read these counters to show in > UI > > as following : > > > > 1. Dump these counters to database , however this may be overhead > > 2. Write web service and UI will invoke the functions from these > > service to show in UI ( However since we need to run "*Counter reader > > program " *with Hadoop command it might not be feasible to write web > > service ? ) > > > > so the question is can we achive to read the counters using simple > > Java APIs ? Does anyone have idea how does the default jobtracker JSP > works > > ? we wanted to built something similar to this > > > > thanks > > > > > > > > -- > > Nipen Mark > > > > > > -- > Best Regards > > Jeff Zhang > -- Nipen Mark --001485f5ea343db781047ec43b19--