hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark N <nipen.m...@gmail.com>
Subject Job Tracker questions
Date Wed, 03 Feb 2010 12:34:51 GMT
We have a hadoop job running and have used custom counters to track  few
counters ( like no of successfully processed documents matching certain
conditions)


Since we need to get this counters even while the Hadoop job is running , we
wrote another Java program to read these counters


*Counter reader  program *will do the following :


1)      List all the running jobs.

2)   Get the running job using Job name

2)     Get all the counter for individual running jobs

3)      Set this counters in variables.
        We could successfully read these counters  , but since we need to
show these counters to custom UI , how can we show these counters?

        we looked into various options to read these counters to show in UI
as following :

      1. Dump these counters to database , however this may be overhead
      2. Write web service   and UI will invoke the functions from these
service to show in UI ( However since we need to run "*Counter reader
program "  *with Hadoop command it might not be feasible to write web
service ?   )

      so the question is can we achive to read the counters using simple
Java APIs ? Does anyone have idea how does the default jobtracker JSP works
? we wanted to built something similar to this

thanks



-- 
Nipen Mark

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message