hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3708) Provide accounting functionality for Hadoop resource manager
Date Wed, 22 Oct 2008 22:47:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Matei Zaharia updated HADOOP-3708:

    Attachment: history-scripts.patch

I've attached a patch with the scripts I built at Facebook. There is also a readme with an
explanation of what they do. Basically there are two scripts: One that just parses the history
logs and jobconfs and puts the exact same data into MySQL, creating tables of jobs, jobconf
XML key-value pairs, tasks and task attempts, and then a second script that performs some
joins on these tables to create a set of job summary reports with ~1 KB of data per job that
can be used to run queries quickly for the purposes of visualization. You can build a visualization
based on this database using your favorite tool. Unfortunately I can't open source the one
used at Facebook because it depends on a lot of internal web libraries.

> Provide accounting functionality for Hadoop resource manager
> ------------------------------------------------------------
>                 Key: HADOOP-3708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3708
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: contrib/capacity-sched
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.20.0
>         Attachments: history-scripts.patch
> HADOOP-3421 describes requirements for a new resource manager in Hadoop to schedule Map/Reduce
jobs. In production systems, it would be useful to produce accounting information related
to the scheduling - such as job start and run times, resources used, etc. This information
can be consumed by other systems to build accounting for shared resources. This JIRA is for
tracking the requirements, approach and implementation for producing accounting information.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message