hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3759) Provide ability to run memory intensive jobs without affecting other running tasks on the nodes
Date Mon, 11 Aug 2008 14:49:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621466#action_12621466

Hemanth Yamijala commented on HADOOP-3759:

The latest patch adds a JUnit test class that works as follows:

- The test class defines a custom TaskScheduler that extends JobQueueTaskScheduler. This is
only a dummy scheduler, that overrides the assignTasks method in which it verifies the values
of the memory related variables that are reported by the tasktrackers. This is the core of
the test.
- It uses the SleepJob example as it just needs a dummy job to run.
- Each test case sets up a MiniMRCluster to use the custom TaskScheduler, adds a SleepJob
to it, and configures various values for the memory related configuration variables defined
- Then it lets the schedule's assignTasks API to verify that the free memory etc are computed
and reported correctly to the JobTracker.

In order for this test to work, I had to make a few changes to some core classes. Please comment
on whether these are reasonable. The changes are:

- In JobTracker, defined a package-private API  {code}TaskScheduler getTaskScheduler(){code}
- In MiniMRCluster, defined an API to get the JobTracker instance that is created
- Again in MiniMRCluster, starting the TaskTracker with a configured JobConf variable, rather
than null. I needed this because the TaskTracker should get the memory related configuration

> Provide ability to run memory intensive jobs without affecting other running tasks on
the nodes
> -----------------------------------------------------------------------------------------------
>                 Key: HADOOP-3759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3759
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.19.0
>         Attachments: HADOOP-3759.patch, HADOOP-3759.patch, HADOOP-3759.patch
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from affecting
Hadoop daemons and other tasks running on a node. A related requirement is that users be provided
an ability to run jobs which are memory intensive. The system must provide enough knobs to
allow such jobs to be run while still maintaining the requirements of HADOOP-3581.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message