hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "rahul k singh (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-5919) Memory management variables need a backwards compatibility option after HADOOP-5881
Date Mon, 08 Jun 2009 09:48:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717210#action_12717210
] 

rahul k singh edited comment on HADOOP-5919 at 6/8/09 2:46 AM:
---------------------------------------------------------------

All physical memory related stuff have been removed.

mapred.task.default.maxvmem => removed completely
mapred.tasktracker.vmem.reserved ==> this key is removed.Admin can attain similar behaviour
using keys mapred.cluster.map.memory.mb and mapred.cluster.map.memory.mb. 

mapred.task.limit.maxvmem   ==> split into  mapred.cluster.max.map.memory.mb mapred.cluster.max.reduce.memory.mb

for example if you set "mapred.task.limit.maxvmem" = 10 , then "mapred.cluster.max.map.memory.mb"
= 10/1024. same with reduce.

mapred.task.maxvmem ==> split into mapred.job.map.memory.mb,mapred.job.reduce.memory.mb.
The set behaviour would be same 
as "mapred.task.limit.maxvmem"

Design.
Assumption:

1. Incase there is 1-1 mapping of the key , the value of the deprecated key is applied to
the new key.

Data Structures.
{code}
deprecatedKeys<String,DeprecatedKeyMapping>
Skeleton of DeprecatedKeyMapping

static class DeprecatedKeyMapping{

   String[] keyMappings;
   String customMessage;
}
{code}

for keys which are removed , keyMappings would be null and keys would have only custom message.


while setting the conf , if there is 1-1 mapping ,Configuration would take care of that ,
incase of multiple values for a deprecated key , respective code paths would be setting the
value.


      was (Author: rksingh):
    All physical memory related stuff have been removed.

mapred.task.default.maxvmem => removed completely
mapred.tasktracker.vmem.reserved ==> this key is removed.Admin can attain similar behaviour
using keys mapred.cluster.map.memory.mb and mapred.cluster.map.memory.mb. 

mapred.task.limit.maxvmem   ==> split into  mapred.cluster.max.map.memory.mb mapred.cluster.max.reduce.memory.mb

for example if you set "mapred.task.limit.maxvmem" = 10 , then "mapred.cluster.max.map.memory.mb"
= 10/1024. same with reduce.

mapred.task.maxvmem ==> split into mapred.job.map.memory.mb,mapred.job.reduce.memory.mb.
The set behaviour would be same 
as "mapred.task.limit.maxvmem"

Design.
Assumption:

1. Incase there is 1-1 mapping of the key , the value of the deprecated key is applied to
the new key.

Data Structures.

deprecatedKeys<String,DeprecatedKeyMapping>
Skeleton of DeprecatedKeyMapping

static class DeprecatedKeyMapping{

   String[] keyMappings;
   String customMessage;
}


for keys which are removed , keyMappings would be null and keys would have only custom message.


while setting the conf , if there is 1-1 mapping ,Configuration would take care of that ,
incase of multiple values for a deprecated key , respective code paths would be setting the
value.

  
> Memory management variables need a backwards compatibility option after HADOOP-5881
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-5919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5919
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: rahul k singh
>            Priority: Blocker
>
> HADOOP-5881 modified variables related to memory management without looking at the backwards
compatibility angle. This JIRA is to adress the gap. Marking it a blocker for 0.20.1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message