hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brice Arnould (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3581) Prevent memory intensive user tasks from taking down nodes
Date Fri, 11 Jul 2008 13:44:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612873#action_12612873

Brice Arnould commented on HADOOP-3581:

Hi !
I think that there is a more general problem, that is task insulation, because a bugged process
could many other things than just overloading the memory.

The userBasedInsulator.sh that I proposed in [HADOOP-3675] could solve this issue (and a few
others) in an easier way. We don't need [HADOOP-3675] to be complete to introduce that approch
The idea is to provide a "wrapper" charged to enforce local policies. This wrapper can be
written as a shell script to work on must Unix (and maybe cygwin), and requires much less
change to the core of Hadoop, that is :
+      ArrayList<String> vargs = new ArrayList<String>(8);
+      //  Check for the eventual wrapper script
+      final String wrapper = conf.get("mapred.child.wrapper");
+      if (wrapper != null)
+        vargs.add(wrapper);
-      Vector<String> vargs = new Vector<String>(8);

> Prevent memory intensive user tasks from taking down nodes
> ----------------------------------------------------------
>                 Key: HADOOP-3581
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3581
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: patch_3581_0.1.txt
> Sometimes user Map/Reduce applications can get extremely memory intensive, maybe due
to some inadvertent bugs in the user code, or the amount of data processed. When this happens,
the user tasks start to interfere with the proper execution of other processes on the node,
including other Hadoop daemons like the DataNode and TaskTracker. Thus, the node would become
unusable for any Hadoop tasks. There should be a way to prevent such tasks from bringing down
the node.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message