hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2765) setting memory limits for tasks
Date Wed, 27 Feb 2008 07:12:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572808#action_12572808

Hadoop QA commented on HADOOP-2765:

-1 overall.  Here are the results of testing the latest attachment 
against trunk revision 619744.

    @author +1.  The patch does not contain any @author tags.

    tests included -1.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no tests are needed for this patch.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new javac compiler warnings.

    release audit +1.  The applied patch does not generate any new release audit warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests +1.  The patch passed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1840/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1840/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1840/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1840/console

This message is automatically generated.

> setting memory limits for tasks
> -------------------------------
>                 Key: HADOOP-2765
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2765
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: contrib/streaming
>    Affects Versions: 0.15.3
>            Reporter: Joydeep Sen Sarma
>            Assignee: Amareshwari Sri Ramadasu
>             Fix For: 0.17.0
>         Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt,
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing
down nodes. this setting is much lower than the max. memory that can be used (since most likely
these tend to be scripting bugs). At the same time - for careful users, we want to be able
to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they
are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting
a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process
will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a
stream.wrapper option. the value of this option is a program through which streaming mapper
and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit
and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory
usage - but users can easily override this wrapper (eg by invoking it with different memory
limit argument). we can use the wrapper for other system wide resource limits (or any environment
settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This
setup has saved our ass on many occasions while allowing sophisticated users to use high memory
> Can submit patch if this sounds interesting.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message