Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <911326967.1208810601491.JavaMail.jira@brutus>
Date: Mon, 21 Apr 2008 13:43:21 -0700 (PDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3280) virtual address space limits break
 streaming apps
In-Reply-To: <8271611.1208538501722.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591040#action_12591040 ] 

Allen Wittenauer commented on HADOOP-3280:
------------------------------------------

On Linux, we should be able to do this as part of /etc/security/limits.conf.  Solaris has the whole rctl subsystem.  I'm sure other UNIX and UNIX-like systems have similar things.  Point is, ops folks should be able to limit memory consumption without modifying hadoop at all, even the hadoop-env.sh file.

> virtual address space limits break streaming apps
> -------------------------------------------------
>
>                 Key: HADOOP-3280
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3280
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/streaming
>            Reporter: Rick Cox
>            Priority: Blocker
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-3280_0_20080418.patch
>
>
> HADOOP-2765 added a mandatory, hard virtual address space limit to streaming apps based on the Java process's -Xmx setting.
> This makes it impossible to run a 64-bit streaming app that needs large address spaces under a 32-bit JVM, even if one is otherwise willing to dramatically increase the -Xmx setting without cause. Also, unlike Java's -Xmx limit, the virtual address space limit for an arbitrary UNIX process does not necessarily correspond to RAM usage, so it's likely to be a relatively difficult to configure limit.
> 2765 was originally opened to allow an optional wrapper script around streaming tasks, one use case for which was setting a ulimit. That approach seems much less intrusive and more flexible than the final implementation. The ulimit can also be trivially set by the streaming task itself without any support from Hadoop.
> Marking this as an 0.17 blocker because it will break deployed apps and there is no workaround available.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.