hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3280) virtual address space limits break streaming apps
Date Tue, 29 Apr 2008 22:47:55 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley updated HADOOP-3280:
----------------------------------

      Resolution: Fixed
    Release Note: This patch adds the mapred.child.ulimit to limit the virtual memory for
children processes to the given value.
    Hadoop Flags: [Incompatible change, Reviewed]
          Status: Resolved  (was: Patch Available)

I just committed this. Thanks, Arun!

> virtual address space limits break streaming apps
> -------------------------------------------------
>
>                 Key: HADOOP-3280
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3280
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/streaming
>            Reporter: Rick Cox
>            Assignee: Arun C Murthy
>            Priority: Blocker
>             Fix For: 0.17.0
>
>         Attachments: HADOOP-3280_0_20080418.patch, HADOOP-3280_1_20080423.patch, HADOOP-3280_2_20080424.patch,
HADOOP-3280_3_20080425.patch, patch-3280-1.txt, patch-3280-2.txt, patch-3280-3.txt, patch-3280.txt,
patch-3280.txt
>
>
> HADOOP-2765 added a mandatory, hard virtual address space limit to streaming apps based
on the Java process's -Xmx setting.
> This makes it impossible to run a 64-bit streaming app that needs large address spaces
under a 32-bit JVM, even if one is otherwise willing to dramatically increase the -Xmx setting
without cause. Also, unlike Java's -Xmx limit, the virtual address space limit for an arbitrary
UNIX process does not necessarily correspond to RAM usage, so it's likely to be a relatively
difficult to configure limit.
> 2765 was originally opened to allow an optional wrapper script around streaming tasks,
one use case for which was setting a ulimit. That approach seems much less intrusive and more
flexible than the final implementation. The ulimit can also be trivially set by the streaming
task itself without any support from Hadoop.
> Marking this as an 0.17 blocker because it will break deployed apps and there is no workaround
available.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message