hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rick Cox (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-3280) virtual address space limits break streaming apps
Date Fri, 18 Apr 2008 17:08:21 GMT
virtual address space limits break streaming apps

                 Key: HADOOP-3280
                 URL: https://issues.apache.org/jira/browse/HADOOP-3280
             Project: Hadoop Core
          Issue Type: Bug
          Components: contrib/streaming
            Reporter: Rick Cox
            Priority: Blocker
             Fix For: 0.17.0

HADOOP-2765 added a mandatory, hard virtual address space limit to streaming apps based on
the Java process's -Xmx setting.

This makes it impossible to run a 64-bit streaming app that needs large address spaces under
a 32-bit JVM, even if one is otherwise willing to dramatically increase the -Xmx setting without
cause. Also, unlike Java's -Xmx limit, the virtual address space limit for an arbitrary UNIX
process does not necessarily correspond to RAM usage, so it's likely to be a relatively difficult
to configure limit.

2765 was originally opened to allow an optional wrapper script around streaming tasks, one
use case for which was setting a ulimit. That approach seems much less intrusive and more
flexible than the final implementation. The ulimit can also be trivially set by the streaming
task itself without any support from Hadoop.

Marking this as an 0.17 blocker because it will break deployed apps and there is no workaround

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message