hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip Zeyliger <phi...@cloudera.com>
Subject Re: Setting thread stack size for child JVM
Date Fri, 08 May 2009 22:18:35 GMT
On Fri, May 8, 2009 at 1:11 PM, Ken Krugler <kkrugler_lists@transpac.com>wrote:

>  You an set the mapred.child.java.opts on a per job basis
>> either via -D mapred.child.java.ops="java options" or via
>> conf.set("mapred.child.java.opts", "java options").
>>
>> Note: the conf.set must be done before the job is submitted.
>>
>> On Fri, May 8, 2009 at 11:57 AM, Philip Zeyliger <philip@cloudera.com
>> >wrote:
>>
>>   You could add "-Xss<n>" to the "mapred.child.java.opts" configuration
>>>  setting.  That's controlling the Java stack size, which I think is the
>>>  relevant bit for you.
>>>
>>
> That's part of it, but there's also native memory used when you start a
> thread with most JREs.


It doesn't look like Hadoop lets you run an arbitrary ulimit command, but if
you take a look at Shell.getUlimitMemoryCommand(conf) (called from
TaskRunner.java), you'll see that it lets you specify "ulimit -v N"
commands.  You could probably augment that for "ulimit -s" pretty easily.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message