hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <omal...@apache.org>
Subject Re: how to start the customized Child JVM task process of m/r?
Date Tue, 26 Apr 2011 15:05:15 GMT
On Tue, Apr 26, 2011 at 6:02 AM, Juwei Shi <shijuwei@gmail.com> wrote:

As you know, Hadoop MapReduce start child JVM processes to run task. I want
> to start the process of m/r task myself thus I can pass some OS level
> parameters to the JVM process.

That currently isn't possible. The best that is available
is mapred.child.java.opts, which adds options to the child jvm.
Alternatively, you could fork a subprocess out of the map or reduce like
streaming does. Actually, I guess you could use streaming itself and get
control over how your code runs.

In MapReduce v2, which is described here
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ ,
the user gets a lot more control over the task launching. In particular,
mapreduce becomes a user-land library controlled by the user.

-- Owen

View raw message