hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juwei Shi <shiju...@gmail.com>
Subject Re: how to start the customized Child JVM task process of m/r?
Date Tue, 26 Apr 2011 15:52:43 GMT
Owen, thanks a lot!

Is is possible to do this by changing some source code of current MR?

2011/4/26 Owen O'Malley <omalley@apache.org>

> On Tue, Apr 26, 2011 at 6:02 AM, Juwei Shi <shijuwei@gmail.com> wrote:
> As you know, Hadoop MapReduce start child JVM processes to run task. I want
>> to start the process of m/r task myself thus I can pass some OS level
>> parameters to the JVM process.
> That currently isn't possible. The best that is available
> is mapred.child.java.opts, which adds options to the child jvm.
> Alternatively, you could fork a subprocess out of the map or reduce like
> streaming does. Actually, I guess you could use streaming itself and get
> control over how your code runs.
> In MapReduce v2, which is described here
> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ ,
> the user gets a lot more control over the task launching. In particular,
> mapreduce becomes a user-land library controlled by the user.
> -- Owen

- Juwei Shi (史巨伟)

View raw message