hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DSuiter RDX <dsui...@rdx.com>
Subject Re: only one map or reduce job per time on one node
Date Wed, 06 Nov 2013 14:43:28 GMT
I suspect that the reason no-one is responding with good answers is that
fundamentally, it seems like what you are trying to do runs against the
reason Hadoop is designed the way it is. A parallel process framework is
defeated if you force it to not work concurrently...

Maybe you should look into Oozie, and set a workflow that runs your map
task as a map-only job, and then do the calculation that the reducer would
do in a different program?

There is a Hadoop parameter that allows for task JVM reuse that might solve
your problem also.

Or, add a combiner to your mr job if there is not one already, and then try
dropping the maximum heap sizes of the respective slots?

*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com

On Wed, Nov 6, 2013 at 5:09 AM, John <johnnyenglish739@gmail.com> wrote:

> Does no one have an idea if it is possible? In my example the heapsize of
> the tasktracker is 2gb. The max map heap size is 1gb and the mac heap size
> of the reduce process 1,5gb. If there are running 2 maps or just the reduce
> job everything is fine. But if there are running 2 maps + 1 reduce process
> I got a "Error: java.lang.OutOfMemoryError: Java heap space at" message.
> kind regards
> 2013/11/5 John <johnnyenglish739@gmail.com>
>> Because my node swaps the memory if the 2 map slots + 1 reduce is
>> occupied with my job. Sure I can minimize the max memory for the map/reduce
>> process. I tried this already, but I got a out of memory exception if set
>> the max heap size for the map/reduce process to low for my mr job.
>> kind regards
>> 2013/11/5 Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
>>> Why do you want to do this?
>>> +Vinod
>>> On Nov 5, 2013, at 9:17 AM, John wrote:
>>> Is it possible to force the jobtracker executing only 2 map jobs or 1
>>> reduce job per time?
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.

View raw message