hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drake민영근 <drake....@nexr.com>
Subject Re: Not able to run more than one map task
Date Wed, 15 Apr 2015 00:31:13 GMT
Hi, Amit.

Test these:
Increase "yarn.nodemanager.resource.memory-mb" beyond 8192. That's ok for
testing. And decrease "mapreduce.map.memory.mb" to 256 and add
"yarn.nodemanager.vmem-check-enabled" to "false" in yarn-site.xml.

Thanks.

Drake 민영근 Ph.D
kt NexR

On Sat, Apr 11, 2015 at 8:01 AM, Niels Basjes <Niels@basjes.nl> wrote:

> Just curious: what is the input for your job ? If it is a single gzipped
> file then that is the cause of getting exactly 1 mapper.
>
> Niels
>
> On Fri, Apr 10, 2015, 09:21 Amit Kumar <amitiitd@msn.com> wrote:
>
>> Thanks a lot Harsha for replying
>>
>> This problem has waster at least last one week.
>>
>> We tried what you suggested. Could you please take a look at the
>> configuration and suggest if we missed c?
>>
>> System RAM : 8GB
>> CPU : 4 threads each with 2 cores.
>> # Disks : 1
>>
>> MR2:
>>
>> mapreduce.map.memory.mb : 512
>> mapreduce.tasktracker.map.tasks.maximum : 4
>>
>> Yarn:
>>
>> yarn.app.mapreduce.am.resource.mb : 512
>> yarn.nodemanager.resource.cpu-vcores : 4
>> yarn.scheduler.minimum-allocation-mb : 512
>> yarn.nodemanager.resource.memory-mb : 5080
>>
>>
>> Regards,
>> Amit
>>
>>
>>
>> > From: harsh@cloudera.com
>> > Date: Fri, 10 Apr 2015 10:20:24 +0530
>> > Subject: Re: Not able to run more than one map task
>> > To: user@hadoop.apache.org
>>
>> >
>> > You are likely memory/vcore starved in the NM's configs. Increase your
>> > yarn.nodemanager.resource.memory-mb and
>> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
>> > MR job memory request values to gain more parallelism.
>> >
>> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <amitiitd@msn.com> wrote:
>> > > Hi All,
>> > >
>> > > We recently started working on Hadoop. We have setup the hadoop in
>> pseduo
>> > > distribution mode along with oozie.
>> > >
>> > > Every developer has set it up on his laptop. The problem is that we
>> are not
>> > > able to run more than one map task concurrently on our laptops.
>> Resource
>> > > manager is not allowing more than one task on our machine.
>> > >
>> > > My task gets completed if I submit it without Oozie. Oozie requires
>> one map
>> > > task for its own functioning. Actual task that oozie submit does not
>> start.
>> > >
>> > > Here is my configuration
>> > >
>> > > -- Hadoop setup in Pseudo distribution mode
>> > > -- Hadoop Version - 2.6
>> > > -- Oozie Version - 4.0.1
>> > >
>> > > Regards,
>> > > Amit
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>

Mime
View raw message