hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: MR job fails with too many mappers
Date Tue, 18 Nov 2014 16:59:35 GMT
If log aggregation is enabled, log folder will be deleted. So I suggest disable “yarn.log-aggregation-enable”
and run job again. All the logs remains at log folder. Then you can find container logs

Thanks & Regards
Rohith Sharma K S

This e-mail and its attachments contain confidential information from HUAWEI, which is intended
only for the person or entity whose address is listed above. Any use of the information contained
herein in any way (including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive
this e-mail in error, please notify the sender by phone or email immediately and delete it!

From: francexo83 [mailto:francexo83@gmail.com]
Sent: 18 November 2014 22:15
To: user@hadoop.apache.org
Subject: Re: MR job fails with too many mappers

Hi,

thank you for your quick response, but I was not able to see the logs for the container.

I get a  "no such file or directory" when I try to access the logs of the container from the
shell:

cd /var/log/hadoop-yarn/containers/application_1416304409718_0032


It seems that the container has never been created.



thanks





2014-11-18 16:43 GMT+01:00 Rohith Sharma K S <rohithsharmaks@huawei.com<mailto:rohithsharmaks@huawei.com>>:
Hi

Could you get syserr and sysout log for contrainer.? These logs will be available in the same
location  syslog for container.
${yarn.nodemanager.log-dirs}/<app-id>/<container-id>
This helps to find problem!!


Thanks & Regards
Rohith Sharma K S

From: francexo83 [mailto:francexo83@gmail.com<mailto:francexo83@gmail.com>]
Sent: 18 November 2014 20:53
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: MR job fails with too many mappers

Hi All,

I have a small  hadoop cluster with three nodes and HBase 0.98.1 installed on it.

The hadoop version is 2.3.0 and below my use case scenario.

I wrote a map reduce program that reads data from an hbase table and does some transformations
on these data.
Jobs are very simple so they didn't need the  reduce phase. I also wrote a TableInputFormat
 extension in order to maximize the number of concurrent maps on the cluster.
In other words, each  row should be processed by a single map task.

Everything goes well until the number of rows and consequently  mappers exceeds 300000 quota.

This is the only exception I see when the job fails:

Application application_1416304409718_0032 failed 2 times due to AM Container for appattempt_1416304409718_0032_000002
exited with exitCode: 1 due to:


Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1


Cluster configuration details:
Node1: 12 GB, 4 core
Node2: 6 GB, 4 core
Node3: 6 GB, 4 core

yarn.scheduler.minimum-allocation-mb=2048
yarn.scheduler.maximum-allocation-mb=4096
yarn.nodemanager.resource.memory-mb=6144



Regards

Mime
View raw message