hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: Job fails if I change HADOOP_USER_NAME
Date Mon, 24 Mar 2014 03:18:51 GMT
Hi Ashwin,

How I enable debug for AM container logs ?
>>  Set below configurations changing log level for AM, map and reducer task. Default
values are INFO.

and to which location are they written to ?
>> These are written into  {yarn.nodemanager.log-dirs}/<application-id> while
executing job.
      Once application is finished,

1.       If log aggregation is enabled, then all container logs aggregated to HDFS. The log
path in hdfs is {yarn.nodemanager.remote-app-log-dir}/${user}

2.       If log aggregation is disabled,then all container logs remain in local machine where
containers has run i.e {yarn.nodemanager.log-dirs}/<application-id.

Thanks & Regards
Rohith Sharma K S

From: Ashwin Shankar [mailto:ashwinshankar77@gmail.com]
Sent: 22 March 2014 03:38
To: user@hadoop.apache.org
Subject: Re: Job fails if I change HADOOP_USER_NAME

Hi Rohit,
How I enable debug for AM container logs ? and to which location are they written to ?
I tried changing log4j.prop and can see DEBUGs for RM,NM etc but I don't see AM related debug


On Fri, Mar 21, 2014 at 3:05 AM, Rohith Sharma K S <rohithsharmaks@huawei.com<mailto:rohithsharmaks@huawei.com>>

The below stack trace is generic for any am launcher failed to launch. Can debug on AM container
logs, so get proper stacktrace.?

Thanks & Regards
Rohith Sharma K S

From: Ashwin Shankar [mailto:ashwinshankar77@gmail.com<mailto:ashwinshankar77@gmail.com>]
Sent: 21 March 2014 14:02
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Job fails if I change HADOOP_USER_NAME

I'm writing a new feature in Fair scheduler and wanted to test it out
by running jobs submitted by different users from my laptop.

My sleep job runs fine as long as the user name is my mac user name.
If I change my hadoop user name by setting HADOOP_USER_NAME,
my jobs fail with the exception org.apache.hadoop.util.Shell$ExitCodeException.
I also tried creating a new user account on my laptop and running a job as that user but I
get the same exception.

Please let me know if any of you have come across this.
I tried changing ulimits max proc(to 1024),but doesn't solve the problem.

Here is the stack trace :

Job job_1395389889916_0001 failed with state FAILED due to: Application application_1395389889916_0001
failed 3 times due to AM Container for appattempt_1395389889916_0001_000003 exited with  exitCode:
1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
            at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
            at org.apache.hadoop.util.Shell.run(Shell.java:418)
            at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
            at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)



View raw message