hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2390) JobTracker and TaskTrackers fail with a misleading error if one of the mapreduce.cluster.dir has unusable permissions / is unavailable.
Date Sat, 11 Jun 2011 18:37:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047965#comment-13047965
] 

Harsh J commented on MAPREDUCE-2390:
------------------------------------

MAPREDUCE-1382 fixes this, but what's odd is that:

JT says this:
{code}
2011-06-12 00:00:34,924 WARN org.apache.hadoop.util.MRAsyncDiskService: Cannot create toBeDeleted
in /Users/harshchouraria/Work/installs/temp-space/mapred/local1. Ignored.
{code}

TT does this instead:
{code}
2011-06-12 00:00:35,980 WARN org.apache.hadoop.util.DiskChecker: Incorrect permissions were
set on /Users/harshchouraria/Work/installs/temp-space/mapred/local1, expected: rwxr-xr-x,
while actual: ---------. Fixing...
{code}

May be worth making JT do this too, if its gonna use {{mapred.local.dir}} (or w/e its new
alias is) after all.

> JobTracker and TaskTrackers fail with a misleading error if one of the mapreduce.cluster.dir
has unusable permissions / is unavailable.
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2390
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2390
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker, tasktracker
>    Affects Versions: 0.20.2
>         Environment: CDH3 and Apache 0.20 || Linux
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: configuration, directory-permissions, mapreduce
>
> To reproduce, have a mapred.local.dir property set to a few directories. Before starting
up the JT, set one of these directories' permission as 'd---------', and then start the JT/TT.
The JT, although it tries to ignore this directory, fails with an odd and misleading message
claiming that its configured address in use.
> Fixing the permission clears this issue!
> This was also reported in the mailing lists by Ted Yu, quite a few months ago. But I
had forgotten about filing a bug for it here. Still seems to happen. A log is attached below.
> {code}
> 2011-03-17 00:40:32,321 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker:
java.io.IOException: Cannot create toBeDeleted in /home/hack/.tmplocalz/2
>         at org.apache.hadoop.util.MRAsyncDiskService.<init>(MRAsyncDiskService.java:86)
>         at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2189)
>         at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2022)
>         at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:276)
>         at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:268)
>         at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4712)
> 2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
> 2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
> 2011-03-17 00:40:33,322 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
> 2011-03-17 00:40:33,322 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured
with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks)
(-1, -1, -1, -1)
> 2011-03-17 00:40:33,322 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts
(include/exclude) list
> 2011-03-17 00:40:33,350 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker
with owner as hack
> 2011-03-17 00:40:33,351 FATAL org.apache.hadoop.mapred.JobTracker: java.net.BindException:
Problem binding to localhost/127.0.0.1:8021 : Address already in use
>         at org.apache.hadoop.ipc.Server.bind(Server.java:227)
>         at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:314)
>         at org.apache.hadoop.ipc.Server.<init>(Server.java:1411)
>         at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:510)
>         at org.apache.hadoop.ipc.RPC.getServer(RPC.java:471)
>         at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2112)
>         at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2022)
>         at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:276)
>         at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:268)
>         at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4712)
> Caused by: java.net.BindException: Address already in use
>         at sun.nio.ch.Net.bind(Native Method)
>         at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
>         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>         at org.apache.hadoop.ipc.Server.bind(Server.java:225)
>         ... 9 more
> 2011-03-17 00:40:33,352 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG: 
> /************************************************************
> SHUTDOWN_MSG: Shutting down JobTracker at QDuo/127.0.0.1
> ************************************************************/
> {code}
> The list conversation in context, at {{search-hadoop.com}}:
> http://search-hadoop.com/m/FzN7iqreL/problem+starting+cdh3b2+jobtracker&subj=problem+starting+cdh3b2+jobtracker
> I'll try to investigate and post the exact problem / solution soon.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message