hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Trouble starting up Task Tracker
Date Wed, 23 Jan 2013 17:36:09 GMT
This is the problem:

drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..

Your /data/1 directory seems to be owned by hdfs and restricted only
to it (700). I'm not sure this is necessary and you can perhaps make
it 755 at least.

Or perhaps what you may have is a misconfig wherein you've set your DN
config of dfs.datanode.data.dir to a set of /data/{1..n} instead of
the regular /data/{1..n}/dn or so. The DN has a feature where it
applies dfs.datanode.data.dir.perm (700) to its configured disks for
security - thereby marking all of your mount entry points as 700 and
causing trouble with the mapred.local.dir config of
/data/{1..n}/mapred which resides under it.

On Wed, Jan 23, 2013 at 10:58 PM, Corbett Martin <comartin@nhin.com> wrote:
> Question
>
>
>
> We’re trying out Cloudera Manager and CDH4 in a Clustered deployment and
> having trouble getting the Task Trackers to start up.
>
>
>
> The error says (full stacktrace below)
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
>      at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
>      at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
>      at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> What directories is it referring to?  We have the following specified on the
> local file system.
>
>
>
> ls -al /data/1/mapred/
>
> drwxr-xr-x 4 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/2/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> ls –al /data/3/mapred/
>
> drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
>
> drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
>
> drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local
>
>
>
> When we try to start the Map Reduce service from Cloudera Manager the Task
> Trackers fail to start.  The Job Tracker starts fine.  Here’s the error from
> the log file.
>
>
>
> ************************************************************/
>
> 2013-01-23 10:48:35,951 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
>
> /************************************************************
>
> STARTUP_MSG: Starting TaskTracker
>
> STARTUP_MSG:   host = foo-dev-arch-02/10.204.12.63
>
> STARTUP_MSG:   args = []
>
> STARTUP_MSG:   version = 2.0.0-mr1-cdh4.1.1
>
> STARTUP_MSG:   classpath =
> /var/run/cloudera-scm-agent/process/499-mapreduce-TASKTRACKER:/usr/java/jdk1.6.0_30/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.1.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/hue-plugins-2.1.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-auth.jar::
>
> STARTUP_MSG:   build =
> file:///data/1/jenkins/workspace/generic-package-sles64-11/topdir/BUILD/hadoop-2.0.0-mr1-cdh4.1.1
> -r Unknown; compiled by 'jenkins' on Tue Oct 16 11:50:57 PDT 2012
>
> ************************************************************/
>
> 2013-01-23 10:48:37,095 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
>
> 2013-01-23 10:48:37,189 INFO org.apache.hadoop.http.HttpServer: Added global
> filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context task
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
>
> 2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
>
> 2013-01-23 10:48:37,441 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/1/mapred/local error File /data/1/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,442 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/2/mapred/local error File /data/2/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker local dir /data/3/mapred/local error File /data/3/mapred/local
> does not exist, removing from local dirs
>
> 2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not
> start task tracker because
> org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local
> directories are writable
>
>      at
> org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
>
>      at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
>
>      at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)
>
>
>
> 2013-01-23 10:48:37,446 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down TaskTracker at foo-dev-arch-02/10.204.12.63
>
> ************************************************************/
>
>
>
> Thanks in advance
>
>
>
> ~Corbett Martin
>
> Software Architect
>
> AbsoluteAR Accounts Receivable Services - An NHIN Solution
>
>
>
>
> ________________________________
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action based
> on information contained herein is strictly prohibited. Unauthorized use of
> information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you should
> delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.



-- 
Harsh J

Mime
View raw message