hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corbett Martin <comar...@nhin.com>
Subject Trouble starting up Task Tracker
Date Wed, 23 Jan 2013 17:28:51 GMT
Question

We're trying out Cloudera Manager and CDH4 in a Clustered deployment and having trouble getting
the Task Trackers to start up.

The error says (full stacktrace below)
2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker
because org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local directories
are writable
     at org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
     at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)

What directories is it referring to?  We have the following specified on the local file system.

ls -al /data/1/mapred/
drwxr-xr-x 4 mapred hadoop 4096 Jan 15 16:37 .
drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local

ls -al /data/2/mapred/
drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local

ls -al /data/3/mapred/
drwxr-xr-x 3 mapred hadoop 4096 Jan 15 16:37 .
drwx------ 4 hdfs   hdfs   4096 Jan 15 16:37 ..
drwxr-xr-x 2 mapred hadoop 4096 Jan 15 16:37 local

When we try to start the Map Reduce service from Cloudera Manager the Task Trackers fail to
start.  The Job Tracker starts fine.  Here's the error from the log file.

************************************************************/
2013-01-23 10:48:35,951 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG:   host = foo-dev-arch-02/10.204.12.63
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.0.0-mr1-cdh4.1.1
STARTUP_MSG:   classpath = /var/run/cloudera-scm-agent/process/499-mapreduce-TASKTRACKER:/usr/java/jdk1.6.0_30/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.1.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.1.1.jar:/usr/share/cmf/lib/plugins/event-publish-4.1.1-shaded.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.3-cdh4.1.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/hue-plugins-2.1.0-cdh4.1.1.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.1.1-tests.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/hadoop-auth.jar::
STARTUP_MSG:   build = file:///data/1/jenkins/workspace/generic-package-sles64-11/topdir/BUILD/hadoop-2.0.0-mr1-cdh4.1.1
-r Unknown; compiled by 'jenkins' on Tue Oct 16 11:50:57 PDT 2012
************************************************************/
2013-01-23 10:48:37,095 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
2013-01-23 10:48:37,189 INFO org.apache.hadoop.http.HttpServer: Added global filter 'safety'
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context task
2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2013-01-23 10:48:37,193 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2013-01-23 10:48:37,441 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker local dir /data/1/mapred/local
error File /data/1/mapred/local does not exist, removing from local dirs
2013-01-23 10:48:37,442 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker local dir /data/2/mapred/local
error File /data/2/mapred/local does not exist, removing from local dirs
2013-01-23 10:48:37,443 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker local dir /data/3/mapred/local
error File /data/3/mapred/local does not exist, removing from local dirs
2013-01-23 10:48:37,443 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker
because org.apache.hadoop.util.DiskChecker$DiskErrorException: No mapred local directories
are writable
     at org.apache.hadoop.mapred.TaskTracker$LocalStorage.checkDirs(TaskTracker.java:275)
     at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1693)
     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4006)

2013-01-23 10:48:37,446 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at foo-dev-arch-02/10.204.12.63
************************************************************/

Thanks in advance

~Corbett Martin
Software Architect
AbsoluteAR Accounts Receivable Services - An NHIN Solution


________________________________
This message and its contents (to include attachments) are the property of National Health
Systems, Inc. and may contain confidential and proprietary information. This email and any
files transmitted with it are intended solely for the use of the individual or entity to whom
they are addressed. You are hereby notified that any unauthorized disclosure, copying, or
distribution of this message, or the taking of any unauthorized action based on information
contained herein is strictly prohibited. Unauthorized use of information contained herein
may subject you to civil and criminal prosecution and penalties. If you are not the intended
recipient, you should delete this message immediately and notify the sender immediately by
telephone or by replying to this transmission.

Mime
View raw message