hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth mathur <sidh1...@gmail.com>
Subject Issues Running Hadoop 1.1.2 on multi-node cluster
Date Tue, 09 Jul 2013 16:56:02 GMT
Hi,

I have installed Hadoop 1.1.2 on a 5 nodes cluster. I installed it watching
this tutorial *
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
*

When I startup the hadoop, I get the folloing error in *all* the
tasktrackers.

"
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051611_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker:* Failed
to get system directory*...
2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
"

*But everything looks fine in the webUI. *

When I run a job, I get the following error but the job completes anyways.
I have* attached the* *screenshots* of the maptask failed error log in the
UI.

*"*
13/07/09 12:29:37 INFO input.FileInputFormat: Total input paths to process
: 2
13/07/09 12:29:37 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
13/07/09 12:29:37 WARN snappy.LoadSnappy: Snappy native library not loaded
13/07/09 12:29:37 INFO mapred.JobClient: Running job: job_201307091215_0001
13/07/09 12:29:38 INFO mapred.JobClient:  map 0% reduce 0%
13/07/09 12:29:41 INFO mapred.JobClient: Task Id :
attempt_201307091215_0001_m_000001_0, Status : FAILED
Error initializing attempt_201307091215_0001_m_000001_0:
ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
    at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
    at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
    at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
    at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
    at
org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
    at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
    at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
    at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
    at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
    at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
    at java.lang.Thread.run(Thread.java:724)

13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stdout
13/07/09 12:29:41 WARN mapred.JobClient: Error reading task
outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000001_0&filter=stderr
13/07/09 12:29:45 INFO mapred.JobClient:  map 50% reduce 0%
13/07/09 12:29:53 INFO mapred.JobClient:  map 50% reduce 16%
13/07/09 12:30:38 INFO mapred.JobClient: Task Id :
attempt_201307091215_0001_m_000000_1, Status : FAILED
Error initializing attempt_201307091215_0001_m_000000_1:
ENOENT: No such file or directory
    at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
    at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
    at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
    at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
    at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
    at
org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
    at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
    at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1331)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
    at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1306)
    at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1221)
    at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2581)
    at java.lang.Thread.run(Thread.java:724)

13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stdout
13/07/09 12:30:38 WARN mapred.JobClient: Error reading task
outputhttp://dmkd-1:50060/tasklog?plaintext=true&attemptid=attempt_201307091215_0001_m_000000_1&filter=stderr
13/07/09 12:30:44 INFO mapred.JobClient:  map 100% reduce 16%
13/07/09 12:30:50 INFO mapred.JobClient:  map 100% reduce 100%
13/07/09 12:30:52 INFO mapred.JobClient: Job complete: job_201307091215_0001
13/07/09 12:30:52 INFO mapred.JobClient: Counters: 30
13/07/09 12:30:52 INFO mapred.JobClient:   Job Counters
13/07/09 12:30:52 INFO mapred.JobClient:     Launched reduce tasks=1
13/07/09 12:30:52 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=68494
13/07/09 12:30:52 INFO mapred.JobClient:     Total time spent by all
reduces waiting after reserving slots (ms)=0
13/07/09 12:30:52 INFO mapred.JobClient:     Total time spent by all maps
waiting after reserving slots (ms)=0
13/07/09 12:30:52 INFO mapred.JobClient:     Rack-local map tasks=1
13/07/09 12:30:52 INFO mapred.JobClient:     Launched map tasks=5
13/07/09 12:30:52 INFO mapred.JobClient:     Data-local map tasks=4
13/07/09 12:30:52 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=64711
13/07/09 12:30:52 INFO mapred.JobClient:   File Output Format Counters
13/07/09 12:30:52 INFO mapred.JobClient:     Bytes Written=51
13/07/09 12:30:52 INFO mapred.JobClient:   FileSystemCounters
13/07/09 12:30:52 INFO mapred.JobClient:     FILE_BYTES_READ=111
13/07/09 12:30:52 INFO mapred.JobClient:     HDFS_BYTES_READ=271
13/07/09 12:30:52 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=153612
13/07/09 12:30:52 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=51
13/07/09 12:30:52 INFO mapred.JobClient:   File Input Format Counters
13/07/09 12:30:52 INFO mapred.JobClient:     Bytes Read=45
13/07/09 12:30:52 INFO mapred.JobClient:   Map-Reduce Framework
13/07/09 12:30:52 INFO mapred.JobClient:     Map output materialized
bytes=117
13/07/09 12:30:52 INFO mapred.JobClient:     Map input records=2
13/07/09 12:30:52 INFO mapred.JobClient:     Reduce shuffle bytes=117
13/07/09 12:30:52 INFO mapred.JobClient:     Spilled Records=20
13/07/09 12:30:52 INFO mapred.JobClient:     Map output bytes=85
13/07/09 12:30:52 INFO mapred.JobClient:     Total committed heap usage
(bytes)=602996736
13/07/09 12:30:52 INFO mapred.JobClient:     CPU time spent (ms)=2560
13/07/09 12:30:52 INFO mapred.JobClient:     Combine input records=10
13/07/09 12:30:52 INFO mapred.JobClient:     SPLIT_RAW_BYTES=226
13/07/09 12:30:52 INFO mapred.JobClient:     Reduce input records=10
13/07/09 12:30:52 INFO mapred.JobClient:     Reduce input groups=8
13/07/09 12:30:52 INFO mapred.JobClient:     Combine output records=10
13/07/09 12:30:52 INFO mapred.JobClient:     Physical memory (bytes)
snapshot=520691712
13/07/09 12:30:52 INFO mapred.JobClient:     Reduce output records=8
13/07/09 12:30:52 INFO mapred.JobClient:     Virtual memory (bytes)
snapshot=3025334272
13/07/09 12:30:52 INFO mapred.JobClient:     Map output records=10

*"*


Following is the complete tasktracker log on one node after starting the
hadoop (*not running the job*):

*"*
2013-07-09 12:15:15,426 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG:   host = dmkd-1/172.31.21.10
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.1.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r
1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
2013-07-09 12:15:15,722 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2013-07-09 12:15:15,823 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2013-07-09 12:15:15,824 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-07-09 12:15:15,824 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
system started
2013-07-09 12:15:16,089 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2013-07-09 12:15:16,223 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
registered.
2013-07-09 12:15:16,230 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
exists!
2013-07-09 12:15:21,528 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2013-07-09 12:15:21,600 INFO org.apache.hadoop.http.HttpServer: Added
global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-07-09 12:15:21,631 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-07-09 12:15:21,636 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tasktracker with owner as hduser
2013-07-09 12:15:21,637 INFO org.apache.hadoop.mapred.TaskTracker: Good
mapred local directories are: /app/hadoop/tmp/mapred/local
2013-07-09 12:15:21,652 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
registered.
2013-07-09 12:15:21,653 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
TaskTrackerMetrics registered.
2013-07-09 12:15:21,680 INFO org.apache.hadoop.ipc.Server: Starting
SocketReader
2013-07-09 12:15:21,683 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RpcDetailedActivityForPort36783 registered.
2013-07-09 12:15:21,684 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
RpcActivityForPort36783 registered.
2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2013-07-09 12:15:21,690 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 36783: starting
2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 36783: starting
2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 36783: starting
2013-07-09 12:15:21,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 36783: starting
2013-07-09 12:15:21,692 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 36783: starting
2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: 127.0.0.1/127.0.0.1:36783
2013-07-09 12:15:21,692 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tracker tracker_dmkd-1:127.0.0.1/127.0.0.1:36783
2013-07-09 12:15:21,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetcher for all reduce tasks on tracker_dmkd-1:
127.0.0.1/127.0.0.1:36783
2013-07-09 12:15:21,762 INFO org.apache.hadoop.util.ProcessTree: setsid
exited with exit code 0
2013-07-09 12:15:21,766 INFO org.apache.hadoop.mapred.TaskTracker:  Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7ec79676
2013-07-09 12:15:21,770 WARN org.apache.hadoop.mapred.TaskTracker:
TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
disabled.
2013-07-09 12:15:21,777 INFO org.apache.hadoop.mapred.IndexCache:
IndexCache created with max memory = 10485760
2013-07-09 12:15:21,785 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
ShuffleServerMetrics registered.
2013-07-09 12:15:21,788 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50060
2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2013-07-09 12:15:21,789 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50060
2013-07-09 12:15:21,789 INFO org.mortbay.log: jetty-6.1.26
2013-07-09 12:15:22,286 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2013-07-09 12:15:22,287 INFO org.apache.hadoop.mapred.TaskTracker:
FILE_CACHE_SIZE for mapOutputServlet set to : 2000
2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0003 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051627_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0002 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,297 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051135_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0002 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051327_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051135_0005 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307050127_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307050127_0002 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051500_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,298 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051634_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051611_0001 for user-log deletion with
retainTimeStamp:1373472921773
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0003 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0003 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,299 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051627_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0002 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051135_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0002 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051327_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051135_0005 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,300 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307050127_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307050127_0002 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051500_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051634_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307032034_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051203_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,301 INFO org.apache.hadoop.mapred.UserLogCleaner:
Adding job_201307051611_0001 for user-log deletion with
retainTimeStamp:1373472921775
2013-07-09 12:15:22,601 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:25,164 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:27,901 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:30,144 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:33,738 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:35,802 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:37,308 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:41,620 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:44,310 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:48,939 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:50,121 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:52,549 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:53,751 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:56,464 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:15:58,423 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:16:03,413 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:16:04,567 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...
2013-07-09 12:16:08,177 INFO org.apache.hadoop.mapred.TaskTracker: Failed
to get system directory...

*"*

Please help.


Thanks and Regards,
Siddharth

Mime
View raw message