hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raj V <rajv...@yahoo.com>
Subject Fw: pointing mapred.local.dir to a ramdisk
Date Mon, 03 Oct 2011 14:37:09 GMT
Sending it to the hadoop mailing list - I think this is a hadoop related problem and not related
to Cloudera distribution.

Raj


----- Forwarded Message -----
>From: Raj V <rajvish@yahoo.com>
>To: CDH Users <cdh-user@cloudera.org>
>Sent: Friday, September 30, 2011 5:21 PM
>Subject: pointing mapred.local.dir to a ramdisk
>
>
>Hi all
>
>
>I have been trying some experiments to improve performance. One of the experiments involved
pointing mapred.local.dir to a RAM disk. To this end I created a 128MB RAM disk ( each of
my map outputs are smaller than this) but I have not been able to get the task tracker to
start.
>
>
>I am running CDH3B3 ( hadoop-0.20.2+737) and here the error message from the task tracker
log.
>
>
>Tasktracker logs
>
>
>2011-09-30 16:50:00,689 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
>2011-09-30 16:50:00,930 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>2011-09-30 16:50:01,000 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort()
before open() is -1. Opening the listener on 50060
>2011-09-30 16:50:01,023 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort()
returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
>2011-09-30 16:50:01,024 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
>2011-09-30 16:50:01,024 INFO org.mortbay.log: jetty-6.1.14
>2011-09-30 16:50:02,388 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
>2011-09-30 16:50:02,400 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing
logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>2011-09-30 16:50:02,422 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker
with owner as mapred
>2011-09-30 16:50:02,493 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task
tracker because java.lang.NullPointerException
>        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:213)
>        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
>        at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:253)
>        at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:404)
>        at org.apache.hadoop.util.MRAsyncDiskService.moveAndDeleteRelativePath(MRAsyncDiskService.java:255)
>        at org.apache.hadoop.util.MRAsyncDiskService.cleanupAllVolumes(MRAsyncDiskService.java:311)
>        at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:618)
>        at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1351)
>        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3504)
>
>
>2011-09-30 16:50:02,497 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
>/************************************************************
>SHUTDOWN_MSG: Shutting down TaskTracker at HADOOP52-4/10.52.1.5
>
>
>and here is my mapred-site.xml file
>
>
><property>
>    <name>mapred.local.dir</name>
>    <value>/ramdisk1</value>
>  </property>
>
>
>If I have a regular directory on a regular drive such as below - it works. If I don't
mount the ramdisk - it works.
>
>
><property>
>    <name>mapred.local.dir</name>
>    <value>/hadoop-dsk0/local,/hadoop-dsk1/local</value>
>  </property>
>
>
>
>
>
>The NullPointerException does not tell me what the error is or how to fix it.
>
>
>From the logs it looks like some disk based operation failed. I can't guess I must also
confess that this is the first time I am using an ext2 file system.
>
>
>Any ideas?
>
>
>
>
>Raj
>
>
>
>
>
>
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message