hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ridwan Rashid Noel <ridwan...@gmail.com>
Subject "jobtoken not found" while running Hadoop with cephFS
Date Thu, 09 Apr 2015 01:22:02 GMT
Hi,

I have a 5 node cluster (each ubuntu 14.04) where I have setup Hadoop
(1.1.1) and ceph (v0.87). I want to replace HDFS with cephFS and run Hadoop
jobs with cephFS. I ran the wordcount example with normal hadoop setting
and it worked fine. But when I change the Hadoop configuration as mentioned
in the “Using Hadoop with CephFS” documentation
http://ceph.com/docs/master/cephfs/hadoop/, I am facing the following
error: (I have mounted ceph in /mnt/mycephfs directory)

ceph@admin-node:/usr/local/hadoop-1.1.1$ bin/hadoop jar
hadoop*examples*.jar wordcount /mnt/mycephfs/wc-input
/mnt/mycephfs/wc-output-r8


15/03/26 02:54:35 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
15/03/26 02:54:35 INFO input.FileInputFormat: Total input paths to process
: 1
15/03/26 02:54:35 WARN snappy.LoadSnappy: Snappy native library not loaded
15/03/26 02:54:35 INFO mapred.JobClient: Running job: job_201503260253_0001
15/03/26 02:54:36 INFO mapred.JobClient: map 0% reduce 0%
15/03/26 02:54:36 INFO mapred.JobClient: Task Id :
attempt_201503260253_0001_m_000021_0, Status : FAILED
Error initializing attempt_201503260253_0001_m_000021_0:
java.io.FileNotFoundException: File
file:/tmp/hadoop-ceph/mapred/system/job_201503260253_0001/jobToken does not
exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at
org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:4445)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1272)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
at java.lang.Thread.run(Thread.java:745)

15/03/26 02:54:36 WARN mapred.JobClient: Error reading task
outputhttp://node2:50060/tasklog?plaintext=true&attemptid=attempt_201503260253_0001_m_000021_0&filter=stdout
15/03/26 02:54:37 WARN mapred.JobClient: Error reading task
outputhttp://node2:50060/tasklog?plaintext=true&attemptid=attempt_201503260253_0001_m_000021_0&filter=stderr
15/03/26 02:54:38 INFO mapred.JobClient: Task Id :
attempt_201503260253_0001_r_000002_0, Status : FAILED
Error initializing attempt_201503260253_0001_r_000002_0:
java.io.FileNotFoundException: File
file:/tmp/hadoop-ceph/mapred/system/job_201503260253_0001/jobToken does not
exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at
org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:4445)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1272)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
at java.lang.Thread.run(Thread.java:745)

15/03/26 02:54:38 WARN mapred.JobClient: Error reading task
outputhttp://node2:50060/tasklog?plaintext=true&attemptid=attempt_201503260253_0001_r_000002_0&filter=stdout
15/03/26 02:54:38 WARN mapred.JobClient: Error reading task
outputhttp://node2:50060/tasklog?plaintext=true&attemptid=attempt_201503260253_0001_r_000002_0&filter=stderr
15/03/26 02:54:38 INFO mapred.JobClient: Task Id :
attempt_201503260253_0001_m_000021_1, Status : FAILED
Error initializing attempt_201503260253_0001_m_000021_1:
java.io.FileNotFoundException: File
file:/tmp/hadoop-ceph/mapred/system/job_201503260253_0001/jobToken does not
exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at
org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:4445)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1272)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
at java.lang.Thread.run(Thread.java:745) …………………….

Using cephFS instead of HDFS requires only the mapred daemons so only the
jobtracker and tasktrackers are running in the node. My core-site.xml file
of Hadoop:

<configuration>

<property>
<name>fs.ceph.impl</name>
<value>org.apache.hadoop.fs.ceph.CephFileSystem</value>
<description>
</description>
</property>

<property>
<name>fs.default.name</name>
<value>ceph:///</value>
</property>

<property>
<name>ceph.conf.file</name>
<value>/etc/ceph/ceph.conf</value>
</property>
<property>

<name>ceph.root.dir</name>
<value>/</value>
</property>
<property>

<name>ceph.mon.address</name>
<value>10.242.144.225:6789</value>
<description>This is the primary monitor node IP address in our
installation.</description>
</property>

<property>
<name>ceph.auth.id</name>
<value>admin</value>
</property>

<property>
<name>ceph.auth.keyring</name>
<value>/etc/ceph/ceph.client.admin.keyring</value>
</property>

<property>
<name>ceph.object.size</name>
<value>67108864</value>
</property>

<property>
<name>ceph.data.pools</name>
<value>data</value>
</property>

<property>
<name>ceph.localize.reads</name>
<value>true</value>
</property>
</configuration>

Please let me know, how can I solve the problem.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message