hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kim Chew <kchew...@gmail.com>
Subject using "-libjars" in Hadoop 2.2.1
Date Tue, 08 Apr 2014 22:27:43 GMT
It seems to me that in Hadoop 2.2.1, using the "libjars" option does not
search the jars located in the the local file system but HDFS. For example,

hadoop jar target/myJar.jar Foo -libjars /home/kchew/test-libs/testJar.jar
/user/kchew/inputs/raw.vector /user/kchew/outputs hdfs://remoteNN:8020
remoteJT:8021

14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
14/04/08 15:11:02 INFO mapreduce.JobSubmitter: Cleaning up the staging area
file:/tmp/hadoop-kchew/mapred/staging/kchew202924688/.staging/job_local202924688_0001
14/04/08 15:11:02 ERROR security.UserGroupInformation:
PriviledgedActionException as:kchew (auth:SIMPLE)
cause:java.io.FileNotFoundException: File does not exist:
hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar
java.io.FileNotFoundException: File does not exist:
hdfs:/remoteNN:8020/home/kchew/test-libs/testJar.jar
    at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
    at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
    at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
    at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
    at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
    at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
    at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
    at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)

So under Hadoop 2.2.1, do I have to explicitly set some configurations so
when using the "libjars" option it will copy the file to hdfs from local fs?

TIA

Kim

Mime
View raw message