hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shengjun Xin <s...@gopivotal.com>
Subject Re: using "-libjars" in Hadoop 2.2.1
Date Thu, 10 Apr 2014 03:59:10 GMT
add '-Dmapreduce.user.classpath.first=true' to your command and try again



On Wed, Apr 9, 2014 at 6:27 AM, Kim Chew <kchew534@gmail.com> wrote:

> It seems to me that in Hadoop 2.2.1, using the "libjars" option does not
> search the jars located in the the local file system but HDFS. For example,
>
> hadoop jar target/myJar.jar Foo -libjars /home/kchew/test-libs/testJar.jar
> /user/kchew/inputs/raw.vector /user/kchew/outputs hdfs://remoteNN:8020
> remoteJT:8021
>
> 14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 14/04/08 15:11:02 INFO mapreduce.JobSubmitter: Cleaning up the staging
> area
> file:/tmp/hadoop-kchew/mapred/staging/kchew202924688/.staging/job_local202924688_0001
> 14/04/08 15:11:02 ERROR security.UserGroupInformation:
> PriviledgedActionException as:kchew (auth:SIMPLE)
> cause:java.io.FileNotFoundException: File does not exist:
> hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar
> java.io.FileNotFoundException: File does not exist:
> hdfs:/remoteNN:8020/home/kchew/test-libs/testJar.jar
>     at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>     at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>     at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>     at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>     at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
>     at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
>     at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
>     at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
>     at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
>
> So under Hadoop 2.2.1, do I have to explicitly set some configurations so
> when using the "libjars" option it will copy the file to hdfs from local fs?
>
> TIA
>
> Kim
>



-- 
Regards
Shengjun

Mime
View raw message