hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stanley.Burnitt" <Stanley.Burn...@huawei.com>
Subject FW: Yarn -> job.jar not found in yarn staging dir (works with TestDFSIO but not ImportTsv )
Date Mon, 05 May 2014 20:48:12 GMT
Hello,  I am trying to run HBase's ImportTsv against Yarn (Hadoop 2.2.0).

I can run the Hadoop TestDFSIO Yarn job with no problems:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar
TestDFSIO -write -nrFiles 20 -fileSize 10

I cannot run the HBase ImportTsv job:

hadoop jar $HBASE_HOME/lib/hbase-server-0.96.0-hadoop2.jar importtsv |
 -Dimporttsv.separator=               \
 -Dimporttsv.columns=$IMPORTTSV_COLUMNS \
 -Dimporttsv.bulk.output=hdfs://$HADOOP_NAMENODE:54310/user/bihadoop/importtsv-output \
 bulkloaded-xdremail  \
 hdfs://$HADOOP_NAMENODE:54310/user/bihadoop/kv-data-files

... because the job.jar file is never successfully downloaded to any yarn node:

2014-05-05 19:44:06,641 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Writing credentials to the nmPrivate file /tmp/hadoop-bihadoop/nm-local-dir/nmPrivate/container_1399316513410_0007_01_000001.tokens.
Credentials list:
2014-05-05 19:44:06,642 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Initializing user bihadoop
2014-05-05 19:44:06,690 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Copying from /tmp/hadoop-bihadoop/nm-local-dir/nmPrivate/container_1399316513410_0007_01_000001.tokens
to /tmp/hadoop-bihadoop/nm-local-dir/usercache/bihadoop/appcache/application_1399316513410_0007/container_1399316513410_0007_01_000001.tokens
2014-05-05 19:44:06,690 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
CWD set to /tmp/hadoop-bihadoop/nm-local-dir/usercache/bihadoop/appcache/application_1399316513410_0007
= file:/tmp/hadoop-bihadoop/nm-local-dir/usercache/bihadoop/appcache/application_1399316513410_0007
2014-05-05 19:44:06,791 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:bihadoop (auth:SIMPLE) cause:java.io.FileNotFoundException: File file:/tmp/hadoop-yarn/staging/bihadoop/.staging/job_1399316513410_0007/job.jar
does not exist
2014-05-05 19:44:06,810 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
DEBUG: FAILED { file:/tmp/hadoop-yarn/staging/bihadoop/.staging/job_1399316513410_0007/job.jar,
1399319045000, PATTERN, (?:classes/|lib/).* }, File file:/tmp/hadoop-yarn/staging/bihadoop/.staging/job_1399316513410_0007/job.jar
does not exist
2014-05-05 19:44:06,812 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource file:/tmp/hadoop-yarn/staging/bihadoop/.staging/job_1399316513410_0007/job.jar transitioned
from DOWNLOADING to FAILED


I have also tried running importtsv from the hbase script, i.e.:
hbase  org.apache.hadoop.hbase.mapreduce.ImportTsv \
...
...

The end result is the same: job.jar not found in staging dir.  (The job.jar is found on the
master node, where I run the job.)

There seems to be no problems distributing job.jar to the staging dirs when I run the TestDFSIO
job.
But the job.jar containing ImportTsv is never distributed.

Very frustrating,  time eating... ;/

Does anyone have any suggestions?  I am runinng Hadoop 2.2.0 w/ HBase 0.96.0

Thanks,
Stan




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message