Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E518D17D78 for ; Mon, 29 Sep 2014 18:57:55 +0000 (UTC) Received: (qmail 27216 invoked by uid 500); 29 Sep 2014 18:57:46 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 27111 invoked by uid 500); 29 Sep 2014 18:57:46 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 27100 invoked by uid 99); 29 Sep 2014 18:57:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Sep 2014 18:57:46 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of iainwrig@gmail.com designates 209.85.216.176 as permitted sender) Received: from [209.85.216.176] (HELO mail-qc0-f176.google.com) (209.85.216.176) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Sep 2014 18:57:11 +0000 Received: by mail-qc0-f176.google.com with SMTP id o8so10237054qcw.21 for ; Mon, 29 Sep 2014 11:57:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=n+nJ7WtUXwdiGMsi4hP9qnrHkTUv02nEP+epanCkTVY=; b=Mx1iuy69tWdvWcB+M7dSWKIQIYGBh+gHTEFKYPN2riQB3tRRdVFXk8L6D8j5PhUGts eHlts6zx7byWtGQ+9ZuuRhRBDXYiFRBpQAhf9ZgBaQTeY3PfNaCfIqXK2Vpy257AgRt8 j1JCklJGBaLAV2Lorlb2H6NlxOXaHmAl/NUY+Kz5KHSK+adPkqQuYo94fwWUV/gBuQj0 +orjKL2YNfIx46emqpwS96Ey5SP7/5K4RUvIpX1Kngva2Twe+GSJ6M4RHVbL4vuvR9h0 kg9lxch8uvW6NQWFOu9QxiRTQqLOSeBz1M6A2ncvgzAj+V+6i4MB+Q0X7VsRuHV8fZbS 1wXg== X-Received: by 10.229.212.66 with SMTP id gr2mr5209891qcb.27.1412017029565; Mon, 29 Sep 2014 11:57:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.37.19 with HTTP; Mon, 29 Sep 2014 11:56:49 -0700 (PDT) From: iain wright Date: Mon, 29 Sep 2014 11:56:49 -0700 Message-ID: Subject: YARN application failing to localize needed jars (Xpost: hbase users) To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a11339b5876d628050438d548 X-Virus-Checked: Checked by ClamAV on apache.org --001a11339b5876d628050438d548 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi folks, I'm having trouble using HBASE copyTable to seed an existing tables data to a replication peer. Surely its an oversight in configuration on our part, but I've scoured the web and doc's for a couple days now. We have been able to run these jobs with success (perhaps they don't require localization of any jars since the jars are already included in the environment on each nodemanager?): ./hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fileSize 5GB ./hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 5 When running copyTable it seems like the resources are placed on the yarn application manager but not localized to the nodemanagers. I've tried several settings for yarn.app.mapreduce.am.staging-dir and in the below iteration im trying the defaults, the path exists with correct permissions on each NM/AM hadoop.tmp.dir is specified in core-site.xml Here is the full command we are running: [hbase@master2 bin]$ ./hbase org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=3D1409128964 --peer.adr=3Dmaster0.hbasex1.test.cloud.domain.tv, master1.hbasex1.test.cloud.v.tv,master2.hbasex1.test.cloud.domain.tv:2181:/= hbase moderation SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hbase-0.98.6.1-1a/lib/slf4j-log4j12-1.6.4.jar!/org/slf= 4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.5.0-1a/share/hadoop/common/lib/slf4j-log4j12-= 1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2014-09-26 23:51:24,141 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-09-26 23:51:24,863 WARN [main] shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2014-09-26 23:51:24,910 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.HConstants, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,911 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.client.Put, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,913 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.zookeeper.ZooKeeper, using jar /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.jboss.netty.channel.ChannelFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar 2014-09-26 23:51:24,915 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.protobuf.Message, using jar /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.common.collect.Lists, using jar /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cloudera.htrace.Trace, using jar /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar 2014-09-26 23:51:24,917 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cliffc.high_scale_lib.Counter, using jar /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar 2014-09-26 23:51:24,921 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.LongWritable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,922 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Text, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.LongWritable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,924 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Text, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,925 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-c= ore-2.5.0.jar 2014-09-26 23:51:24,926 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-c= ore-2.5.0.jar 2014-09-26 23:51:24,952 INFO [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.HConstants, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.client.Put, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,965 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.zookeeper.ZooKeeper, using jar /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.jboss.netty.channel.ChannelFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.protobuf.Message, using jar /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.common.collect.Lists, using jar /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cloudera.htrace.Trace, using jar /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cliffc.high_scale_lib.Counter, using jar /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar 2014-09-26 23:51:24,971 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Writable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Writable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,974 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableOutputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,975 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-c= ore-2.5.0.jar 2014-09-26 23:51:25,035 INFO [main] client.RMProxy: Connecting to ResourceManager at master2.hbase1a.test.cloud.domain.tv/172.16.0.145:8032 2014-09-26 23:51:25,191 INFO [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2014-09-26 23:51:25,247 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=3Dhconnection-0x6bf3ede0 connecting to ZooKeeper ensembl= e=3D master1.hbasex1.test.cloud.domain.tv:2181, master0.hbasex1.test.cloud.domain.tv:2181, master2.hbasex1.test.cloud.domain.tv:2181 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3D3.4.6-1569965, built on 02/20/2014 09:09 GM= T 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=3Dmaster2.hbase1a.test.cloud.domain.tv 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=3D1.7.0_65 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=3DOracle Corporation 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=3D/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=3D/usr/local/hbase/bin/../conf:/usr/lib/jvm/jav= a/lib/tools.jar:/usr/local/hbase/bin/..:/usr/local/hbase/bin/../lib/activat= ion-1.1.jar:/usr/local/hbase/bin/../lib/aopalliance-1.0.jar:/usr/local/hbas= e/bin/../lib/asm-3.1.jar:/usr/local/hbase/bin/../lib/avro-1.7.4.jar:/usr/lo= cal/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/bin/../li= b/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/bin/../lib/commons-cli-= 1.2.jar:/usr/local/hbase/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase/= bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase/bin/../lib/common= s-compress-1.4.1.jar:/usr/local/hbase/bin/../lib/commons-configuration-1.6.= jar:/usr/local/hbase/bin/../lib/commons-daemon-1.0.13.jar:/usr/local/hbase/= bin/../lib/commons-digester-1.8.jar:/usr/local/hbase/bin/../lib/commons-el-= 1.0.jar:/usr/local/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/local/h= base/bin/../lib/commons-io-2.4.jar:/usr/local/hbase/bin/../lib/commons-lang= -2.6.jar:/usr/local/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/local/h= base/bin/../lib/commons-math-2.1.jar:/usr/local/hbase/bin/../lib/commons-ne= t-3.1.jar:/usr/local/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr= /local/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar:/usr/local/hbase/bin/= ../lib/grizzly-framework-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http= -2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-server-2.1.2.jar:/usr/l= ocal/hbase/bin/../lib/grizzly-http-servlet-2.1.2.jar:/usr/local/hbase/bin/.= ./lib/grizzly-rcm-2.1.2.jar:/usr/local/hbase/bin/../lib/guava-12.0.1.jar:/u= sr/local/hbase/bin/../lib/guice-3.0.jar:/usr/local/hbase/bin/../lib/guice-s= ervlet-3.0.jar:/usr/local/hbase/bin/../lib/hamcrest-core-1.3.jar:/usr/local= /hbase/bin/../lib/hbase-client-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/..= /lib/hbase-common-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-co= mmon-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-examples-= 0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop2-compat-0.98.= 6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop-compat-0.98.6.1-ha= doop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2.jar:/usr/lo= cal/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/b= in/../lib/hbase-prefix-tree-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../li= b/hbase-protocol-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-ser= ver-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-= hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-shell-0.98.6.1-hadoop2.= jar:/usr/local/hbase/bin/../lib/hbase-testing-util-0.98.6.1-hadoop2.jar:/us= r/local/hbase/bin/../lib/hbase-thrift-0.98.6.1-hadoop2.jar:/usr/local/hbase= /bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/bin/../lib/htrace-cor= e-2.04.jar:/usr/local/hbase/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbas= e/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase/bin/../lib/jackson-core-as= l-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/= hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/j= ackson-xc-1.8.8.jar:/usr/local/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/us= r/local/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/bin/..= /lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/bin/../lib/javax.inject-1.j= ar:/usr/local/hbase/bin/../lib/javax.servlet-3.1.jar:/usr/local/hbase/bin/.= ./lib/javax.servlet-api-3.0.1.jar:/usr/local/hbase/bin/../lib/jaxb-api-2.2.= 2.jar:/usr/local/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/bi= n/../lib/jersey-client-1.9.jar:/usr/local/hbase/bin/../lib/jersey-core-1.8.= jar:/usr/local/hbase/bin/../lib/jersey-grizzly2-1.9.jar:/usr/local/hbase/bi= n/../lib/jersey-guice-1.9.jar:/usr/local/hbase/bin/../lib/jersey-json-1.8.j= ar:/usr/local/hbase/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase/bin/.= ./lib/jersey-test-framework-core-1.9.jar:/usr/local/hbase/bin/../lib/jersey= -test-framework-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jets3t-0.6.1.j= ar:/usr/local/hbase/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase/bin/../l= ib/jetty-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:= /usr/local/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase/bin/../l= ib/jruby-complete-1.6.8.jar:/usr/local/hbase/bin/../lib/jsch-0.1.42.jar:/us= r/local/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsp= -api-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsr305-1.3.9.jar:/usr/local= /hbase/bin/../lib/junit-4.11.jar:/usr/local/hbase/bin/../lib/libthrift-0.9.= 0.jar:/usr/local/hbase/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase/bin/../= lib/management-api-3.0.0-b012.jar:/usr/local/hbase/bin/../lib/metrics-core-= 2.2.0.jar:/usr/local/hbase/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbas= e/bin/../lib/paranamer-2.3.jar:/usr/local/hbase/bin/../lib/protobuf-java-2.= 5.0.jar:/usr/local/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/h= base/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase/bin/../lib/slf4j-log4j= 12-1.6.4.jar:/usr/local/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local= /hbase/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase/bin/../lib/xz-1.0.jar:/u= sr/local/hbase/bin/../lib/zookeeper-3.4.6.jar:/usr/local/hadoop/etc/hadoop:= /usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/ha= doop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/sha= re/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/co= mmon/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paran= amer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2= .5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.j= ar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local= /hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/had= oop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/j= axb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logg= ing-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1= .jar:/usr/local/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/usr/loca= l/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/shar= e/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/had= oop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/a= pi-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-1= 1.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr= /local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/ha= doop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/a= vro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.5.0.j= ar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:= /usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/= local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop= /share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/ha= doop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common= /lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/common= s-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-= 3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/= usr/local/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/loc= al/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/ha= doop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/= hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/co= mmon/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/co= mmon/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson= -mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/activation= -1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/loc= al/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/sha= re/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/c= ommon/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-= api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar= :/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/u= sr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoo= p/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/= hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/= lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jetti= son-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.= 1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/= local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoo= p/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/c= ommon/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/= asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-cod= ec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.= jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0.jar:/usr/loca= l/hadoop/share/hadoop/common/hadoop-nfs-2.5.0.jar:/usr/local/hadoop/share/h= adoop/common/hadoop-common-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/h= dfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/h= adoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/sha= re/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/l= ib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-= 6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/lo= cal/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/ha= doop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/= protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli= -1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:= /usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lo= cal/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/sh= are/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/li= b/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xml= enc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/u= sr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop= /share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop= /hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/net= ty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar= :/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/shar= e/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hd= fs/hadoop-hdfs-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdf= s-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr= /local/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop/shar= e/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn= /lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guic= e-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/= local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share= /hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/= lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all= -1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/lo= cal/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/= yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/common= s-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.= jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/loca= l/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/h= adoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/= commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-g= uice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.1= 3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/lo= cal/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/h= adoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop= /yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey= -core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar= :/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/had= oop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/had= oop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/y= arn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/st= ax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/= usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/loca= l/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/shar= e/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn= /lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/= usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoo= p/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/usr/local/hadoop/s= hare/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/= usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/usr/local/= hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/usr/local/hado= op/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/usr/local/hadoop/share/hado= op/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/usr/local/hadoop/share/hado= op/yarn/hadoop-yarn-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/ha= doop-yarn-applications-distributedshell-2.5.0.jar:/usr/local/hadoop/share/h= adoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/usr/local/hadoop/share= /hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/usr/lo= cal/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/= usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/h= adoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/= hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/h= adoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/= lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leve= ldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:= /usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hado= op/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/h= adoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/= mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/l= ib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jer= sey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core= -asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.j= ar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.j= ar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/lo= cal/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hado= op/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop= /mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/ma= preduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/= lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-= 3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-co= re-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-clie= nt-hs-plugins-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-map= reduce-client-shuffle-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/ha= doop-mapreduce-client-hs-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce= /hadoop-mapreduce-client-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/ma= preduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/usr/local/hadoop= /share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/usr/local/ha= doop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/us= r/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/= usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/etc/had= oop/ 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=3D/usr/local/hadoop/lib 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=3D/tmp 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=3D 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=3DLinux 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=3Damd64 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3D3.10.42-52.145.amzn1.x86_64 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=3Dhbase 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=3D/home/hbase 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=3D/usr/local/hbase-0.98.6.1-1a/bin 2014-09-26 23:51:25,254 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=3Dmaster1.hbasex1.test.cloud.domain.tv:2181, master0.hbasex1.test.cloud.domain.tv:2181, master2.hbasex1.test.cloud.domain.tv:2181 sessionTimeout=3D90000 watcher=3Dhconnection-0x6bf3ede0, quorum=3D master1.hbasex1.test.cloud.domain.tv:2181, master0.hbasex1.test.cloud.domain.tv:2181, master2.hbasex1.test.cloud.domain.tv:2181, baseZNode=3D/hbase 2014-09-26 23:51:25,270 INFO [main-SendThread( master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening socket connection to server master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-26 23:51:25,276 INFO [main-SendThread( master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket connection established to master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, initiating session 2014-09-26 23:51:25,282 INFO [main-SendThread( master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Session establishment complete on server master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, sessionid =3D 0x248afb6109b007f, negotiated timeout =3D 40000 2014-09-26 23:51:25,308 INFO [main] mapreduce.TableOutputFormat: Created table instance for moderation 2014-09-26 23:51:25,469 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=3Dhconnection-0xa7a371 connecting to ZooKeeper ensemble= =3D master1.hbase1a.test.cloud.domain.tv:2181, master0.hbase1a.test.cloud.domain.tv:2181, master2.hbase1a.test.cloud.domain.tv:2181 2014-09-26 23:51:25,469 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=3Dmaster1.hbase1a.test.cloud.domain.tv:2181, master0.hbase1a.test.cloud.domain.tv:2181, master2.hbase1a.test.cloud.domain.tv:2181 sessionTimeout=3D90000 watcher=3Dhconnection-0xa7a371, quorum=3D master1.hbase1a.test.cloud.domain.tv:2181, master0.hbase1a.test.cloud.domain.tv:2181, master2.hbase1a.test.cloud.domain.tv:2181, baseZNode=3D/hbase 2014-09-26 23:51:25,470 INFO [main-SendThread( master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening socket connection to server master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-26 23:51:25,470 INFO [main-SendThread( master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket connection established to master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, initiating session 2014-09-26 23:51:25,472 INFO [main-SendThread( master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Session establishment complete on server master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, sessionid =3D 0x34877671747028e, negotiated timeout =3D 40000 2014-09-26 23:51:25,490 INFO [main] util.RegionSizeCalculator: Calculating region sizes for table "moderation". 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region moderation,,1411774704618.68a1839eacfa355028522286caaf7cd5. has size 0 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region sizes calculated 2014-09-26 23:51:25,709 DEBUG [main] client.ClientSmallScanner: Finished with small scan at {ENCODED =3D> 1588230740, NAME =3D> 'hbase:meta,,1', STARTKEY =3D> '', ENDKEY =3D> ''} 2014-09-26 23:51:25,722 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: split -> 0 -> ip-172-16-0-247.us-west-1.compute.internal:, 2014-09-26 23:51:25,739 INFO [main] mapreduce.JobSubmitter: number of splits:1 2014-09-26 23:51:25,748 INFO [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2014-09-26 23:51:25,895 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1411771844660_0004 2014-09-26 23:51:26,033 INFO [main] impl.YarnClientImpl: Submitted application application_1411771844660_0004 2014-09-26 23:51:26,066 INFO [main] mapreduce.Job: The url to track the job: http://master2.hbase1a.test.cloud.domain.tv:8088/proxy/application_14117718= 44660_0004/ 2014-09-26 23:51:26,067 INFO [main] mapreduce.Job: Running job: job_1411771844660_0004 2014-09-26 23:51:29,082 INFO [main] mapreduce.Job: Job job_1411771844660_0004 running in uber mode : false 2014-09-26 23:51:29,084 INFO [main] mapreduce.Job: map 0% reduce 0% 2014-09-26 23:51:29,094 INFO [main] mapreduce.Job: Job job_1411771844660_0004 failed with state FAILED due to: Application application_1411771844660_0004 failed 2 times due to AM Container for appattempt_1411771844660_0004_000002 exited with exitCode: -1000 due to: File file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar does not exist .Failing this attempt.. Failing the application. 2014-09-26 23:51:29,111 INFO [main] mapreduce.Job: Counters: 0 The directory on the AM (master) from which the command is initiated as hbase user: [hbase@master2 bin]$ ls -lah /tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/ total 3.7M drwx------ 3 hbase hbase 4.0K Sep 26 23:51 . drwx------ 10 hbase hadoop 4.0K Sep 26 23:51 .. -rw-r--r-- 1 hbase hbase 3.5M Sep 26 23:51 job.jar -rw-rw-r-- 1 hbase hbase 28K Sep 26 23:51 .job.jar.crc -rw-r--r-- 1 hbase hbase 111 Sep 26 23:51 job.split -rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.split.crc -rw-r--r-- 1 hbase hbase 56 Sep 26 23:51 job.splitmetainfo -rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.splitmetainfo.crc -rw-r--r-- 1 hbase hbase 102K Sep 26 23:51 job.xml -rw-rw-r-- 1 hbase hbase 824 Sep 26 23:51 .job.xml.crc drwx------ 2 hbase hbase 4.0K Sep 26 23:51 libjars The same directory is empty on each NM. Heres also the log from a node that got the job/application: 2014-09-26 23:51:26,635 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1411771844660_0004_000001 (auth:SIMPLE) 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManager= Impl: Start request for container_1411771844660_0004_01_000001 by user hbase 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManager= Impl: Creating a new application reference for app application_1411771844660_0004 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=3Dhbase IP=3D172.16.0.145 OPERATION=3DStart Container Request TARGET=3DContainerManageImpl RESULT=3DSUCCESS APPID=3Dapplication_1411771844660_0004 CONTAINERID=3Dcontainer_1411771844660_0004_01_000001 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= ication: Application application_1411771844660_0004 transitioned from NEW to INITING 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= ication: Adding container_1411771844660_0004_01_000001 to application application_1411771844660_0004 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= ication: Application application_1411771844660_0004 transitioned from INITING to RUNNING 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Contai= ner: Container container_1411771844660_0004_01_000001 transitioned from NEW to LOCALIZING 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1411771844660_0004 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Locali= zedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar transitioned from INIT to DOWNLOADING 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Locali= zedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.spl= itmetainfo transitioned from INIT to DOWNLOADING 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Locali= zedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.spl= it transitioned from INIT to DOWNLOADING 2014-09-26 23:51:26,641 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Locali= zedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.xml transitioned from INIT to DOWNLOADING 2014-09-26 23:51:26,641 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Resour= ceLocalizationService: Created localizer for container_1411771844660_0004_01_000001 2014-09-26 23:51:26,645 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Resour= ceLocalizationService: Writing credentials to the nmPrivate file /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.t= okens. Credentials list: 2014-09-26 23:51:26,648 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hbase 2014-09-26 23:51:26,661 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.t= okens to /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660= _0004/container_1411771844660_0004_01_000001.tokens 2014-09-26 23:51:26,661 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660= _0004 =3D file:/data0/hadoop/yarn/local/usercache/hbase/appcache/application_14117718= 44660_0004 2014-09-26 23:51:26,696 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Resour= ceLocalizationService: DEBUG: FAILED { file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar= , 1411775485000, PATTERN, (?:classes/|lib/).* }, File file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar does not exist 2014-09-26 23:51:26,697 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Locali= zedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar= (->/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844= 660_0004/filecache/10/job.jar) transitioned from DOWNLOADING to FAILED 2014-09-26 23:51:26,697 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Contai= ner: Container container_1411771844660_0004_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2014-09-26 23:51:26,697 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalR= esourcesTrackerImpl: Container container_1411771844660_0004_01_000001 sent RELEASE event on a resource request { file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar= , 1411775485000, PATTERN, (?:classes/|lib/).* } not present in cache. 2014-09-26 23:51:26,698 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660= _0004/container_1411771844660_0004_01_000001 2014-09-26 23:51:26,698 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Resour= ceLocalizationService: Unknown localizer with localizerId container_1411771844660_0004_01_000001 is sending heartbeat. Ordering it to DIE 2014-09-26 23:51:26,698 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: delete returned false for path: [/data0/hadoop/yarn/local/usercache/hbase/appcache/application_141177184466= 0_0004/container_1411771844660_0004_01_000001] 2014-09-26 23:51:26,698 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660= _0004/container_1411771844660_0004_01_000001 2014-09-26 23:51:26,699 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: delete returned false for path: [/data1/hadoop/yarn/local/usercache/hbase/appcache/application_141177184466= 0_0004/container_1411771844660_0004_01_000001] 2014-09-26 23:51:26,699 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=3Dhbase OPERATION=3DContainer Finished - Failed TARGET=3DContainerImpl RESULT=3DFAILURE DESCRIPTION=3DContainer failed with state: LOCALIZATION_FAILED APPID=3Dapplication_1411771844660_0004 CONTAINERID=3Dcontainer_1411771844660_0004_01_000001 2014-09-26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Contai= ner: Container container_1411771844660_0004_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2014-09-26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= ication: Removing container_1411771844660_0004_01_000001 from application application_1411771844660_0004 2014-09-26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1411771844660_0004 2014-09-26 23:51:27,629 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1411771844660_0004_01_000001] 2014-09-26 23:51:28,252 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.Containe= rsMonitorImpl: Stopping resource-monitoring for container_1411771844660_0004_01_000001 2014-09-26 23:51:29,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= ication: Application application_1411771844660_0004 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660= _0004 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660= _0004 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1411771844660_0004 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= ication: Application application_1411771844660_0004 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAg= gregatingLogHandler: Scheduling Log Deletion for application: application_1411771844660_0004, with delay of 10800 seconds Thank you in advance for any advice --=20 Iain Wright This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message. --001a11339b5876d628050438d548 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi folks,

I'm having trouble using = HBASE copyTable to seed=20 an existing tables data to a replication peer. Surely its an oversight=20 in configuration on our part, but I've scoured the web and doc's fo= r a=20 couple days now.

We have been able to run these jobs with success (perhaps they don't require localization of any jars= =20 since the jars are already included in the environment on each nodemanager?= ):
./hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fil= eSize 5GB
./hbase org.apache.hadoop.hbase.PerformanceEvaluation = =C2=A0randomWrite 5

When running copyTable it seems like the resources are placed on the yarn=20 application manager but not localized to the nodemanagers.=C2=A0
=
=C2=A0I've tried several settings for yarn.app.mapreduce.am.staging= -dir and in the below iteration im trying the defaults, the path exists wit= h correct permissions on each NM/AM=C2=A0

hadoop.t= mp.dir is specified in core-site.xml

Here is the f= ull command we are running:

[hbase@master2 bin]$ ./h= base org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=3D1409128964 -= -peer.adr=3Dmaster0.hbasex1.test.cloud.domain.tv,master1.hbasex1.test.cloud.v= .tv,master2.hbasex1.test.cloud.domain.tv:2181:/hbase moderation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hbase-0.98.6.1-1a/lib/slf4j-l= og4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J:= Found binding in [jar:file:/usr/local/hadoop-2.5.0-1a/share/hadoop/common/= lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
=
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings = for an explanation.
2014-09-26 23:51:24,141 WARN =C2=A0[main] util.NativeCodeLoader: Unable to load=20 native-hadoop library for your platform... using builtin-java classes=20 where applicable
2014-09-26 23:51:24,863 WARN =C2=A0[main] shortc= ircuit.DomainSocketFactory: The short-circuit local reads feature cannot be= used because libhadoop cannot be loaded.
2014-09-26 23:51:24,910= DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hba= se.HConstants, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98= .6.1-hadoop2.jar
2014-09-26 23:51:24,911 DEBUG [main] mapreduce.T= ableMapReduceUtil: For class org.apache.hadoop.hbase.protobuf.generated.Cli= entProtos, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6= .1-hadoop2.jar
2014-09-26 23:51:24,912 DEBUG [main] mapreduce.Tab= leMapReduceUtil: For class org.apache.hadoop.hbase.client.Put, using jar /u= sr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar
= 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For clas= s org.apache.hadoop.hbase.CompatibilityFactory, using jar /usr/local/hbase-= 0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar
2014-09-= 26 23:51:24,913 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.ap= ache.hadoop.hbase.mapreduce.TableMapper, using jar /usr/local/hbase-0.98.6.= 1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,91= 4 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.zookeeper= .ZooKeeper, using jar /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar<= /div>
2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil= : For class org.jboss.netty.channel.ChannelFactory, using jar /usr/local/hb= ase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar
2014-09-26 23:51:24,915 DEBUG [main] mapreduce.TableMapReduceUtil: For class=20 com.google.protobuf.Message, using jar /usr/local/hbase-0.98.6.1-1a/lib/pro= tobuf-java-2.5.0.jar
2014-09-26 23:51:24,916 DEBUG [main] mapredu= ce.TableMapReduceUtil: For class com.google.common.collect.Lists, using jar= /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar
2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For class=20 org.cloudera.htrace.Trace, using jar /usr/local/hbase-0.98.6.1-1a/lib/htrac= e-core-2.04.jar
2014-09-26 23:51:24,917 DEBUG [main] mapreduce.Ta= bleMapReduceUtil: For class org.cliffc.high_scale_lib.Counter, using jar /u= sr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar
2014-09-2= 6 23:51:24,921 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apa= che.hadoop.io.LongWritable, using jar /usr/local/hadoop-2.5.0-1a/share/hado= op/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,922 DEBUG [main] mapreduce.TableMapReduceUtil: For class=20 org.apache.hadoop.io.Text, using jar /usr/local/hadoop-2.5.0-1a/share/hadoo= p/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,923 DEBUG [m= ain] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapred= uce.TableInputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-serv= er-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,923 DEBUG [main] mapr= educe.TableMapReduceUtil: For class org.apache.hadoop.io.LongWritable, usin= g jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.ja= r
2014-09-26 23:51:24,924 DEBUG [main] mapreduce.TableMapReduceUtil: For class=20 org.apache.hadoop.io.Text, using jar /usr/local/hadoop-2.5.0-1a/share/hadoo= p/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,925 DEBUG [m= ain] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.mapreduce.li= b.output.TextOutputFormat, using jar /usr/local/hadoop-2.5.0-1a/share/hadoo= p/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
2014-09-26 23:= 51:24,926 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.h= adoop.mapreduce.lib.partition.HashPartitioner, using jar /usr/local/hadoop-= 2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
2014-09-26 23:51:24,952 INFO =C2=A0[main] Configuration.deprecation:=20 io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: = For class org.apache.hadoop.hbase.HConstants, using jar /usr/local/hbase-0.= 98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
2014-09-26 23:51:= 24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hado= op.hbase.protobuf.generated.ClientProtos, using jar /usr/local/hbase-0.98.6= .1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24= ,964 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop= .hbase.client.Put, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-= 0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,964 DEBUG [main] mapredu= ce.TableMapReduceUtil: For class org.apache.hadoop.hbase.CompatibilityFacto= ry, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1= -hadoop2.jar
2014-09-26 23:51:24,965 DEBUG [main] mapreduce.Table= MapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableMapper, usi= ng jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil:= For class org.apache.zookeeper.ZooKeeper, using jar /usr/local/hbase-0.98.= 6.1-1a/lib/zookeeper-3.4.6.jar
2014-09-26 23:51:24,966 DEBUG [mai= n] mapreduce.TableMapReduceUtil: For class org.jboss.netty.channel.ChannelF= actory, using jar /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar
2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For class=20 com.google.protobuf.Message, using jar /usr/local/hbase-0.98.6.1-1a/lib/pro= tobuf-java-2.5.0.jar
2014-09-26 23:51:24,967 DEBUG [main] mapredu= ce.TableMapReduceUtil: For class com.google.common.collect.Lists, using jar= /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar
2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For class=20 org.cloudera.htrace.Trace, using jar /usr/local/hbase-0.98.6.1-1a/lib/htrac= e-core-2.04.jar
2014-09-26 23:51:24,968 DEBUG [main] mapreduce.Ta= bleMapReduceUtil: For class org.cliffc.high_scale_lib.Counter, using jar /u= sr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar
2014-09-2= 6 23:51:24,971 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apa= che.hadoop.hbase.io.ImmutableBytesWritable, using jar /usr/local/hbase-0.98= .6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For class=20 org.apache.hadoop.io.Writable, using jar /usr/local/hadoop-2.5.0-1a/share/h= adoop/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,972 DEBU= G [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.ma= preduce.TableInputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-= server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,973 DEBUG [main] = mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.io.Immutabl= eBytesWritable, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.9= 8.6.1-hadoop2.jar
2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For class=20 org.apache.hadoop.io.Writable, using jar /usr/local/hadoop-2.5.0-1a/share/h= adoop/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,974 DEBU= G [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.ma= preduce.TableOutputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase= -server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,975 DEBUG [main]= mapreduce.TableMapReduceUtil: For class org.apache.hadoop.mapreduce.lib.pa= rtition.HashPartitioner, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/= mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
2014-09-26 23:51= :25,035 INFO =C2=A0[main] client.RMProxy: Connecting to ResourceManager at = master2.hbase1a.test.cloud.domain.tv/172.16.0.145:8032=
2014-09-26 23:51:25,191 INFO =C2=A0[main] Configuration.deprecation:=20 io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-09-26 23:51:25,247 INFO =C2=A0[main] zookeeper.RecoverableZooK= eeper: Process identifier=3Dhconnection-0x6bf3ede0 connecting to ZooKeeper = ensemble=3Dmaster1.hbasex1.test.cloud.domain.tv:2181,master0.hb= asex1.test.cloud.domain.tv:2181,master2.hbasex1.test.cloud.domain.t= v:2181
2014-09-26 23:51:25,252 INFO =C2=A0[main] zookeeper.Zo= oKeeper: Client environment:zookeeper.version=3D3.4.6-1569965, built on 02/= 20/2014 09:09 GMT
2014-09-26 23:51:25,252 INFO =C2=A0[main] zooke= eper.ZooKeeper: Client environment:host.name=3Dmaster2.hbase1a.test.cloud.domain.tv
20= 14-09-26 23:51:25,252 INFO =C2=A0[main] zookeeper.ZooKeeper: Client environ= ment:java.version=3D1.7.0_65
2014-09-26 23:51:25,252 INFO =C2=A0[= main] zookeeper.ZooKeeper: Client environment:java.vendor=3DOracle Corporat= ion
2014-09-26 23:51:25,252 INFO =C2=A0[main] zookeeper.ZooKeeper= : Client environment:java.home=3D/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x= 86_64/jre
2014-09-26 23:51:25,252 INFO =C2=A0[main] zookeeper.Zoo= Keeper: Client environment:java.class.path=3D/usr/local/hbase/bin/../conf:/= usr/lib/jvm/java/lib/tools.jar:/usr/local/hbase/bin/..:/usr/local/hbase/bin= /../lib/activation-1.1.jar:/usr/local/hbase/bin/../lib/aopalliance-1.0.jar:= /usr/local/hbase/bin/../lib/asm-3.1.jar:/usr/local/hbase/bin/../lib/avro-1.= 7.4.jar:/usr/local/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/= hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/bin/../l= ib/commons-cli-1.2.jar:/usr/local/hbase/bin/../lib/commons-codec-1.7.jar:/u= sr/local/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase/bi= n/../lib/commons-compress-1.4.1.jar:/usr/local/hbase/bin/../lib/commons-con= figuration-1.6.jar:/usr/local/hbase/bin/../lib/commons-daemon-1.0.13.jar:/u= sr/local/hbase/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase/bin/../= lib/commons-el-1.0.jar:/usr/local/hbase/bin/../lib/commons-httpclient-3.1.j= ar:/usr/local/hbase/bin/../lib/commons-io-2.4.jar:/usr/local/hbase/bin/../l= ib/commons-lang-2.6.jar:/usr/local/hbase/bin/../lib/commons-logging-1.1.1.j= ar:/usr/local/hbase/bin/../lib/commons-math-2.1.jar:/usr/local/hbase/bin/..= /lib/commons-net-3.1.jar:/usr/local/hbase/bin/../lib/findbugs-annotations-1= .3.9-1.jar:/usr/local/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar:/usr/l= ocal/hbase/bin/../lib/grizzly-framework-2.1.2.jar:/usr/local/hbase/bin/../l= ib/grizzly-http-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-server-2= .1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-servlet-2.1.2.jar:/usr/lo= cal/hbase/bin/../lib/grizzly-rcm-2.1.2.jar:/usr/local/hbase/bin/../lib/guav= a-12.0.1.jar:/usr/local/hbase/bin/../lib/guice-3.0.jar:/usr/local/hbase/bin= /../lib/guice-servlet-3.0.jar:/usr/local/hbase/bin/../lib/hamcrest-core-1.3= .jar:/usr/local/hbase/bin/../lib/hbase-client-0.98.6.1-hadoop2.jar:/usr/loc= al/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/= ../lib/hbase-common-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/= hbase-examples-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoo= p2-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop-com= pat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hado= op2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2-tests.jar:/us= r/local/hbase/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop2.jar:/usr/local/= hbase/bin/../lib/hbase-protocol-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/.= ./lib/hbase-server-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-s= erver-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-shell-0.= 98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-testing-util-0.98.6.1-= hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-thrift-0.98.6.1-hadoop2.jar:/= usr/local/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/bin/..= /lib/htrace-core-2.04.jar:/usr/local/hbase/bin/../lib/httpclient-4.1.3.jar:= /usr/local/hbase/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase/bin/../lib/= jackson-core-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-jaxrs-1.8.8.= jar:/usr/local/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hba= se/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase/bin/../lib/jamon-runtim= e-2.3.1.jar:/usr/local/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/loc= al/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/bin/../lib/j= avax.inject-1.jar:/usr/local/hbase/bin/../lib/javax.servlet-3.1.jar:/usr/lo= cal/hbase/bin/../lib/javax.servlet-api-3.0.1.jar:/usr/local/hbase/bin/../li= b/jaxb-api-2.2.2.jar:/usr/local/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr= /local/hbase/bin/../lib/jersey-client-1.9.jar:/usr/local/hbase/bin/../lib/j= ersey-core-1.8.jar:/usr/local/hbase/bin/../lib/jersey-grizzly2-1.9.jar:/usr= /local/hbase/bin/../lib/jersey-guice-1.9.jar:/usr/local/hbase/bin/../lib/je= rsey-json-1.8.jar:/usr/local/hbase/bin/../lib/jersey-server-1.8.jar:/usr/lo= cal/hbase/bin/../lib/jersey-test-framework-core-1.9.jar:/usr/local/hbase/bi= n/../lib/jersey-test-framework-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib= /jets3t-0.6.1.jar:/usr/local/hbase/bin/../lib/jettison-1.3.1.jar:/usr/local= /hbase/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-ssleng= ine-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/local= /hbase/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase/bin/../lib/jsch= -0.1.42.jar:/usr/local/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase= /bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsr305-1.3.9= .jar:/usr/local/hbase/bin/../lib/junit-4.11.jar:/usr/local/hbase/bin/../lib= /libthrift-0.9.0.jar:/usr/local/hbase/bin/../lib/log4j-1.2.17.jar:/usr/loca= l/hbase/bin/../lib/management-api-3.0.0-b012.jar:/usr/local/hbase/bin/../li= b/metrics-core-2.2.0.jar:/usr/local/hbase/bin/../lib/netty-3.6.6.Final.jar:= /usr/local/hbase/bin/../lib/paranamer-2.3.jar:/usr/local/hbase/bin/../lib/p= rotobuf-java-2.5.0.jar:/usr/local/hbase/bin/../lib/servlet-api-2.5-6.1.14.j= ar:/usr/local/hbase/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase/bin/../= lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase/bin/../lib/snappy-java-1.0.4.1= .jar:/usr/local/hbase/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase/bin/../li= b/xz-1.0.jar:/usr/local/hbase/bin/../lib/zookeeper-3.4.6.jar:/usr/local/had= oop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.ja= r:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/l= ocal/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/= share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/c= ommon/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoo= p-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-co= mpiler-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5= .jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/h= adoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hado= op/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/l= ib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/comm= ons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-el-1.= 0.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/lo= cal/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/h= adoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hado= op/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/com= mon/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api= -2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/= hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hado= op/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hado= op-auth-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configu= ration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-= 1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/u= sr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/= hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/shar= e/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/co= mmon/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/comm= ons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-jav= a-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.= 23.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.ja= r:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/loca= l/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/= share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/= share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/com= mon/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common= /lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.= 52.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/l= ocal/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop= /share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/comm= on/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/common= s-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collectio= ns-3.2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/= usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/loca= l/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share= /hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/share/hadoop/c= ommon/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/common= s-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-serve= r-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/= usr/local/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop= /share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/had= oop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/apache= ds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/j= ersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.= 0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.5.0.jar:/usr/local= /hadoop/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/usr/local/hadoop= /share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.j= ar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/l= ocal/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/shar= e/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/= hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.= 0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/= hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/h= adoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/= lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jasper-runt= ime-5.5.23.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9= .13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/= local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share= /hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hado= op/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-= core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/u= sr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hado= op/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hado= op/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/j= sr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/lo= cal/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/usr/local/hadoop/= share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/= hdfs/hadoop-hdfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-= io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/lo= cal/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/sh= are/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoo= p/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-= 3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/loc= al/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/sha= re/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib= /leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.= 2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoo= p/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/= yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/proto= buf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.1= 3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local= /hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/h= adoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/ya= rn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson= -core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1= .26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local= /hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hado= op/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/= yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey= -client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar= :/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/h= adoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop= /share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/had= oop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jet= tison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.= 1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/lo= cal/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/sh= are/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/yarn/li= b/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/= usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/usr= /local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launc= her-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.5.0.= jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:= /usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/usr/local/ha= doop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/usr/local/ha= doop/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/usr/local/hadoop/share= /hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/usr/local= /hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/usr/loc= al/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.= 5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemana= ger-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.j= ar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/loca= l/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/usr/local= /hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/ha= doop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/map= reduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/= lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:= /usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local= /hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoo= p/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/had= oop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/ma= preduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/l= ib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib= /log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapp= er-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-= 1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:= /usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hado= op/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/= share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/ha= doop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/map= reduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapr= educe-client-core-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop= -mapreduce-client-hs-plugins-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapre= duce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/usr/local/hadoop/share/hado= op/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/usr/local/hadoop/share/h= adoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/usr/local/hadoop/= share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/u= sr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.ja= r:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclien= t-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examp= les-2.5.0.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local= /hadoop/etc/hadoop/
2014-09-26 23:51:25,253 INFO =C2=A0[main] zoo= keeper.ZooKeeper: Client environment:java.library.path=3D/usr/local/hadoop/= lib
2014-09-26 23:51:25,253 INFO =C2=A0[main] zookeeper.ZooKeeper= : Client environment:java.io.tmpdir=3D/tmp
2014-09-26 23:51:25,25= 3 INFO =C2=A0[main] zookeeper.ZooKeeper: Client environment:java.compiler= =3D<NA>
2014-09-26 23:51:25,253 INFO =C2=A0[main] zookeeper= .ZooKeeper: Client environment:os.name=3DLinux
2014-09-26 23:51:25,253 INFO =C2=A0[main] zo= okeeper.ZooKeeper: Client environment:os.arch=3Damd64
2014-09-26 = 23:51:25,253 INFO =C2=A0[main] zookeeper.ZooKeeper: Client environment:os.v= ersion=3D3.10.42-52.145.amzn1.x86_64
2014-09-26 23:51:25,253 INFO= =C2=A0[main] zookeeper.ZooKeeper: Client environment:user.name=3Dhbase
2014-09-26 23:51:= 25,253 INFO =C2=A0[main] zookeeper.ZooKeeper: Client environment:user.home= =3D/home/hbase
2014-09-26 23:51:25,253 INFO =C2=A0[main] zookeepe= r.ZooKeeper: Client environment:user.dir=3D/usr/local/hbase-0.98.6.1-1a/bin=
2014-09-26 23:51:25,254 INFO =C2=A0[main] zookeeper.ZooKeeper: I= nitiating client connection, connectString=3Dmaster1.hbasex1.test.cloud= .domain.tv:2181,master0.hbasex1.test.cloud.domain.tv:2181,m= aster2.hbasex1.test.cloud.domain.tv:2181 sessionTimeout=3D90000 watcher= =3Dhconnection-0x6bf3ede0, quorum=3Dmaster1.hbasex1.test.cloud.domain.t= v:2181,master0.hbasex1.test.cloud.domain.tv:2181,master2.hb= asex1.test.cloud.domain.tv:2181, baseZNode=3D/hbase
2014-09-2= 6 23:51:25,270 INFO =C2=A0[main-SendThread(master1.hbasex1.test.cloud.d= omain.tv:2181)] zookeeper.ClientCnxn: Opening socket connection to serv= er master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181. Will not attempt to authenticate using SASL (unknown error)
2014-09-26 23:51:25,282 INFO =C2=A0[mai= n-SendThread(master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.= ClientCnxn: Session establishment complete on server master= 1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, sessionid =3D 0x248afb= 6109b007f, negotiated timeout =3D 40000
2014-09-26 23:51:25,308 I= NFO =C2=A0[main] mapreduce.TableOutputFormat: Created table instance for mo= deration
2014-09-26 23:51:25,469 INFO =C2=A0[main] zookeeper.Reco= verableZooKeeper: Process identifier=3Dhconnection-0xa7a371 connecting to Z= ooKeeper ensemble=3Dmaster1.hbase1a.test.cloud.domain.tv:2181,m= aster0.hbase1a.test.cloud.domain.tv:2181,master2.hbase1a.test.cloud= .domain.tv:2181
2014-09-26 23:51:25,469 INFO =C2=A0[main] zoo= keeper.ZooKeeper: Initiating client connection, connectString=3Dmaster1= .hbase1a.test.cloud.domain.tv:2181,master0.hbase1a.test.cloud.domai= n.tv:2181,master2.hbase1a.test.cloud.domain.tv:2181 sessionTime= out=3D90000 watcher=3Dhconnection-0xa7a371, quorum=3Dmaster1.hbase1a.te= st.cloud.domain.tv:2181,master0.hbase1a.test.cloud.domain.tv:2181,master2.hbase1a.test.cloud.domain.tv:2181, baseZNode=3D/hbase
2014-09-26 23:51:25,470 INFO =C2=A0[main-SendThread(master2.hba= se1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening socket c= onnection to server master2.hbase1a.test.cloud.domain.tv/1= 72.16.0.145:2181. Will not attempt to authenticate using SASL (unknown = error)
2014-09-26 23:51:25,470 INFO =C2=A0[main-SendThread(ma= ster2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket = connection established to master2.hbase1a.test.cloud.domai= n.tv/172.16.0.145:2181, initiating session
2014-09-26 23:51:2= 5,472 INFO =C2=A0[main-SendThread(master2.hbase1a.test.cloud.domain.tv:= 2181)] zookeeper.ClientCnxn: Session establishment complete on server <= a href=3D"http://master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181" ta= rget=3D"_blank">master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181,= sessionid =3D 0x34877671747028e, negotiated timeout =3D 40000
20= 14-09-26 23:51:25,490 INFO =C2=A0[main] util.RegionSizeCalculator: Calculat= ing region sizes for table "moderation".
2014-09-26 23:= 51:25,686 DEBUG [main] util.RegionSizeCalculator: Region moderation,,141177= 4704618.68a1839eacfa355028522286caaf7cd5. has size 0
2014-09-26 2= 3:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region sizes calculated=
2014-09-26 23:51:25,709 DEBUG [main] client.ClientSmallScanner: Finished with=20 small scan at {ENCODED =3D> 1588230740, NAME =3D> 'hbase:meta,,1&= #39;,=20 STARTKEY =3D> '', ENDKEY =3D> ''}
2014-09-2= 6 23:51:25,722 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: spli= t -> 0 -> ip-172-16-0-247.us-west-1.compute.internal:,
2014= -09-26 23:51:25,739 INFO =C2=A0[main] mapreduce.JobSubmitter: number of spl= its:1
2014-09-26 23:51:25,748 INFO =C2=A0[main] Configuration.deprecation:=20 io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-09-26 23:51:25,895 INFO =C2=A0[main] mapreduce.JobSubmitter: S= ubmitting tokens for job: job_1411771844660_0004
2014-09-26 23:51= :26,033 INFO =C2=A0[main] impl.YarnClientImpl: Submitted application applic= ation_1411771844660_0004
2014-09-26 23:51:26,066 INFO =C2=A0[main= ] mapreduce.Job: The url to track the job: http://master2.hbase1a.test.cloud.domain.tv:8088/proxy/applicat= ion_1411771844660_0004/
2014-09-26 23:51:26,067 INFO =C2=A0[m= ain] mapreduce.Job: Running job: job_1411771844660_0004
2014-09-2= 6 23:51:29,082 INFO =C2=A0[main] mapreduce.Job: Job job_1411771844660_0004 = running in uber mode : false
2014-09-26 23:51:29,084 INFO =C2=A0[= main] mapreduce.Job: =C2=A0map 0% reduce 0%
2014-09-26 23:51:29,094 INFO =C2=A0[main] mapreduce.Job: Job job_1411771844660_0004= =20 failed with state FAILED due to: Application=20 application_1411771844660_0004 failed 2 times due to AM Container for=20 appattempt_1411771844660_0004_000002 exited with =C2=A0exitCode: -1000 due = to: File file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_000= 4/job.jar does not exist
.Failing this attempt.. Failing the appl= ication.
2014-09-26 23:51:29,111 INFO =C2=A0[main] mapreduce.Job:= Counters: 0

The directory on the AM (master= ) from which the command is initiated as hbase user:
[hbase@= master2 bin]$ ls -lah /tmp/hadoop-yarn/staging/hbase/.staging/job_141177184= 4660_0004/
total 3.7M
drwx------ =C2=A03 hbase hbase = =C2=A04.0K Sep 26 23:51 .
drwx------ 10 hbase hadoop 4.0K Sep 26 = 23:51 ..
-rw-r--r-- =C2=A01 hbase hbase =C2=A03.5M Sep 26 23:51 j= ob.jar
-rw-rw-r-- =C2=A01 hbase hbase =C2=A0 28K Sep 26 23:51 .jo= b.jar.crc
-rw-r--r-- =C2=A01 hbase hbase =C2=A0 111 Sep 26 23:51 = job.split
-rw-rw-r-- =C2=A01 hbase hbase =C2=A0 =C2=A012 Sep 26 2= 3:51 .job.split.crc
-rw-r--r-- =C2=A01 hbase hbase =C2=A0 =C2=A05= 6 Sep 26 23:51 job.splitmetainfo
-rw-rw-r-- =C2=A01 hbase hbase = =C2=A0 =C2=A012 Sep 26 23:51 .job.splitmetainfo.crc
-rw-r--r-- = =C2=A01 hbase hbase =C2=A0102K Sep 26 23:51 job.xml
-rw-rw-r-- = =C2=A01 hbase hbase =C2=A0 824 Sep 26 23:51 .job.xml.crc
drwx----= -- =C2=A02 hbase hbase =C2=A04.0K Sep 26 23:51 libjars

=
The same directory is empty on each NM.

Heres also the log from a node that got the job/application:
2014-09-26 23:51:26,635 INFO SecurityLogger.org.apache.hadoop.ipc.Server:= Auth successful for appattempt_1411771844660_0004_000001 (auth:SIMPLE)
2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanag= er.containermanager.ContainerManagerImpl: Start request for container_14117= 71844660_0004_01_000001 by user hbase
2014-09-26 23:51:26,639 INF= O org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManag= erImpl: Creating a new application reference for app application_1411771844= 660_0004
2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.serv= er.nodemanager.NMAuditLogger: USER=3Dhbase =C2=A0 =C2=A0 =C2=A0 =C2=A0IP=3D172.16.0.145 OPERATION=3DStar= t Container Request =C2=A0 =C2=A0 =C2=A0 TARGET=3DContainerManageImpl =C2=A0 =C2=A0 =C2=A0RESULT=3DSUCCESS = =C2=A0APPID=3Dapplication_1411771844660_0004 =C2=A0 =C2=A0CONTAINERID=3Dcon= tainer_1411771844660_0004_01_000001
2014-09-26 23:51:26,639 INFO = org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appl= ication: Application application_1411771844660_0004 transitioned from NEW t= o INITING
2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.ser= ver.nodemanager.containermanager.application.Application: Adding container_= 1411771844660_0004_01_000001 to application application_1411771844660_0004<= /div>
2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodema= nager.containermanager.application.Application: Application application_141= 1771844660_0004 transitioned from INITING to RUNNING
2014-09-26 2= 3:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager= .container.Container: Container container_1411771844660_0004_01_000001 tran= sitioned from NEW to LOCALIZING
2014-09-26 23:51:26,640 INFO org.= apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got eve= nt CONTAINER_INIT for appId application_1411771844660_0004
2014-0= 9-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containerm= anager.localizer.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/= hbase/.staging/job_1411771844660_0004/job.jar transitioned from INIT to DOW= NLOADING
2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.serv= er.nodemanager.containermanager.localizer.LocalizedResource: Resource file:= /tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.splitmet= ainfo transitioned from INIT to DOWNLOADING
2014-09-26 23:51:26,6= 40 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localize= r.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/= job_1411771844660_0004/job.split transitioned from INIT to DOWNLOADING
2014-09-26 23:51:26,641 INFO org.apache.hadoop.yarn.server.nodemanage= r.containermanager.localizer.LocalizedResource: Resource file:/tmp/hadoop-y= arn/staging/hbase/.staging/job_1411771844660_0004/job.xml transitioned from= INIT to DOWNLOADING
2014-09-26 23:51:26,641 INFO org.apache.hado= op.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationS= ervice: Created localizer for container_1411771844660_0004_01_000001
<= div>2014-09-26 23:51:26,645 INFO org.apache.hadoop.yarn.server.nodemanager.= containermanager.localizer.ResourceLocalizationService: Writing credentials= to the nmPrivate file /data0/hadoop/yarn/local/nmPrivate/container_1411771= 844660_0004_01_000001.tokens. Credentials list:=C2=A0
2014-09-26 = 23:51:26,648 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContaine= rExecutor: Initializing user hbase
2014-09-26 23:51:26,661 INFO o= rg.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying = from /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000= 001.tokens to /data0/hadoop/yarn/local/usercache/hbase/appcache/application= _1411771844660_0004/container_1411771844660_0004_01_000001.tokens
2014-09-26 23:51:26,661 INFO org.apache.hadoop.yarn.server.nodemanager.Def= aultContainerExecutor: CWD set to /data0/hadoop/yarn/local/usercache/hbase/= appcache/application_1411771844660_0004 =3D file:/data0/hadoop/yarn/local/u= sercache/hbase/appcache/application_1411771844660_0004
2014-09-26= 23:51:26,696 INFO org.apache.hadoop.yarn.server.nodemanager.containermanag= er.localizer.ResourceLocalizationService: DEBUG: FAILED { file:/tmp/hadoop-= yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar, 1411775485000, = PATTERN, (?:classes/|lib/).* }, File file:/tmp/hadoop-yarn/staging/hbase/.s= taging/job_1411771844660_0004/job.jar does not exist
2014-09-26 2= 3:51:26,697 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager= .localizer.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/= .staging/job_1411771844660_0004/job.jar(->/data0/hadoop/yarn/local/userc= ache/hbase/appcache/application_1411771844660_0004/filecache/10/job.jar) tr= ansitioned from DOWNLOADING to FAILED
2014-09-26 23:51:26,697 INF= O org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Cont= ainer: Container container_1411771844660_0004_01_000001 transitioned from L= OCALIZING to LOCALIZATION_FAILED
2014-09-26 23:51:26,697 INFO org= .apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalReso= urcesTrackerImpl: Container container_1411771844660_0004_01_000001 sent REL= EASE event on a resource request { file:/tmp/hadoop-yarn/staging/hbase/.sta= ging/job_1411771844660_0004/job.jar, 1411775485000, PATTERN, (?:classes/|li= b/).* } not present in cache.
2014-09-26 23:51:26,698 INFO org.ap= ache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting abso= lute path : /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1= 411771844660_0004/container_1411771844660_0004_01_000001
2014-09-= 26 23:51:26,698 INFO org.apache.hadoop.yarn.server.nodemanager.containerman= ager.localizer.ResourceLocalizationService: Unknown localizer with localize= rId container_1411771844660_0004_01_000001 is sending heartbeat. Ordering i= t to DIE
2014-09-26 23:51:26,698 WARN org.apache.hadoop.yarn.serv= er.nodemanager.DefaultContainerExecutor: delete returned false for path: [/= data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_= 0004/container_1411771844660_0004_01_000001]
2014-09-26 23:51:26,= 698 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor= : Deleting absolute path : /data1/hadoop/yarn/local/usercache/hbase/appcach= e/application_1411771844660_0004/container_1411771844660_0004_01_000001
2014-09-26 23:51:26,699 WARN org.apache.hadoop.yarn.server.nodemanag= er.DefaultContainerExecutor: delete returned false for path: [/data1/hadoop= /yarn/local/usercache/hbase/appcache/application_1411771844660_0004/contain= er_1411771844660_0004_01_000001]
2014-09-26 23:51:26,699 WARN org= .apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=3Dhbase =C2=A0 =C2=A0 =C2=A0 =C2=A0OPERATION=3DContainer Finished - F= ailed =C2=A0=20 TARGET=3DContainerImpl =C2=A0 =C2=A0RESULT=3DFAILURE =C2=A0DESCRIPTION=3DCo= ntainer failed=20 with state: LOCALIZATION_FAILED =C2=A0 =C2=A0APPID=3Dapplication_1411771844= 660_0004 =C2=A0 =C2=A0CONTAINERID=3Dcontainer_1411771844660_0004_01_000001<= /div>
2014-09-26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodema= nager.containermanager.container.Container: Container container_14117718446= 60_0004_01_000001 transitioned from LOCALIZATION_FAILED to DONE
2= 014-09-26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodemanager.conta= inermanager.application.Application: Removing container_1411771844660_0004_= 01_000001 from application application_1411771844660_0004
2014-09= -26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodemanager.containerma= nager.AuxServices: Got event CONTAINER_STOP for appId application_141177184= 4660_0004
2014-09-26 23:51:27,629 INFO org.apache.hadoop.yarn.ser= ver.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM= context: [container_1411771844660_0004_01_000001]
2014-09-26 23:= 51:28,252 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.m= onitor.ContainersMonitorImpl: Stopping resource-monitoring for container_14= 11771844660_0004_01_000001
2014-09-26 23:51:29,631 INFO org.apach= e.hadoop.yarn.server.nodemanager.containermanager.application.Application: = Application application_1411771844660_0004 transitioned from RUNNING to APP= LICATION_RESOURCES_CLEANINGUP
2014-09-26 23:51:29,632 INFO org.ap= ache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting abso= lute path : /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1= 411771844660_0004
2014-09-26 23:51:29,632 INFO org.apache.hadoop.= yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : = /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660= _0004
2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.= nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for ap= pId application_1411771844660_0004
2014-09-26 23:51:29,632 INFO o= rg.apache.hadoop.yarn.server.nodemanager.containermanager.application.Appli= cation: Application application_1411771844660_0004 transitioned from APPLIC= ATION_RESOURCES_CLEANINGUP to FINISHED
2014-09-26 23:51:29,632 IN= FO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.No= nAggregatingLogHandler: Scheduling Log Deletion for application: applicatio= n_1411771844660_0004, with delay of 10800 seconds

Thank you in advance for any advice
= --
Iain Wright

T= his email message is confidential, intended only for the recipient(s) named= above and may contain information that is privileged, exempt from disclosu= re under applicable law. If you are not the intended recipient, do not disc= lose or disseminate the message to anyone except the intended recipient. If= you have received this message in error, or are not the named recipient(s)= , please immediately notify the sender by return email, and delete all copi= es of this message.
--001a11339b5876d628050438d548--