hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: My first TEZ job fails
Date Mon, 30 May 2016 11:54:11 GMT
Hi Gopal,

can you add anything to this please?

thanks

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com



On 21 May 2016 at 09:12, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:

> thanks,
>
> This works now at least the jobs succeed as seen below
>
> [image: Inline images 1]
>
> However, when I run this below I don't see any result
>
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hadoop jar
> ./tez-examples-0.7.1.jar orderedwordcount /tmp/input/test.txt
> /tmp/out/test.log
>
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin>
>
>
> This is the input file
>
> cat /tmp/input/test.txt
> foo
> bar
> foo
> bar
> foo
>
> But nothing in /tmp/out/test.log and the file is not created
>
> cat /tmp/out/test.log
> cat: /tmp/out/test.log: cannot open [No such file or directory]
>
>
>
> Any ideas what is happening?
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 May 2016 at 01:48, Hitesh Shah <hitesh@apache.org> wrote:
>
>> To clarify,
>>
>> if you do the following:
>>
>> hdfs -mkdir /tez/
>> hdfs -put $TEZ_HOME/share/tez-0.7.1.tar.gz /tez/
>>
>> Then tez.lib.uris should be “/tez/tez-0.7.1.tar.gz”
>>
>> — HItesh
>>
>>
>> > On May 20, 2016, at 5:11 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>> wrote:
>> >
>> > I think the instructions can be simplified as follows
>> >
>> > wherever you have your tarball say $TEZ_HOME/share/tez-0.7.1.tar.gz
>> >
>> > you create the said absolute directory in your hdfs and put you tarball
>> there
>> >
>> > Also your uris should refer to the said absolute path (a single
>> directory structure)
>> >
>> >   <name>tez.lib.uris</name>
>> >     <value>/usr/lib/apache-tez-0.7.1-bin/share</value>
>> >
>> >
>> > Dr Mich Talebzadeh
>> >
>> > LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> >
>> > http://talebzadehmich.wordpress.com
>> >
>> >
>> > On 21 May 2016 at 01:06, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>> wrote:
>> > Thanks Hitesh,
>> >
>> > Here
>> >
>> > tez.lib.uris should be a single value and the value should be the
>> absolute path pointing the tez tarball on HDFS
>> >
>> > OK
>> >
>> > are we talking about the one under $TEZ_HOME/share?
>> >
>> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr
>> > -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
>> >
>> >
>> >
>> > And my uris would be the abosulte path to the above tarball under
>> /usr/lib/apache-tez-0.7.1-bin/share
>> >
>> > cat tez-site.xml
>> > <configuration>
>> >   <property>
>> >     <name>tez.version</name>
>> >     <value>0.7.1</value>
>> >   </property>
>> >   <property>
>> >     <name>tez.lib.uris</name>
>> >     <value>/usr/lib/apache-tez-0.7.1-bin/share</value>
>> >   </property>
>> > </configuration>
>> > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/Hadoop>
>> >
>> >
>> >
>> > Dr Mich Talebzadeh
>> >
>> > LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> >
>> > http://talebzadehmich.wordpress.com
>> >
>> >
>> > On 21 May 2016 at 00:58, Hitesh Shah <hitesh@apache.org> wrote:
>> > Hello Mich,
>> >
>> > Yes those are the instructions.
>> >
>> > tez.lib.uris should be a single value and the value should be the
>> absolute path pointing the tez tarball on HDFS. The tarball to be uploaded
>> to HDFS can be found at apache-tez-{x.y.z}/share/tez.tar.gz ( or
>> tez-dist/target/tez-x.y.x.tar.gz if you are using your local build ).
>> >
>> > thanks
>> > — Hitesh
>> >
>> > > On May 20, 2016, at 4:39 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>> > >
>> > > This is the instruction?
>> > >
>> > > Created by Hitesh Shah, last modified on May 02, 2016 Go to start of
>> metadata
>> > > Making use of the Tez Binary Release tarball
>> > >
>> > >       • If the binary tarball's name does not include anything
>> referring to a hadoop version, then this implies that the tarball was
>> compiled against the hadoop version that the Tez release compiles against
>> by default. For example, for 0.7.0 and 0.8.0, the default hadoop version
>> used is 2.6.0 ( this can be found by looking for the hadoop.version
>> property in the top-level pom.xml in the source tarball for the release).
>> > >
>> > >       • The tarball structure is as follows:
>> > >
>> > >
>> > > ?
>> > > apache-tez-{x.y.z}/
>> > >                   /tez*.jar
>> > >                   /lib/*.jar
>> > >                   /conf/tez*.xml.template
>> > >                   /share/tez.tar.gz
>> > >       • Set up Tez by following INSTALL.txt and use
>> apache-tez-{x.y.z}/share/tez.tar.gz as the full tarball to be uploaded to
>> HDFS.
>> > >       • Use the config templates under apache-tez-{x.y.z}/conf/ to
>> create the tez-site.xml as needed in an appropriate conf directory. If you
>> end up using apache-tez-{x.y.z}/conf/, then do an export
>> TEZ_CONF_DIR="apache-tez-{x.y.z}/conf/"
>> > >       • Add
>> "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}" to
>> HADOOP_CLASSPATH so as to get the tez client jars onto the classpath
>> invoked when using the "bin/hadoop jar" command to run an example job.
>> > >
>> > >
>> > > Dr Mich Talebzadeh
>> > >
>> > > LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> > >
>> > > http://talebzadehmich.wordpress.com
>> > >
>> > >
>> > > On 21 May 2016 at 00:37, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>> wrote:
>> > > Thanks both
>> > >
>> > > so this is the file that needs to go in hdfs correct?
>> > >
>> > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr
>> tez-0.7.1.tar.gz
>> > > -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
>> > >
>> > >
>> > > In hdfs I have now
>> > >
>> > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls
>> /usr/lib/apache-tez-0.7.1-bin
>> > >
>> > > -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31
>> /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
>> > >
>> > >
>> > > Now I only installed tez under
>> > >
>> > > /usr/lib/apache-tez-0.7.1-bin
>> > >
>> > > MY Hadoop is installed in
>> > >
>> > > echo $HADOOP_HOME
>> > > /home/hduser/hadoop-2.6.0
>> > >
>> > > and my xml-site in in
>> > >
>> > > $HADOOP_HOME/etc/hadoop
>> > >
>> > > OK and this is my sml-site content
>> > >
>> > > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat
>> tez-site.xml
>> > > <configuration>
>> > >   <property>
>> > >     <name>tez.version</name>
>> > >     <value>0.7.1</value>
>> > >   </property>
>> > >   <property>
>> > >     <name>tez.lib.uris</name>
>> > >
>>  <value>/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib</value>
>> > >   </property>
>> > > </configuration>
>> > >
>> > > Is the red correct please?
>> > >
>> > >
>> > >
>> > > Dr Mich Talebzadeh
>> > >
>> > > LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> > >
>> > > http://talebzadehmich.wordpress.com
>> > >
>> > >
>> > > On 21 May 2016 at 00:24, Bikas Saha <bikas@apache.org> wrote:
>> > > >> tez.lib.uris assumes that paths are based on the default fs and
>> therefore if your setup is using HDFS as default, the paths /usr/lib would
>> be invalid
>> > >
>> > > Are you sure? The below paths looks right to me except that the
>> contents of the directories are wrong.
>> > >
>> > > <name>tez.lib.uris</name>
>> > >
>> <value>/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib</value>
>> > >
>> > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
>> /usr/lib/apache-tez-0.7.1-bin Found 2 items
>> > > -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15
>> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
>> > > drwxr-xr-x   - hduser supergroup          0 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Hitesh Shah [mailto:hitesh@apache.org]
>> > > Sent: Friday, May 20, 2016 4:18 PM
>> > > To: user@tez.apache.org
>> > > Subject: Re: My first TEZ job fails
>> > >
>> > > Can you try the instructions mentioned at
>> https://cwiki.apache.org/confluence/display/TEZ/Tez+Release+FAQ ?
>> > >
>> > > tez.lib.uris assumes that paths are based on the default fs and
>> therefore if your setup is using HDFS as default, the paths /usr/lib would
>> be invalid.
>> > >
>> > > — HItesh
>> > >
>> > > > On May 20, 2016, at 3:39 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>> > > >
>> > > > Still failing with /apache-tez-0.7.1-bin I am afraid.
>> > > >
>> > > > OK this is my tez-site.xml
>> > > >
>> > > > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat
>> tez-site.xml
>> > > > <configuration>
>> > > >   <property>
>> > > >     <name>tez.version</name>
>> > > >     <value>0.7.1</value>
>> > > >   </property>
>> > > >   <property>
>> > > >     <name>tez.lib.uris</name>
>> > > >
>>  <value>/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib</value>
>> > > >   </property>
>> > > > </configuration>
>> > > >
>> > > > This is what I have put in hdfs directory
>> > > >
>> > > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
>> > > > /usr/lib/apache-tez-0.7.1-bin Found 2 items
>> > > > -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15
>> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
>> > > > drwxr-xr-x   - hduser supergroup          0 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib
>> > > >
>> > > > Also I put all /usr/lib/apache-tez-0.7.1-bin/lib/*.jar in
>> > > > /usr/lib/apache-tez-0.7.1-bin/lib
>> > > >
>> > > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
>> > > > /usr/lib/apache-tez-0.7.1-bin/lib Found 22 items
>> > > > -rw-r--r--   2 hduser supergroup     124846 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/RoaringBitmap-0.4.9.jar
>> > > > -rw-r--r--   2 hduser supergroup      41123 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-cli-1.2.jar
>> > > > -rw-r--r--   2 hduser supergroup      58160 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-codec-1.4.jar
>> > > > -rw-r--r--   2 hduser supergroup     588337 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections-3.2.2.jar
>> > > > -rw-r--r--   2 hduser supergroup     751238 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections4-4.1.jar
>> > > > -rw-r--r--   2 hduser supergroup     185140 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-io-2.4.jar
>> > > > -rw-r--r--   2 hduser supergroup     284220 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-lang-2.6.jar
>> > > > -rw-r--r--   2 hduser supergroup    1599627 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-math3-3.1.1.jar
>> > > > -rw-r--r--   2 hduser supergroup    1648200 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/guava-11.0.2.jar
>> > > > -rw-r--r--   2 hduser supergroup     664918 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-common-2.6.0.jar
>> > > > -rw-r--r--   2 hduser supergroup    1509399 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-core-2.6.0.jar
>> > > > -rw-r--r--   2 hduser supergroup     130458 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-client-1.9.jar
>> > > > -rw-r--r--   2 hduser supergroup     147952 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-json-1.9.jar
>> > > > -rw-r--r--   2 hduser supergroup      81743 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/jettison-1.3.4.jar
>> > > > -rw-r--r--   2 hduser supergroup     539912 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/jetty-6.1.26.jar
>> > > > -rw-r--r--   2 hduser supergroup     177131 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/jetty-util-6.1.26.jar
>> > > > -rw-r--r--   2 hduser supergroup      33031 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/jsr305-2.0.3.jar
>> > > > -rw-r--r--   2 hduser supergroup     111908 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/metrics-core-3.1.0.jar
>> > > > -rw-r--r--   2 hduser supergroup     533455 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/protobuf-java-2.5.0.jar
>> > > > -rw-r--r--   2 hduser supergroup     105112 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/servlet-api-2.5.jar
>> > > > -rw-r--r--   2 hduser supergroup      26084 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/slf4j-api-1.7.5.jar
>> > > > -rw-r--r--   2 hduser supergroup       8869 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib/slf4j-log4j12-1.7.5.jar
>> > > >
>> > > > Still getting this error
>> > > >
>> > > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hadoop jar
>> > > > ./tez-examples-0.7.1.jar orderedwordcount /tmp/input/test.txt
>> /tmp/out
>> > > >
>> > > > Application application_1463775706014_0008 failed 2 times due to
>> Error
>> > > > launching appattempt_1463775706014_0008_000002. Got exception:
>> > > > org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
>> > > >
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)
>> > > > I at
>> > > >
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(Ya
>> > > > rnProtos.java:2616) at
>> > > >
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hash
>> > > > Code(YarnProtos.java:3154) at
>> > > >
>> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifier
>> > > > Proto.hashCode(YarnSecurityTokenProtos.java:410)
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIden
>> > > > tifier.java:126) at java.util.HashMap.hash(HashMap.java:338)
>> > > >
>> > > > My protoc is 2.6.1 but everything else Hadoop, spark, hive, etx all
>> work fine!
>> > > >
>> > > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> protoc --version
>> > > > libprotoc 2.6.1
>> > > >
>> > > > Thanks
>> > > >
>> > > > Dr Mich Talebzadeh
>> > > >
>> > > > LinkedIn
>> > > >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC
>> > > > dOABUrV8Pw
>> > > >
>> > > > http://talebzadehmich.wordpress.com
>> > > >
>> > > >
>> > > > On 20 May 2016 at 23:04, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>> wrote:
>> > > > Thanks Bikas,
>> > > >
>> > > > Downloaded it and installed under /usr/lib/apache-tez-0.7.1-bin
>> > > >
>> > > > Are these env variables correct?
>> > > >
>> > > > export TEZ_HOME=/usr/lib/apache-tez-0.7.1-bin
>> > > > export TEZ_JARS=/usr/lib/apache-tez-0.7.1-bin
>> > > >
>> > > >
>> > > > Dr Mich Talebzadeh
>> > > >
>> > > > LinkedIn
>> > > >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC
>> > > > dOABUrV8Pw
>> > > >
>> > > > http://talebzadehmich.wordpress.com
>> > > >
>> > > >
>> > > > On 20 May 2016 at 22:43, Bikas Saha <bikas@apache.org> wrote:
>> > > > Yes. Almost all Hadoop ecosystem components are on 2.5.1 for protoc.
>> > > >
>> > > >
>> > > >
>> > > > Btw, unless you need something specific from 0.8 branch and are on
>> the right Hadoop version 2.6.0 or higher, you could use the recent 0.7.1
>> binary release so you wont have to build the code yourself. May save you
>> from these mismatch issues for the most part.
>> > > >
>> > > >
>> > > >
>> > > > Bikas
>> > > >
>> > > >
>> > > >
>> > > > From: Mich Talebzadeh [mailto:mich.talebzadeh@gmail.com]
>> > > > Sent: Friday, May 20, 2016 2:25 PM
>> > > > To: user@tez.apache.org
>> > > > Subject: Re: My first TEZ job fails
>> > > >
>> > > >
>> > > >
>> > > > the issue is that I have protobuf-2.6.1 and I believe I modified
>> the pom.xml under apache-tez-0.8.3-src and changed it from 2.5.0 to 2.6.1.
>> > > >
>> > > >
>> > > >
>> > > > Now I have to install protobuf-2.5.0 and It is not working.
>> > > >
>> > > >
>> > > >
>> > > > Dr Mich Talebzadeh
>> > > >
>> > > > LinkedIn
>> > > >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC
>> > > > dOABUrV8Pw
>> > > >
>> > > > http://talebzadehmich.wordpress.com
>> > > >
>> > > >
>> > > >
>> > > > On 20 May 2016 at 21:46, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>> wrote:
>> > > >
>> > > > sounds like this may ve the issue
>> > > >
>> > > >
>> > > >
>> > > > java.lang.NoSuchMethodError:
>> > > >
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)
>> > > > I
>> > > >
>> > > >
>> > > >
>> > > > Dr Mich Talebzadeh
>> > > >
>> > > > LinkedIn
>> > > >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC
>> > > > dOABUrV8Pw
>> > > >
>> > > > http://talebzadehmich.wordpress.com
>> > > >
>> > > >
>> > > >
>> > > > On 20 May 2016 at 19:28, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>> wrote:
>> > > >
>> > > > Hi Hitesh,
>> > > >
>> > > >
>> > > >
>> > > > This is the content of tez-site.xml]
>> > > >
>> > > >
>> > > >
>> > > > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat
>> tez-site.xml
>> > > > <configuration>
>> > > >   <property>
>> > > >     <name>tez.version</name>
>> > > >     <value>0.8.3</value>
>> > > >   </property>
>> > > >
>> > > >   <property>
>> > > >     <name>tez.lib.uris</name>
>> > > >     <value>/usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/</value>
>> > > >   </property>
>> > > > </configuration>
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > This is what is in hdfs
>> > > >
>> > > >
>> > > >
>> > > > drwxr-xr-x   - hduser supergroup          0 2016-05-20 19:25
>> /usr/lib/tez-0.8.3/lib
>> > > > -rw-r--r--   2 hduser supergroup   42821282 2016-05-20 09:03
>> /usr/lib/tez-0.8.3/tez-0.8.3.tar.gz
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Tez is installed in
>> > > >
>> > > >
>> > > >
>> > > > cd $TEZ_HOME
>> > > > hduser@rhes564: /usr/lib/tez-0.8.3>
>> > > >
>> > > >
>> > > >
>> > > > tez-site.xml is soft linked in $TEZ_HOME as follows
>> > > >
>> > > >
>> > > >
>> > > > hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml lrwxrwxrwx
1
>> > > > hduser hadoop 49 May 20 19:22 tez-site.xml ->
>> > > > /home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml
>> > > >
>> > > >
>> > > >
>> > > > This is error
>> > > >
>> > > >
>> > > >
>> > > > Application application_1463758195355_0004 failed 2 times due to
>> Error
>> > > > launching appattempt_1463758195355_0004_000002. Got exception:
>> > > > org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
>> > > >
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)
>> > > > I
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(Ya
>> > > > rnProtos.java:2616)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hash
>> > > > Code(YarnProtos.java:3154)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifier
>> > > > Proto.hashCode(YarnSecurityTokenProtos.java:410)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIden
>> > > > tifier.java:126)
>> > > >
>> > > > at java.util.HashMap.hash(HashMap.java:338)
>> > > >
>> > > > at java.util.HashMap.put(HashMap.java:611)
>> > > >
>> > > > at java.util.HashSet.add(HashSet.java:219)
>> > > >
>> > > > at
>> javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
>> > > >
>> > > > at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369)
>> > > >
>> > > > at
>> javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(Us
>> > > > erGroupInformation.java:1400)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerMa
>> > > > nagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerMa
>> > > > nagerImpl.startContainers(ContainerManagerImpl.java:699)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocol
>> > > >
>> PBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl
>> > > > .java:60)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerMana
>> > > >
>> gementProtocolService$2.callBlockingMethod(ContainerManagementProtocol
>> > > > .java:95)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call
>> > > > (ProtobufRpcEngine.java:619)
>> > > >
>> > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>> > > >
>> > > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>> > > >
>> > > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>> > > >
>> > > > at java.security.AccessController.doPrivileged(Native Method)
>> > > >
>> > > > at javax.security.auth.Subject.doAs(Subject.java:422)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
>> > > > ion.java:1628)
>> > > >
>> > > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>> > > >
>> > > > at org.apache.hadoop.ipc.Client.call(Client.java:1468)
>> > > >
>> > > > at org.apache.hadoop.ipc.Client.call(Client.java:1399)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngi
>> > > > ne.java:232)
>> > > >
>> > > > at com.sun.proxy.$Proxy80.startContainers(Unknown Source)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolP
>> > > >
>> BClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.ja
>> > > > va:96)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.la
>> > > > unch(AMLauncher.java:119)
>> > > >
>> > > > at
>> > > >
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.ru
>> > > > n(AMLauncher.java:254)
>> > > >
>> > > > at
>> > > >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>> > > > ava:1142)
>> > > >
>> > > > at
>> > > >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>> > > > java:617)
>> > > >
>> > > > at java.lang.Thread.run(Thread.java:745)
>> > > >
>> > > > . Failing the application.
>> > > >
>> > > > Thanks
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Dr Mich Talebzadeh
>> > > >
>> > > > LinkedIn
>> > > >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC
>> > > > dOABUrV8Pw
>> > > >
>> > > > http://talebzadehmich.wordpress.com
>> > > >
>> > > >
>> > > >
>> > > > On 20 May 2016 at 19:06, Hitesh Shah <hitesh@apache.org> wrote:
>> > > >
>> > > > Logs from `bin/yarn logs -applicationId
>> application_1463758195355_0002` would be more useful to debug your setup
>> issue. The RM logs usually do not shed much light on why an application
>> failed.
>> > > > Can you confirm that you configured tez.lib.uris correctly to point
>> to the tez tarball on HDFS (tez tar should be the one obtained from
>> tez-dist/target/tez-0.8.3.tar.gz) ?
>> > > >
>> > > > — Hitesh
>> > > >
>> > > >
>> > > > > On May 20, 2016, at 10:24 AM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>> > > > >
>> > > > > Hi,
>> > > > >
>> > > > > I have just compiled and installed TEZ, trying to do a test with
>> > > > >
>> > > > > hadoop jar ./tez-examples-0.8.3.jar orderedwordcount
>> > > > > /tmp/input/test.txt /tmp/out
>> > > > >
>> > > > > The job fails as follows. This is from yarn log
>> > > > >
>> > > > > 2016-05-20 18:19:26,945 INFO
>> > > > > SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful
for
>> > > > > appattempt_1463758195355_0002_000001 (auth:SIMPLE)
>> > > > > 2016-05-20 18:19:26,950 WARN org.apache.hadoop.ipc.Server: IPC
>> > > > > Server handler 0 on 59093, call
>> > > > >
>> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContai
>> > > > > ners from 50.140.197.217:46784 Call#2 Retry#0
>> > > > > java.lang.NoSuchMethodError:
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
>> > > > >         at
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
>> > > > >         at
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
>> > > > >         at
>> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
>> > > > >         at
>> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
>> > > > >         at java.util.HashMap.hash(HashMap.java:338)
>> > > > >         at java.util.HashMap.put(HashMap.java:611)
>> > > > >         at java.util.HashSet.add(HashSet.java:219)
>> > > > >         at
>> javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
>> > > > >         at
>> javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369)
>> > > > >         at
>> javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
>> > > > >         at
>> org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
>> > > > >         at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
>> > > > >         at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
>> > > > >         at
>> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
>> > > > >         at
>> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
>> > > > >         at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>> > > > >         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>> > > > >         at
>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>> > > > >         at
>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>> > > > >         at java.security.AccessController.doPrivileged(Native
>> Method)
>> > > > >         at javax.security.auth.Subject.doAs(Subject.java:422)
>> > > > >         at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> > > > >         at
>> > > > > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>> > > > > 2016-05-20 18:19:27,929 WARN
>> > > > >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.Container
>> > > > > ManagerImpl: Event EventType: KILL_CONTAINER sent to absent
>> > > > > container container_1463758195355_0002_01_000001
>> > > > > 2016-05-20 18:19:27,944 INFO
>> > > > > SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful
for
>> > > > > appattempt_1463758195355_0002_000002 (auth:SIMPLE)
>> > > > > 2016-05-20 18:19:27,949 WARN org.apache.hadoop.ipc.Server: IPC
>> > > > > Server handler 0 on 59093, call
>> > > > >
>> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContai
>> > > > > ners from 50.140.197.217:46785 Call#3 Retry#0
>> > > > > java.lang.NoSuchMethodError:
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
>> > > > >         at
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
>> > > > >         at
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
>> > > > >         at
>> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
>> > > > >         at
>> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
>> > > > >         at java.util.HashMap.hash(HashMap.java:338)
>> > > > >         at java.util.HashMap.put(HashMap.java:611)
>> > > > >         at java.util.HashSet.add(HashSet.java:219)
>> > > > >         at
>> javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
>> > > > >         at
>> javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369)
>> > > > >         at
>> javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
>> > > > >         at
>> org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
>> > > > >         at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
>> > > > >         at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
>> > > > >         at
>> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
>> > > > >         at
>> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
>> > > > >         at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>> > > > >         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>> > > > >         at
>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>> > > > >         at
>> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>> > > > >         at java.security.AccessController.doPrivileged(Native
>> Method)
>> > > > >         at javax.security.auth.Subject.doAs(Subject.java:422)
>> > > > >         at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> > > > >         at
>> > > > > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>> > > > > 2016-05-20 18:19:28,931 WARN
>> > > > >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.Container
>> > > > > ManagerImpl: Event EventType: KILL_CONTAINER sent to absent
>> > > > > container container_1463758195355_0002_02_000001
>> > > > > 2016-05-20 18:19:28,932 WARN
>> > > > >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.Container
>> > > > > ManagerImpl: Event EventType: FINISH_APPLICATION sent to absent
>> > > > > application application_1463758195355_0002
>> > > > >
>> > > > > Any ideas will be appreciated!
>> > > > >
>> > > > > Thanks
>> > > > >
>> > > > > Dr Mich Talebzadeh
>> > > > >
>> > > > > LinkedIn
>> > > > >
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcP
>> > > > > CCdOABUrV8Pw
>> > > > >
>> > > > > http://talebzadehmich.wordpress.com
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>>
>>
>

Mime
View raw message