hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mich Talebzadeh" <m...@peridale.co.uk>
Subject RE: Hive on TEZ fails starting
Date Wed, 06 Jan 2016 15:02:57 GMT
Not starting at all!

 

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.


co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This message is for the
designated recipient only, if you are not the intended recipient, you should destroy it immediately.
Any information in this message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility
of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd,
its subsidiaries nor their employees accept any responsibility.

 

From: Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com] 
Sent: 06 January 2016 09:18
To: user@hive.apache.org
Subject: Re: Hive on TEZ fails starting

 

Is the job starting and getting stuck in the reducer like you mentioned in the initial mail?
or the job itself is not starting?  

 

~Rajesh.B

 

On Wed, Jan 6, 2016 at 1:47 PM, Mich Talebzadeh <mich@peridale.co.uk <mailto:mich@peridale.co.uk>
> wrote:

Hi,

 

Thanks for your help. I downloaded and installed snappy libraries as they were missing.

 

Setting Hive execution engine to tez and doing a simple query. Hive is stuck

 

    > set hive.execution.engine=tez;

16/01/06 08:20:22 [main]: DEBUG parse.VariableSubstitution: Substitution is on: tez

 

 

 

In debug mode I get

 

16/01/06 08:22:07 [IPC Client (2116259755 <tel:%282116259755> ) connection to localhost/127.0.0.1:8032
<http://127.0.0.1:8032>  from hduser]: DEBUG ipc.Client: IPC Client (2116259755 <tel:%282116259755>
) connection to localhost/127.0.0.1:8032 <http://127.0.0.1:8032>  from hduser got value
#188

16/01/06 08:22:07 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

16/01/06 08:22:07 [LeaseRenewer:hduser@rhes564:9000]: DEBUG hdfs.LeaseRenewer: Lease renewer
daemon for [] with renew id 1 executed

 

 

Which does not progress. Sounds like the same issue as I saw with Spark.

 

Can I do something like?

 

set tez.master=yarn-client;

 

 

Thanks

 

 

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.


co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This message is for the
designated recipient only, if you are not the intended recipient, you should destroy it immediately.
Any information in this message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility
of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd,
its subsidiaries nor their employees accept any responsibility.

 

From: Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com <mailto:rajesh.balamohan@gmail.com>
] 
Sent: 05 January 2016 11:46


To: user@hive.apache.org <mailto:user@hive.apache.org> 
Subject: Re: Hive on TEZ fails starting

 

Try '  beeline --hiveconf tez.task.launch.env="LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native"
--hiveconf tez.am.launch.env="LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native"
'.  Please check if you have the lib*.so available in the native folder (or point it to the
folder which contains the so files)

 

~Rajesh.B

 

 

On Tue, Jan 5, 2016 at 4:00 PM, Mich Talebzadeh <mich@peridale.co.uk <mailto:mich@peridale.co.uk>
> wrote:

Hi,

 

I have added the following to the LD_LIBRARY_PATH and JAVA_LIBRARY_PATH

 

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native

export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native

 

Trying to use TEZ, I still get the same error

 

0: jdbc:hive2://rhes564:10010/default> set hive.execution.engine=tez;

No rows affected (0.002 seconds)

0: jdbc:hive2://rhes564:10010/default> use oraclehadoop;

No rows affected (0.019 seconds)

0: jdbc:hive2://rhes564:10010/default> select count(1) from sales;

INFO  : Tez session hasn't been created yet. Opening session

INFO  :

 

INFO  : Status: Running (Executing on YARN cluster with App id application_1451986680090_0002)

 

INFO  : Map 1: -/-      Reducer 2: 0/1

INFO  : Map 1: 0/1      Reducer 2: 0/1

INFO  : Map 1: 0(+1)/1  Reducer 2: 0/1

INFO  : Map 1: 0(+1,-1)/1       Reducer 2: 0/1

INFO  : Map 1: 0(+1,-1)/1       Reducer 2: 0/1

INFO  : Map 1: 0(+1,-2)/1       Reducer 2: 0/1

INFO  : Map 1: 0(+1,-2)/1       Reducer 2: 0/1

INFO  : Map 1: 0(+1,-3)/1       Reducer 2: 0/1

INFO  : Map 1: 0(+1,-3)/1       Reducer 2: 0/1

 

ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1451986680090_0002_1_00, diagnostics=[Task
failed, taskId=task_1451986680090_0002_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error:
Failure while running task:java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

 

 

 

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.


co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This message is for the
designated recipient only, if you are not the intended recipient, you should destroy it immediately.
Any information in this message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility
of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd,
its subsidiaries nor their employees accept any responsibility.

 

From: Rajesh Balamohan [mailto:rajesh.balamohan@gmail.com <mailto:rajesh.balamohan@gmail.com>
] 
Sent: 05 January 2016 00:35
To: user@hive.apache.org <mailto:user@hive.apache.org> 
Subject: Re: Hive on TEZ fails starting

 

By default it should add "LD_LIBRARY_PATH" in the container (ref: https://github.com/apache/tez/blob/abfc8bfb0a8620d31697a31ad516674a8d3f9f7c/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java#L358)
. In case your snappy native libs are present elsewhere in the cluster deployment, you can
override using "tez.task.launch.env" and "tez.am.launch.env" (refer: https://github.com/apache/tez/blob/abfc8bfb0a8620d31697a31ad516674a8d3f9f7c/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java#L412)

 

~Rajesh.B

 

 





 

-- 

~Rajesh.B





 

-- 

~Rajesh.B


Mime
View raw message