ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Veselovsky <iveselovs...@gridgain.com>
Subject Re: HDP, Hive + Ignite
Date Fri, 28 Apr 2017 16:18:28 GMT
1. The observed "java.lang.ClassNotFoundException: Class
org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem not found " suggests
that TEZ does not have IgniteHadoopFileSystem in class path. Please check
how TEZ composes the  classpath and if it adds all the libs from 
/usr/hdp/2.6.0.3-8/hadoop/lib .

2. Can you please provide full logs: all Ignite logs and the client console
log.

3. Please make sure option --hiveconf ignite.job.shared.classloader=false 
really takes effect: e.g. execute "set ignite.job.shared.classloader" query
on Hive: what does it print? 
This is very important to have this option to be set to false to ensure
correct Hive execution , otherwise very bad consequences like
https://issues.apache.org/jira/browse/IGNITE-4720 and
https://issues.apache.org/jira/browse/IGNITE-5044 are possible. 

The link you're referring is about execution Hive tasks over TEZ, not over
Ignite-MR, so I have some doubt the "vectorized execution" property advice
is relevant there. Also note that the advice mentions 2 properties: "set
hive.vectorized.execution.enabled=false;
set hive.vectorized.execution.reduce.enabled=false; "

About OutOfMem: please see Ignite logs (run ignite.sh with with "-v") -- how
memory is utilized on Ignite nodes? What error happens first? 
In Java8 the metaspace size is configured with
"-XX:MaxMetaspaceSize=<metaspace size>[g|m|k] " option. 

Also please note that "hive.rpc.query.plan=true" property is *required* to
execute Hive queries on Ignite MR successfully.




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12307.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message