ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Veselovsky <iveselovs...@gridgain.com>
Subject Re: HDP, Hive + Ignite
Date Tue, 25 Apr 2017 18:50:19 GMT
Hi, Alena, there are several different requests, let's try to separate them.

A1. Wrong Hive query results: 
Is this use case easily reproducible? Now it appears, it is not. Please try
to track it down, as possible: 
- Run Ignite nodes with -v option and see console logs of the nodes: are
there any errors there? If there are any errors, query result in most the
cases is not predictable.
- use IGFS w/o Ignite Map-Reduce, Ignite Map-Reduce w/o IGFS. 
- try to run the query in 1 node instead of several nodes.
- See https://issues.apache.org/jira/browse/IGNITE-4720 , set 

A2. Run on Ignite is twice-slow than on TEZ.
Need more info on Ignite configuration .

What are the JVM options Ignite nodes are running with? Simplest solution is
to give Ignite more memory. 

Please provide full logs of all Ignite nodes.

Further questions:
1. In code you can create bean objects  (like new FileSystemConfiguration()
) , then take the values  via getters. But may be it's simpler to see the
source code.
2. Not quite understand , what do you mean "cluster is HA"? Config setting
"fs.default.name" in global config will prevent HDFS to start, yes, but in
non-global configs you can use igfs:// value.
3. No, currently Ignite does not provide such functionality.
4. Ignite implements generic map-reduce task execution , it will work  with
any input format, whatever configured.
5. This question does not have simple answer: what is done with these data?
Is it searched for something only (like wordcount), or sorted (like in
terasort) ? Also it should be noted that some data are stored onheap, while
some are stored offheap. These 2 types of used memory  are limited and
observed differently.
6. I suppose, 1 Ignite node per host is okay, if the node can utilize all
the host resources (memory & cpu).
7. Yes. Ignite does not give MR task execution correctness guarantees if a
node is stopped or crashed.

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12248.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View raw message