ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir St." <vlads...@yandex.ru>
Subject Distributed query is too slow.
Date Tue, 31 Jan 2017 16:09:04 GMT
Hello, everyone!
Can I get a piece of help for my querying trouble? I’m measuring performance
of basic SQL-like queries over a cache with various settings and
deployments. Also I compare the results with Oracle11g which is running on
the same hardware. In general, I need something significantly faster than
this DB. I see that local queries and queries on replicated caches are just
perfect. They overtake the DB x5 times. But a problem occurs when I execute
search on a replicated cache at each additional node in the cluster. The
problem is that my search query gets slower sooo much if the cluster has
more that one node :(. This difference in search speed is completely
unreasonable. For example, if the test is executed on single node, it takes
approx. 15 seconds to find, approx. 55 sec. (!) on two local nodes, approx.
65 sec. on three local nodes and approx. 80sec. on 4 nodes :(.The database
does an equal search for about 75 seconds. There are some cores/gc/memory
interferences if I launch more than two nodes on the same machine, but I
take it into account. Sadly, even a bigger lack of performance appears when
I launch my tests on separated machines. Same test might consume x20 slower
time compared to local nodes. I don’t understand what is going on.

The hardware is just an average ones: i5-4570 3.2Ghz, 4 cores, 16Gb, Win7
x64 on Linux x64. The JVM is HotSpot Version of Ignite is 1.8.
The network exhibits 17Mb/sec. real throughput.

*The entity is:*

@Indexed(index = "testRecord")
public class Record implements Serializable {

    @QuerySqlField(index = true, orderedGroups = {@QuerySqlField.Group(name
= "g1", order = 0)})
    private String str;

    @QuerySqlField(index = true, orderedGroups = {@QuerySqlField.Group(name
= "g1", order = 1)})
    private String str2;

    @QuerySqlField(index = true, orderedGroups = {@QuerySqlField.Group(name
= "g1", order = 2)})
    private long l;

    private byte[] bArr;
    private int i;
    private Double d;
    private BigInteger bi;

    public static Record generate() {
        …//Fills fields with random values

I search for 3 indexed fields: 2 strings and one long. 

*The test routine is pretty simple:*

-	A node starts (first node). Only one node at same time.
-	Node acquires the cache and fills it if empty for 100-500k records
depending on the setting. Only first node fills the cache. The key is a
-	Node preloads all the records from the cache to search for in case of
random fields were generated. Or builds an array of known fields to search
for in case of sequential record fields were generated. 
-	Node runs through this search set several times querying for all the
records one by one. Usually I run this cycle 5 times to make sure all is ok
with the GC and the warming.
-	The measurement begins right before each run (after data preloading) and
ends right after. Average search time is observed.
-	Next node starts
-	Cache rebalancing is performed. The cache is acquired on the next node. No
additional data is put into the cache.
-	New node starts and does the same searching.

I.e. if 300k records were generated, 300k searches are launched over the
cache of 300k records.

*The searching core is:*

new SqlQuery<>(Record.class, “str=? and str2=? and l=?”);

IgniteCache<String, Record> cache = …;

List<Record> dataToFind = …;
long cycleTime, avgTime=0;

for( int attempt=0; attempt<cycles;++attempt ){
   cycleTime = System.currentTimeMillis();

   dataToFind.forEach( r-> {
	List<Entry&lt;String, Record>>> lst = cache.query(query.setArgs(r.getStr(),
r.getStr2(),       r.getL()).getAll()

   avgTime += System.currentTimeMillis() - cycleTime;


avgTime /= cycles;

First node always goes well, much faster than the DB. But next ones don’t.
My working machine does the 100k-on-100k-records-search for avg. 2 seconds.
A second node running on equal machine in the network consumes about 45-50
seconds!!! Whhhyyy? The database takes about 35 second to search the table
which has similar structure and indexes.  I mean searching via JDBC

*The cache settings are:*

setIndexedTypes(String.class, Record.class);
//setOffHeapMaxMemory(maxOffheapMegabytes * 1024L * 1024L);

I’ve tried using setting like: atomic mode, 0 backups, async rebalancing,
asyn write mode, off-heap features and so on. I tested non-grouped indexes,
SqlFieldsQuery, searching for only one field. Nothing has helped.

*The explain-plan shows:*

    RECORD._KEY AS __C0,
    RECORD._VAL AS __C1,
    RECORD.STR2 AS __C3,
    RECORD.L AS __C4
FROM "testCache".RECORD
    /* "testCache"."g1": L = ?3
        AND STR = ?1
        AND STR2 = ?2
WHERE (L = ?3)
    AND ((STR = ?1)
    AND (STR2 = ?2))], [SELECT
    __C0 AS _KEY,
    __C1 AS _VAL,
    __C2 AS STR,
    __C3 AS STR2,
    __C4 AS L
    /* "testCache"."merge_scan" */]]

As mentioned in the beginning, local searches and searching replicated cache
work well. So I decided to implement basic manual map-reduce of splited
local searches in the way as it's said in the documentation. I broadcasted
an IgniteCallable which launched a local query ( Query.setLocal(true) ).
After, the results were collected by the id. It’d worked perfectly, x4.5-5
time faster that the DB! 
Of course, I don’t say it’s a perfect way to search. I guess each callable
works in its own transaction. I wouldn’t imagine what happes if the data is
being changed or if the topology is changing at the search time. I just
tried to find out the problem seed.

So, what’s wrong with the distributed query? Or what I’m doing wrong? What
do I need to ckeck? The profiler says time is spent somewhere inside
H2/indexes just like in case of successfull local searches.

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-query-is-too-slow-tp10342.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View raw message