ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Piubelli, Manuel " <manuel.piube...@citi.com>
Subject Ignite performance
Date Wed, 03 Aug 2016 09:49:05 GMT
Hello,
I am currently benchmarking Apache Ignite for a near real-time application and simple operations
seem to be excessively slow for a relatively small sample size. The following is giving the
setup details and timings - please see 2 questions at the bottom.
Setup:
*         Cache mode: Partitioned
*         Number of server nodes: 3
*         CPUs: 4 per node (12)
*         Heap size: 2GB per node (6GB)
The first use case is computing the weighted average over two fields of the object at different
rates.
First method is to run a SQL style query:
...
query = new SqlFieldsQuery("select SUM(field1*field2)/SUM(field2) from MyObject");
cache.query(query).getAll();
....
The observed timings are:
Cache: 500,000 Queries/second: 10
Median: 428ms, 90th percentile: 13,929ms
Cache: 500,000 Queries/second: 50
Median: 191,465ms, 90th percentile: 402,285ms
Clearly this is queuing up with an enormous latency (>400 ms), a simple weighted average
computation on a single jvm (4 Cores) takes 6 ms.
The second approach is to use the IgniteCompute to broadcast Callables across nodes and compute
the weighted average on each node, reducing at the caller, latency is only marginally better,
throughput improves but still at unusable levels.
Cache: 500,000 Queries/second: 10
Median: 408ms, 90th percentile: 507ms
Cache: 500,000 Queries/second: 50
Median: 114,155ms, 90th percentile: 237,521ms
A few things i noticed during the experiment:
*         No disk swapping is happening
*         CPUs run at up to 400%
*         Query is split up in two different weighted averages (map reduce)
*         Entries are evenly split across the nodes
*         No garbage collections are triggered with each heap size around 500MB
To my questions:
1.    Are these timings expected or is there some obvious setting i am missing? I could not
find benchmarks on similar operations.
2.    What is the advised method to run fork-join style computations on ignite without moving
data?

Thank you

Manuel

Mime
View raw message