ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yakov Zhdanov <yzhda...@apache.org>
Subject Re: Ignite performance
Date Wed, 03 Aug 2016 12:37:23 GMT
Manuel,

The numbers you are seeing are pretty strange to me.

Is it correct that you run your query in a loop giving enough time for the
whole cluster to warmup and only then take the final measurements?
I also do not understand why CPU load is 400% which may be interpreted as
full (correct?). This means that at least 4 threads are busy on each node,
but when you broadcast your query it is processed with only 1 thread on
each node. Having this in mind you can try launching 1 node per 1 core on
each server - this will split your data set and will lower the amount of
work for each node. However question with high CPU utilization is still
open. Can you please provide stack for those threads if they are Ignite
threads. You can follow these instructions -
https://blogs.oracle.com/jiechen/entry/analysis_against_jvm_thread_dump

Please tell me what machines you are running this test on. I would ask you
to do all measurements on hardware machines (not virtual) giving all
resources to Ignite.

Please also share your code and configuration for cluster nodes.

--Yakov

2016-08-03 12:49 GMT+03:00 Piubelli, Manuel <manuel.piubelli@citi.com>:

> Hello,
>
> I am currently benchmarking Apache Ignite for a near real-time application
> and simple operations seem to be excessively slow for a relatively small
> sample size. *The following is giving the setup details and timings -
> please see 2 questions at the bottom.*
>
> Setup:
>
> ·         Cache mode: Partitioned
>
> ·         Number of server nodes: 3
>
> ·         CPUs: 4 per node (12)
>
> ·         Heap size: 2GB per node (6GB)
>
> The first use case is computing the weighted average over two fields of
> the object at different rates.
>
> First method is to run a SQL style query:
>
> ...
>
> query = new SqlFieldsQuery("select SUM(field1*field2)/SUM(field2) from
> MyObject");
>
> cache.query(query).getAll();
>
> ....
>
> The observed timings are:
>
> Cache: 500,000 Queries/second: 10
> Median: 428ms, 90th percentile: 13,929ms
>
> Cache: 500,000 Queries/second: 50
> Median: 191,465ms, 90th percentile: 402,285ms
>
> Clearly this is queuing up with an enormous latency (>400 ms), a simple
> weighted average computation on a single jvm (4 Cores) takes 6 ms.
>
> The second approach is to use the IgniteCompute to broadcast Callables
> across nodes and compute the weighted average on each node, reducing at the
> caller, latency is only marginally better, throughput improves but still at
> unusable levels.
>
> Cache: 500,000 Queries/second: 10
> Median: 408ms, 90th percentile: 507ms
>
> Cache: 500,000 Queries/second: 50
> Median: 114,155ms, 90th percentile: 237,521ms
>
> A few things i noticed during the experiment:
>
> ·         No disk swapping is happening
>
> ·         CPUs run at up to 400%
>
> ·         Query is split up in two different weighted averages (map
> reduce)
>
> ·         Entries are evenly split across the nodes
>
> ·         No garbage collections are triggered with each heap size around
> 500MB
>
> To my questions:
>
> 1.    *Are these timings expected or is there some obvious setting i am
> missing? I could not find benchmarks on similar operations.*
>
> 2.    *What is the advised method to run fork-join style computations on
> ignite without moving data?*
>
> Thank you
>
> Manuel
>

Mime
View raw message