ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ihorps <ihor...@gmail.com>
Subject Task management - MapReduce & ForkJoin performance penalty
Date Wed, 30 Aug 2017 13:32:45 GMT
Hi all

[brief overview]
I'm evaluating Apache Ignite framework as a replacement for Hazelcast. One
of usages where it's planned to be compared is task/job processing. We have
implemented tasks management by ourselves based on Hazelcast but not using
their MarReduce framework (such as it was very slow in version 3.5). We
developed a proprietary framework, which is similar to Ignite's
ComputeTaskSplitAdapter and ComputeJobAdapter (failover, task status,
distributed execution, etc) at the end and we would like to simplify it by
switching to Apache Ignite.
To simplify discussion - I've created a trivial project (attached here),
which creates one Task with *N* Jobs in it. Jobs are kind of "dummy" - they
do nothing and return nothing. The goal is to check an overhead of task
management (distribution, failover, etc) overall.

And here are some results:
1. 1 Node  - 1 Task - 5000 Jobs -> ~0.8 sec
2. 2 Nodes - 1 Task - 5000 Jobs -> ~10-11 sec
3. 3 Nodes - 1 Task - 5000 Jobs -> ~15 sec
4. 1 Node  - 1 Task - 10000 Jobs -> ~3 sec
5. 2 Nodes - 1 Task - 10000 Jobs -> ~40 sec

[question]
Are these measurements correct from Ignite point of view? Meaning - I didn't
miss nothing in term of configuration and this is the price which should be
paid for that amount of Jobs to be executed in distributed way (10 sec. for
5000 empty jobs for example). Or it can be tuned/speed-up in some way?

Thank you in advance!

P.S. And I realized that for a run with 1 Task and 5000 "dummy" jobs Ignite
cluster uses almost 3.5Gb heap space.

task-management.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t1316/task-management.zip>  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Mime
View raw message