ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ihorps <ihor...@gmail.com>
Subject Re: Task management - MapReduce & ForkJoin performance penalty
Date Mon, 11 Sep 2017 11:29:20 GMT
hi @yakov

yakov wrote
> Yes, however, you can still return results from each job and use it.
> Please
> see javadoc for org.apache.ignite.compute.ComputeJobResult#getData

yes, it's good to have such opportunity at least on "result" step.
But still I'm very curious, why the overhead is so big when results are
being collected on reduce step.

I've done a quick profile action (without deep analysis) and it seems to me
that with "cached results" for MapReduce process there is a big overhead in
org.jsr166.* package. Meaning there are quite a lot of invocation of
"get/poll" and "put" methods from concurrent impl. of Map
(ConcurentHashMap8, ConcurentLinkedHashMap, ConcurentLinkedDeque8) - 
here is ConcurrentLinkedDeque8.poll exceptional method call

Will try to dig into it more at my spare time...

Sent from: http://apache-ignite-users.70518.x6.nabble.com/

View raw message