ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mimmo Celano <mimmo.celano1...@gmail.com>
Subject Re: Ignite Information
Date Tue, 18 Jul 2017 12:15:24 GMT
Hi Val,

Thanks for reply.

We are trying to eliminate the reduce step in hadoop processing by sending
each map results in a distribuited cache for minimize time of execution. Do
you think it's a reasonable thing? When we send data with DataStream
overwrite true through more parallel DataStreamer the time increases a lot.
So we would like to split Ignite cluster for manage a set of HashMap, (
Node1 word from "a" to "d", Node2 word from "e" to "f" .... ). The HashMap
are divided efficiently in map job but we aren't able to send this Map to
specific node.
Could you advise us on how to send data to node efficiently?

For an HashMap of 10 millions entries
   sizenodebuffer 8192  16 sec    ----> 1 Client send to 1 Node Server   ||
Overwrite true
   sizenodebuffer 8192  42 sec    ----> 3 Client send at the same time to 1
Node Server   || Overwrite true

   sizenodebuffer 8192  12 sec    ----> 1 Client send to 1 Node Server ||
Overwrite false
   sizenodebuffer 8192  23 sec    ----> 3 Client send at the same time to 1
Node Server || Overwrite false

Problem is thah we need to send GB of data, and this approach is too slow.

Thanks
Mimmo


2017-07-17 20:14 GMT+02:00 vkulichenko <valentin.kulichenko@gmail.com>:

> I can't reproduce this behavior. Can you create a small project that I can
> run on my side and investigate?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-Information-tp14330p15025.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Mime
View raw message