ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Cherkasov <mcherka...@gridgain.com>
Subject Re: Ignite DataStream Troughput
Date Fri, 14 Jul 2017 15:36:09 GMT
Hi Mimmo,

I think you should play with the following streamer parameters:

 public int perNodeBufferSize()
 public void perNodeParallelOperations(int parallelOps)

dataStream has a buffer per Node, perNodeBufferSize defines how many items
should be
saved in buffer before send it to server node.

 perNodeParallelOperations defines how many buffers can be sent to the node
without acknowledge
that it was processed, it's recommended to set it to server cpu count * 2.

Thanks,
Mikhail.

On Fri, Jul 14, 2017 at 6:05 PM, mimmo_c <mimmo.celano1993@gmail.com> wrote:

> Hi,
>
> We're testing ignite DataStream troughput with a network of 1Gbps. If we
> send a big file with scp we have a throughput of 150 Mb/s.
> We have 1 Ignite Server and 1 Ignite Client on version 2.0, in Client we
> have an HashMap of 1Million Entry which we send into Server's cache. The
> Cache is configured as Partitioned, Atomic, no backup. With a simple
> DataStream when we send the HashMap into the cache we have a throughput of
> 20/25 Mb/s without Stream Transformer. Is this a limit of DataStream?
>
>
> Here is the code:
>
>   try(IgniteDataStreamer<String, Long> stmr =
> ignite.dataStreamer("default")){
>
>       for(Entry<String, Long> entry : mappa1.entrySet()){
>                 stmr.addData(entry.getKey(),entry.getValue());
>         }
>
>   }
>
>
> When we add another Server Node the each Server Node receive 11 Mb/s
> (effective throughput is the same 20/25 Mb/s)
>
> If we Add a StreamTransformer for update value in cache and send more than
> 1
> Hashmap the throughput is 15 Mb/s and if we add another Server Node the
> throughput is the same.
>
>
> public static class UpdateWord extends StreamTransformer<String, Long> {
>
>         @Override
>         public Object process(MutableEntry<String, Long> e, Object... arg)
> throws EntryProcessorException {
>                 Object streamed = arg[0];
>                 Long temp = Long.valueOf(streamed.toString()).longValue();
>                 Long val = e.getValue();
>                 e.setValue(val == null ? temp : val + temp);
>             return null;
>         }
>     }
>
>
> Is there some limit on DataStream Usage or a bottleneck? How can we  use
> all
> network bandwidth?
> Maybe some DataStream Configuration?
>
> Thanks,
> Mimmo
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Ignite-DataStream-Troughput-tp14918.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Thanks,
Mikhail.

Mime
View raw message