ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yakov Zhdanov <yzhda...@apache.org>
Subject Re: DataStreamer performance question
Date Thu, 14 Jul 2016 10:16:11 GMT
Case of 1 client and 1 server should be about 2 times slower than 2
servers, since in the second case 50% of keys belong to local node and
there is no need to send them over network. In the first case 100% keys and
values go over network.

1. I would think you are running out of memory on server node. Can you
please check you have enough heap during the whole test?
2. Please share logs and threaddumps from server and client nodes.
3. Try disabling shared memory communication
- org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi#setSharedMemoryPort()
set this to -1 in and pass to IgniteConfiguration.setCommunicationSpi()

Thanks!

--Yakov

2016-07-14 11:34 GMT+03:00 November <yufeng7144@126.com>:

> Hi
>
> I start two server node in my machine. And write a program to load
> data(file
> size about 1GB) using DataStreamer. When I run the program in a client
> node.
> It take 40 minutes but just load about 1/5 of data.
> But if I run the program in a server node. It only takes less than 2
> minutes
> to load the data.
>
> Below is my code
>
> var cfg = new IgniteConfiguration
> {
>     CacheConfiguration = new[]
>     {
>
>         new CacheConfiguration("keywordsCache", typeof(string),
> typeof(long))
>         {
>             WriteSynchronizationMode =
> CacheWriteSynchronizationMode.FullAsync,
>             Backups = 0
>         },
>     },
>     //ClientMode = true,  *//Change ClientMode setting significantly affect
> the performance of read data in.*
>     SpringConfigUrl =
>
> @"D:\apache-ignite-fabric-1.6.0-bin\platforms\dotnet\examples\Config\examples-config.xml",
>     BinaryConfiguration = new BinaryConfiguration(typeof(string),
> typeof(long))
> };
>
>
> var keywordsCache = ignite.GetOrCreateCache<string, long>("keywordsCache");
>
> using (var keywordsStreamer = ignite.GetDataStreamer<string,
> long>(keywordsCache.Name))
> {
>     using (StreamReader sr = new StreamReader(keywordsPath))
>     {
>         long lineNum = 0;
>         string line;
>         while ((line = sr.ReadLine()) != null)
>         {
>             string[] strs = line.Split('\t');
>             keywordsStreamer.AddData(Normalization.Stem(strs[3]),
> lineNum++);
>         }
>         keywordsStreamer.Flush();
>     }
> }
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/DataStreamer-performance-question-tp6303.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Mime
View raw message