ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 胡永亮/Bob <hu...@neusoft.com>
Subject Re: Re: Increase Ignite instances can't increase the speed of compute
Date Tue, 13 Sep 2016 09:06:15 GMT

    Only 118 jobs for this test.



Bob
 
From: Taras Ledkov
Date: 2016-09-13 14:52
To: user@ignite.apache.org
Subject: Re: Re: Increase Ignite instances can't increase the speed of compute
Hi,

How many MatchingJobs do you submit?


On Tue, Sep 13, 2016 at 12:29 PM, 胡永亮/Bob <hu-yl@neusoft.com> wrote:
Hello,  Vladislav

    The following is some code. 

    ...
    IgniteCompute compute = ignite.compute();//.withAsync();
        
    compute.run(new MatchingJob(m_mapReadyDataPara));
    }
    
private static class MatchingJob implements IgniteRunnable{

    private Map<String, Map<Long, std>> m_mapReadyData;
    private IgniteCache<String, Map<Long, List<baselink>>> mapMatchingData;
//This is a cache in Ignite cluster.
    ...

    public void run() {
    ...
    Iterator<Entry<String, Map<Long, std>>> entryKeyIterator1 = m_mapReadyData.entrySet().iterator();
   //m_mapReadyData is the input data, its size is 5000 for every job now.
    Map<String, Map<Long, List<baselink>>> local_writeCache = new HashMap<String,
Map<Long, List<baselink>>>();
    ...
    //Then the job read detail data from m_mapReadyData, and compute.
    while (entryKeyIterator1.hasNext()) {
                Entry<String, Map<Long, std>> eKey1 = entryKeyIterator1.next();
                String carKey = eKey1.getKey();
                Map<Long, std> value1 = eKey1.getValue();

                //local node cache
                Map<Long, List<baselink>> existMapbaselink = local_mapMatchingData.get(carKey);
                if(existMapbaselink == null){
                    existMapbaselink = mapMatchingData.get(carKey);  //Read data to compute
with it from Ignite cache. This data's size is 154M for PARTITIONED mode.
                    if(existMapbaselink != null)
                        local_mapMatchingData.putIfAbsent(carKey, existMapbaselink);
                }

                //some compute logic code

                mapbaselink = local_writeCache.get(carKey);
                if(mapbaselink == null){
                    mapbaselink = new TreeMap<Long, List<baselink>>();
                }
                mapbaselink.put(stdtime, ListBaseLink);
                local_writeCache.put(carKey, mapbaselink);
    }

            //batch to write data into Ignite.
            Iterator<Entry<String, Map<Long, List<baselink>>>> it
= local_writeCache.entrySet().iterator();
            while(it.hasNext()){
                Entry<String, Map<Long, List<baselink>>> entry = it.next();
                String carKey = entry.getKey();
                final Map<Long, List<baselink>> value = entry.getValue();
                
                if(!mapMatchingData.containsKey(carKey)){
                    mapMatchingData.put(carKey, value);
                }else{
                    mapMatchingData.invoke(carKey, new EntryProcessor<String, Map<Long,
List<baselink>>, Void>() {
                      @Override 
                      public Void process(MutableEntry<String, Map<Long, List<baselink>>>
entry, Object... args) {
                          Map<Long, List<baselink>> map = entry.getValue();
                          map.putAll(value);
                          entry.setValue(map);
                          return null;
                      }
                  });
                }

    }




bob
 
From: Vladislav Pyatkov
Date: 2016-09-12 18:37
To: user@ignite.apache.org
Subject: Re: Increase Ignite instances can't increase the speed of compute
Hello,

I don't understand, what do you try to measure, without code.
Size of calculation task, size of data moved into network have importance.

Could you please provide code example?

On Mon, Sep 12, 2016 at 12:33 PM, 胡永亮/Bob <hu-yl@neusoft.com> wrote:
Hi, everyone:

    I am using Ignite for computing and cache.

    I use the same input data and the same compute logic.
    When my ignite cluster's node is 2 in 2 machines, the total cost time is 38s.

    But, when I increase the Ignite cluster nodes to 3 in 3 machines, the cost time is 32s/51s/41s
    4 instances in 4 machines, the cost time is 32s/40s.

     The compute speed can't change faster, what may the reason be?

    Thanks.

Bob

---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)

is intended only for the use of the intended recipient and may be confidential and/or privileged
of 
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication
is 
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or
copying 
is strictly prohibited, and may be unlawful.If you have received this communication in error,please

immediately notify the sender by return e-mail, and delete the original message and all copies
from 
your system. Thank you. 
---------------------------------------------------------------------------------------------------



-- 
Vladislav Pyatkov

---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)

is intended only for the use of the intended recipient and may be confidential and/or privileged
of 
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication
is 
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or
copying 
is strictly prohibited, and may be unlawful.If you have received this communication in error,please

immediately notify the sender by return e-mail, and delete the original message and all copies
from 
your system. Thank you. 
---------------------------------------------------------------------------------------------------



---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged
of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication
is
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or
copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies
from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Mime
View raw message