ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siew Wai Yow <wai_...@hotmail.com>
Subject Re: Ignite singel instance
Date Fri, 08 Nov 2019 10:20:43 GMT
Hi,

Ignite server hotspot during the run, seems like there are locks in Ignite consume most of
the time, any idea?

[cid:10ec4a91-63e7-4b5c-9c7d-63f61ec3407b]

Backtrace,
[cid:5ff082b4-745f-4867-b829-7cc8f9d79c80]


[cid:472a9a3a-6bdd-4964-afde-b148c65c1f8e]



threadump(please refer to attachment for complete dump),

"client-connector-#157" - Thread t@194
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <46ac352c> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None


[cid:40257cad-10c5-49cb-9c39-b1726bb8e2a1]

Thanks!

Regards,
Yow




________________________________
From: Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
Sent: Friday, November 8, 2019 5:21 PM
To: user@ignite.apache.org <user@ignite.apache.org>
Subject: Re: Ignite singel instance

Hello!

I was meaning that you should benchmark the Java side (Ignite server) and not ODBC client.

What you should be figuring out is which thread pools are busy and what's they up to.

Regards,
--
Ilya Kasnacheev


пт, 8 нояб. 2019 г. в 11:36, Siew Wai Yow <wai_yow@hotmail.com<mailto:wai_yow@hotmail.com>>:
Hi Ivan,

5k TPS the minimum and it would be great to have 10k TPS. Do you think this is feasible? Or
Ignite expect user to run with multiple instances?

At the moment only 1.7k TPS in local server without persistence. Same test in a 3-nodes-cluster
with persistence enabled, the same application run only 400 TPS which is quite disappointing.
Any tips? Could it be ODBC driver slowness?

Regards,
Yow

________________________________
From: Ivan Pavlukhin <vololo100@gmail.com<mailto:vololo100@gmail.com>>
Sent: Friday, November 8, 2019 4:18 PM
To: user@ignite.apache.org<mailto:user@ignite.apache.org> <user@ignite.apache.org<mailto:user@ignite.apache.org>>
Subject: Re: Ignite singel instance

Hi Yow,

Out of curiosity. What TPS will be good for your cases?

пт, 8 нояб. 2019 г. в 03:34, Siew Wai Yow <wai_yow@hotmail.com<mailto:wai_yow@hotmail.com>>:
Hi!

NetworkIO is just around 1.2MB/s as shown so that is not a problem. We have profiled and did
not found any bottleneck from the environment/resources. We also check the application profiling
and most of the time consumed by sql execution which looks normal to me. Any advice?

[cid:16e4a510dfccb971f164]
________________________________
From: Ilya Kasnacheev <ilya.kasnacheev@gmail.com<mailto:ilya.kasnacheev@gmail.com>>
Sent: Thursday, November 7, 2019 11:04 PM
To: user@ignite.apache.org<mailto:user@ignite.apache.org> <user@ignite.apache.org<mailto:user@ignite.apache.org>>
Subject: Re: Ignite singel instance

Hello!

Is it possible that you ended up saturating the network? Please also consider profiling for
bottlenecks as I have suggested.

Regards,
--
Ilya Kasnacheev


чт, 7 нояб. 2019 г. в 17:55, Siew Wai Yow <wai_yow@hotmail.com<mailto:wai_yow@hotmail.com>>:
Hi llya!

CPU usage is very low. I just did another run as shown, but IO is happening in a spiky pattern,
any clue?
RAM and disk is fine. Only Ignite is running in the host. Thanks.


[cid:16e4a510dfccb971f161]

[cid:16e4a510dfccb971f162]
[cid:16e4a510dfccb971f163]

Regards,
Yow

________________________________
From: Ilya Kasnacheev <ilya.kasnacheev@gmail.com<mailto:ilya.kasnacheev@gmail.com>>
Sent: Thursday, November 7, 2019 10:04 PM
To: user@ignite.apache.org<mailto:user@ignite.apache.org> <user@ignite.apache.org<mailto:user@ignite.apache.org>>
Subject: Re: Ignite singel instance

Hello!

Have you tried collecting thread dumps from node under load, to identify any bottlenecks?

What is the CPU usage by Ignite?

Regards,
--
Ilya Kasnacheev


ср, 6 нояб. 2019 г. в 09:08, Siew Wai Yow <wai_yow@hotmail.com<mailto:wai_yow@hotmail.com>>:
Hi Stan

The result will be the same without transaction, even with atomic mode. So wondering if there
is other reason capped the performance. Attach is our POC source code which give around 1700
TPS, with or without transaction, may be you can advise what's wrong with it.

-DIGNITE_MAX_INDEX_PAYLOAD_SIZE=64 is using already.

We choose ODBC because it is more compatible to existing product's API, which will need to
create table/cache on the fly without pre-define C-object. Ignite C++ need to have the cache
C-object ready before run.

Also in Ignite cluster, how can we connect to cluster, just connect to one of the host? And
when host crash we need to change manually connect to another node? Seems ODBC connection
string not allow to input multiple host address.

Thanks in advance.

Regards,
Yow


________________________________
From: Stanislav Lukyanov <stanlukyanov@gmail.com<mailto:stanlukyanov@gmail.com>>
Sent: Wednesday, November 6, 2019 12:33 AM
To: user@ignite.apache.org<mailto:user@ignite.apache.org> <user@ignite.apache.org<mailto:user@ignite.apache.org>>
Subject: Re: Ignite singel instance

First, 1700 TPS given your transaction structure is 170000 simple operations per second, which
is quite substantial - especially if you're doing that from a single thread / single ODBC
client.

Second, note that TRANSACTIONAL_SNAPSHOT is in beta and is not ready for production use. There
are no claims about performance or stability of that cache mode.

Third, you don't need the index on key01 - it is created automatically because it is a primary
key. But you do need to set INLINE SIZE of the default index. Run your Ignite server with
system property or environment varialy IGNITE_MAX_INDEX_PAYLOAD_SIZE=64.

Finally, the operations you're doing don't look like something to be done with SQL. Consider
using key-value API in Ignite C++ instead - https://apacheignite-cpp.readme.io/docs.

Stan

On Sat, Nov 2, 2019 at 8:15 PM Evgenii Zhuravlev <e.zhuravlev.wk@gmail.com<mailto:e.zhuravlev.wk@gmail.com>>
wrote:
Hi,

Do you use only one ODBC client? Can you start one more ODBC client and check the performance?

Thanks,
Evgenii

сб, 2 нояб. 2019 г. в 16:47, Siew Wai Yow <wai_yow@hotmail.com<mailto:wai_yow@hotmail.com>>:
Hi,

We are doing POC on ignite performance using ODBC driver but the performance is always capped
at around 1700 TPS which is too slow. It is local ignite service. All tuning tips from Ignite
page has been applied, no bottleneck from CPU and Memory. At the moment not turn on persistence
yet, it will be worse if turn on. This POC is very crucial to our product roadmap.

Any tips? Thank you.

Test case,
1 x insert --> 49 x select -->49 x update --> 1 x delete
repeat for 50000 times.

    const char* insert_sql = "insert into CDRTEST values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
    const char* update_sql = "update CDRTEST set value01=?,value02=?,value03=?,value04=?,value05=?,value21=?,value22=?,value23=?,value24=?,value25=?
where key01=?";
    const char* delete_sql = "delete from CDRTEST where key01=?";
    const char* select_sql = "select value01 from CDRTEST where key01=?";

    retcode = SQLExecDirect(hstmt, reinterpret_cast<SQLCHAR*>(const_cast<char*>("CREATE
TABLE IF NOT EXISTS CDRTEST ( "
  "key01 VARCHAR PRIMARY KEY, "
  "value01 LONG, "
  "value02 LONG, "
  "value03 LONG, "
  "value04 LONG, "
  "value05 LONG, "
  "value06 LONG, "
  "value07 LONG, "
  "value08 LONG, "
  "value09 LONG, "
  "value10 LONG, "
  "value11 LONG, "
  "value12 LONG, "
  "value13 LONG, "
  "value14 LONG, "
  "value15 LONG, "
  "value16 LONG, "
  "value17 LONG, "
  "value18 LONG, "
  "value19 LONG, "
  "value20 LONG, "
  "value21 VARCHAR, "
  "value22 VARCHAR, "
  "value23 VARCHAR, "
  "value24 VARCHAR, "
  "value25 VARCHAR, "
  "value26 VARCHAR, "
  "value27 VARCHAR, "
  "value28 VARCHAR, "
  "value29 VARCHAR, "
  "value30 VARCHAR, "
  "value31 VARCHAR, "
  "value32 VARCHAR, "
  "value33 VARCHAR, "
  "value34 VARCHAR, "
  "value35 VARCHAR, "
  "value36 VARCHAR, "
  "value37 VARCHAR, "
  "value38 VARCHAR, "
  "value39 VARCHAR) "
  "WITH \"template=partitioned,atomicity=TRANSACTIONAL_SNAPSHOT,WRITE_SYNCHRONIZATION_MODE=FULL_ASYNC\"")),
SQL_NTS);
            CHECK_ERROR(retcode, "Fail to create table", hstmt, SQL_HANDLE_STMT);

    retcode = SQLExecDirect(hstmt, reinterpret_cast<SQLCHAR*>(const_cast<char*>("CREATE
INDEX key01t_idx ON CDRTEST(key01) INLINE_SIZE 64")), SQL_NTS);
            CHECK_ERROR(retcode, "Fail to create index", hstmt, SQL_HANDLE_STMT);


Below are configuration we used,

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">

    <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="binaryConfiguration">
            <bean class="org.apache.ignite.configuration.BinaryConfiguration">
                <property name="compactFooter" value="false"/>

                <property name="idMapper">
                    <bean class="org.apache.ignite.binary.BinaryBasicIdMapper">
                        <property name="lowerCase" value="true"/>
                    </bean>
                </property>
            </bean>
        </property>

  <property name="clientConnectorConfiguration">
    <bean class="org.apache.ignite.configuration.ClientConnectorConfiguration">
      <!--property name="host" value="127.0.0.1"/-->
      <!--property name="port" value="10800"/-->
      <!--property name="portRange" value="5"/-->
      <property name="maxOpenCursorsPerConnection" value="512"/>
      <property name="socketSendBufferSize" value="65536"/>
      <property name="socketReceiveBufferSize" value="131072"/>
      <!--property name="threadPoolSize" value="4"/-->
    </bean>
  </property>

        <property name="dataStorageConfiguration">
           <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
              <!-- Enabling Apache Ignite Persistent Store. -->
              <property name="defaultDataRegionConfiguration">
                 <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                     <property name="persistenceEnabled" value="false"/>
                 </bean>
              </property>
              <property name="walPath" value="/ignite-wal"/>

              <property name="walArchivePath" value="/ignite-wal"/>
           </bean>
        </property>

    </bean>
</beans>

Regards,
Yow


--
Best regards,
Ivan Pavlukhin

Mime
View raw message