ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Cherkasov <michael.cherka...@gmail.com>
Subject Re: Ignite-cassandra module issue
Date Thu, 09 Nov 2017 17:47:19 GMT
Hi Tobias,

I created a ticket to fix the issue:
https://issues.apache.org/jira/browse/IGNITE-6853

2017-10-25 16:23 GMT+03:00 Tobias Eriksson <tobias.eriksson@qvantel.com>:

> Hi Andrey et al
>
> I believe I now know what the problem is, the Cassandra session is
> refreshed, but before it is a prepared statement is created/used and there,
> and so using a new session with an old prepared statement is not working.
>
>
>
> The way to reproduce is
>
> 1)       Start Ignite Server Node
>
> 2)       Start client which inserts a batch of 100 elements
>
> 3)       End client
>
> 4)       Now Ignite Server Node returns the Cassandra Session to the pool
>
> 5)       Wait 5+ minutes
>
> 6)       Now Ignite Server Node has does a clean-up of the “unused”
> Cassandra sessions
>
> 7)       Start client which inserts a batch of 100 elements
>
> 8)       Boom ! The exception starts to happen
>
>
>
> Reason is
>
> 1)       Execute is called for a BATCH
>
> 2)       Prepared-statement is reused since there is a cache of those
>
> 3)       It is about to do session().execute( batch )
>
> 4)       BUT the call to session() results in refreshing the session, and
> this is where the prepared statements as the old session new them are
> cleaned up
>
> 5)       Now it is looping over 100 times with a NEW session but with an
> OLD prepared statement
>
>
>
> This is a bug,
>
>
>
> -Tobias
>
>
>
>
>
> *From: *Andrey Mashenkov <andrey.mashenkov@gmail.com>
> *Reply-To: *"user@ignite.apache.org" <user@ignite.apache.org>
> *Date: *Wednesday, 25 October 2017 at 14:12
> *To: *"user@ignite.apache.org" <user@ignite.apache.org>
> *Subject: *Re: Ignite-cassandra module issue
>
>
>
> Hi Tobias,
>
>
>
> What ignite version do you use? May be this was already fixed in latest
> one?
>
> I see related fix inclueded in upcoming 2.3 version.
>
>
>
> See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end
> logic, so session should be closed in proper way.
>
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5897
>
>
>
>
>
> On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson <
> tobias.eriksson@qvantel.com> wrote:
>
> Hi
>  Sorry did not include the context when I replied
>  Has anyone been able to resolve this problem, cause I have it too on and
> off
> In fact it sometimes happens just like that, e.g. I have been running my
> Ignite client and then stop it, and then it takes a while and run it again,
> and all by a sudden this error shows up. An that is the first thing that
> happens, and there is NOT a massive amount of load on Cassandra at that
> time. But I have also seen it when I hammer Ignite/Cassandra with
> updates/inserts.
>
> This is a deal-breaker for me, I need to understand how to fix this, cause
> having this in production is not an option.
>
> -Tobias
>
>
> Hi!
> I'm using the cassandra as persistence store for my caches and have one
> issue by handling a huge data (via IgniteDataStreamer from kafka).
> Ignite Configuration:
> final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
>         igniteConfiguration.setIgniteInstanceName("test");
>         igniteConfiguration.setClientMode(true);
>         igniteConfiguration.setGridLogger(new Slf4jLogger());
>         igniteConfiguration.setMetricsLogFrequency(0);
>         igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
> final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
>         binaryConfiguration.setCompactFooter(false);
>         igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
>         igniteConfiguration.setPeerClassLoadingEnabled(true);
> final MemoryPolicyConfiguration memoryPolicyConfiguration = new
> MemoryPolicyConfiguration();
>         memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
>         memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
>         memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);
>
> memoryPolicyConfiguration.setPageEvictionMode(
> DataPageEvictionMode.RANDOM_2_LRU);
> final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
>         memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
>         igniteConfiguration.setMemoryConfiguration(memoryConfiguration);
>
> Cache configuration:
> final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
> CacheConfiguration<>();
>         cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>         cacheConfiguration.setStoreKeepBinary(true);
>         cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
>         cacheConfiguration.setBackups(0);
>         cacheConfiguration.setStatisticsEnabled(false);
>         cacheConfiguration.setName("TestCache");
>
>         cacheConfiguration.setReadThrough(true);
>         cacheConfiguration.setWriteThrough(true);
>
>         cacheConfiguration.setWriteBehindEnabled(true);
>         cacheConfiguration.setWriteBehindFlushFrequency(10000);
>         cacheConfiguration.setWriteBehindFlushSize(0);
>         cacheConfiguration.setWriteBehindFlushThreadCount(2);
>         cacheConfiguration.setWriteBehindBatchSize(10000);
>
>
>         final CassandraCacheStoreFactory<String, BinaryObject>
> cacheStoreFactory = new CassandraCacheStoreFactory<>();
>         final DataSource dataSource = new DataSource();
>         dataSource.setContactPoints(contactPoints);
>         dataSource.setReadConsistency("ONE");
>         dataSource.setWriteConsistency("ONE");
>         dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
> RoundRobinPolicy()));
>         cacheStoreFactory.setDataSource(dataSource);
>
>         final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
> table=\"%s\">" +
>                 "<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
>                 "<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
>                 "</persistence>";
>         final KeyValuePersistenceSettings settings = new
> KeyValuePersistenceSettings(
>                 String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
>                         "java.lang.String", "PRIMITIVE",
>                         "org.apache.ignite.binary.BinaryObject", "BLOB"));
>         cacheStoreFactory.setPersistenceSettings(settings);
>         cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);
>
> When application works some time (hour or more, may be less - from time to
> time) I see this exceptions on the ignite nodes:
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>     ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
>     ^-- H/N/C [hosts=3, nodes=3, CPUs=96]
>     ^-- CPU [cur=0%, avg=1.86%, GC=0%]
>     ^-- PageMemory [pages=118064]
>     ^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
>     ^-- Non heap [used=78MB, free=-1%, comm=80MB]
>     ^-- Public thread pool [active=0, idle=32, qSize=0]
>     ^-- System thread pool [active=0, idle=32, qSize=0]
>     ^-- Outbound messages queue [size=0]
> [15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
> [name=null, buckets=256, dataPages=102080, reusePages=0]
> [15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
> statement cluster error detected, refreshing Cassandra session
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
> session refreshed
> [15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for
> 332ms
> [15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for
> 332ms
> [15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for
> 229ms
> [15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for
> 229ms
> [15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for
> 229ms
> [15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for
> 229ms
> [15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for
> 229ms
> [15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for
> 229ms
> [15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for
> 229ms
> [15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
> statement cluster error detected, refreshing Cassandra session
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> [15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
> [15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
> [15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
> [15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
> [15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
> [15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
> [15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
> attempt to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
> [15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
> attempt to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
> attempt to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
> attempt to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
> attempt to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2
> attempt
> to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
> attempt to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
> [15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
> [15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
> attempt to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
> attempt to execute Cassandra CQL statement: select "value" from
> "triggers"."callback" where "key"=?;
> [15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
> session refreshed
> [15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for
> 229ms
> [15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
> statement cluster error detected, another thread already fixed the problem
> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to
> execute
> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
> used a PreparedStatement that was created with another Cluster instance.
>         at
> com.datastax.driver.core.SessionManager.makeRequestMessage(
> SessionManager.java:568)
>         at
> com.datastax.driver.core.SessionManager.executeAsync(
> SessionManager.java:131)
>         at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
>         at
> org.apache.ignite.cache.store.cassandra.session.
> CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
>         at
> org.apache.ignite.cache.store.cassandra.CassandraCacheStore.
> load(CassandraCacheStore.java:168)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
>         at
> org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.
> load(CacheStoreBalancingWrapper.java:98)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.
> java:316)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAllFromStore(
> GridCacheStoreManagerAdapter.java:416)
>         at
> org.apache.ignite.internal.processors.cache.store.
> GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.
> java:382)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2059)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(
> GridCacheAdapter.java:2057)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(
> GridCacheContext.java:1406)
>         at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.
> java:6621)
>         at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(
> GridClosureProcessor.java:954)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
> After this I have to restart the cluster and application or have
> "transactions deadlock" message.
>
> And I checked: network is fine, no OOME in the ignite and cassandra
> (cassandra logs don't say anything about it). I understand that this is
> issue of persistence into cassandra but don't know how can I fix it. And
> after some attempts ignite node is stopped. I tried to change write behind
> parameters and disable it at all - but have the same issue. May be I have
> to
> see in the other place? May be this is important - i use IgniteDataStreamer
> for reading values from kafka (earlier I used the simple cache operations
> and didn't have this problem at all).
>
> final IgniteDataStreamer<String, BinaryObject> streamer =
> ignite.dataStreamer(callCache().getName());
> streamer.autoFlushFrequency(5000);
> streamer.keepBinary(true);
> streamer.perNodeBufferSize(5120);
>
>
> And one more question about persistence settings. If I'm working with
> BinaryObjects and ignite doesn't know about my class types, should I use
> valuePersistence class =  "org.apache.ignite.binary.BinaryObject"? I did
> it
> because peer class loading doesn't work for my own classes and I don't want
> to add new jars into all ignite nodes and all my apps.
>
> Thanks for help!
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
>
>
> --
>
> Best regards,
> Andrey V. Mashenkov
>

Mime
View raw message