ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From naomgabo <...@ballab.com>
Subject Ignite Cassandra write error
Date Sat, 11 Nov 2017 17:00:11 GMT
We are facing a seeing an intermittent but regular error on writing to
Cassandra. The write is coming from Ignite itself as opposed to some code we
have written. Also these are serialised objects being written to Cassandra.
We suspected that this was some large object in the cache but could not
locate any object that could be over a couple of KB. The system is in
production since June (so around 3-4 months) and stable, and the problems
started only about 10 days ago (no updates done). The system has also been
tested under very heavy traffic before sending into production (around 100x
of what it sees in production).

My questions are
- any suggestion on potential source of the issue
- any suggestion of how we could debug/trace the issue?


Nov 11, 2017 4:43:14 PM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to process 36 of 36 elements, during BULK_WRITE operation
with Cassandra
class org.apache.ignite.IgniteException: Failed to execute Cassandra
BULK_WRITE operation
	at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:298)
	at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.writeAll(CassandraCacheStore.java:273)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:804)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:720)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:75)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1135)
	at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1006)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.WriteFailureException:
Cassandra failure during write query at consistency LOCAL_ONE (1 responses
were required but only 0 replica responded, 3 failed)
	at
com.datastax.driver.core.exceptions.WriteFailureException.copy(WriteFailureException.java:80)
	at
com.datastax.driver.core.exceptions.WriteFailureException.copy(WriteFailureException.java:28)
	at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
	at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
	at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:278)
	... 8 more
Caused by: com.datastax.driver.core.exceptions.WriteFailureException:
Cassandra failure during write query at consistency LOCAL_ONE (1 responses
were required but only 0 replica responded, 3 failed)
	at
com.datastax.driver.core.exceptions.WriteFailureException.copy(WriteFailureException.java:91)
	at com.datastax.driver.core.Responses$Error.asException(Responses.java:126)
	at
com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
	at
com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:184)
	at
com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43)
	at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:798)
	at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:617)
	at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005)
	at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928)
	at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
	at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
	at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
	at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
	at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
	at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
	at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
	at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
	at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
	at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
	at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
	at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
	... 1 more
Caused by: com.datastax.driver.core.exceptions.WriteFailureException:
Cassandra failure during write query at consistency LOCAL_ONE (1 responses
were required but only 0 replica responded, 3 failed)
	at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:73)
	at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37)
	at
com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:266)
	at
com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:246)
	at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
	... 14 more



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Mime
View raw message