I saw this with pelops and only with batch mutate. Other calls worked fine. On 09/01/2010 08:16 AM, Carl Bruecken wrote: > I believe the problem here is with Pelops. The batch_mutate does a > flush and the pelops flush is not required. Removing the flush from > pelops fixes the issue. > > By chance is anyone that has seen this error using Pelops? > > > On 8/31/10 6:54 PM, Carl Bruecken wrote: >> I've made some progress on narrowing this down and am able to >> reproduce easily. I am using pelops as a client and I configured >> the policy in pelops to only establish 1 connection to a cassandra >> node. I'm able to step through the pelops code line by line and see >> the resulting thrift transport logging in cassandra. Seems that >> flushing the transport causes the unwanted TTransportConnection in >> the server and subsequent closing of the connection. The connection >> should stay open after flushing. When there are many connection >> established the behaviour seems intermittent and many operations succeed. >> >> >> >> Here are the details >> >> 1) The trigger from the client side is when the framed transport is >> flushed. >> conn.getAPI().batch_mutate(convertedBatch, cLevel); >> // Flush connection >> conn.flush(); >> >> 2) In CustomTThreadPoolServer.java in Cassandra I modified the code >> to log TTransportExceptions. >> >> catch (TTransportException ttx) { >> LOGGER.error("Transport exception", ttx); >> } catch (TException tx) { >> LOGGER.error("Thrift error occurred during processing of >> message.", tx); >> } catch (Exception x) { >> LOGGER.error("Error occurred during processing of message.", x); >> } >> >> >> 3) Here is the exception that is ignored in cassandra. Flushing the >> transport causes the server to believe the client has closed the >> connection. >> >> org.apache.thrift.transport.TTransportException: Cannot read. Remote >> side has closed. Tried to read 4 bytes, but only got 0 bytes. >> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:369) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:295) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:202) >> at >> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2487) >> at >> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:637) >> >> 4) The next batch mutate to this connection caused the exception in >> the client >> >> WARN [main] 2010-08-31 18:40:06,749 Operand.java (line 72) Operation >> failed as result of network exception. Connection must be destroyed. >> See cause for details... >> org.apache.thrift.transport.TTransportException: >> java.net.SocketException: Connection reset >> at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) >> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >> at >> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) >> at >> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) >> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:369) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:295) >> at >> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:202) >> at >> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:905) >> at >> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:889) >> at org.scale7.cassandra.pelops.Mutator$1.execute(Mutator.java:42) >> at org.scale7.cassandra.pelops.Mutator$1.execute(Mutator.java:38) >> at org.scale7.cassandra.pelops.Operand.tryOperation(Operand.java:53) >> at org.scale7.cassandra.pelops.Mutator.execute(Mutator.java:49) >> at com.aol.data.c7.App.doWork(App.java:41) >> at com.aol.data.c7.App.main(App.java:77) >> Caused by: java.net.SocketException: Connection reset >> at java.net.SocketInputStream.read(SocketInputStream.java:168) >> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) >> at java.io.BufferedInputStream.read(BufferedInputStream.java:317) >> at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) >> ... 15 more >> >> >> >> >> On 8/31/10 4:04 PM, Jonathan Ellis wrote: >>> >>> No, I don't know that anyone has reproduced that. TTransportException >>> always means "something went wrong on the thrift side" in my >>> experience, it shouldn't be cassandra-version specific. >>> >>> On Tue, Aug 31, 2010 at 12:53 PM, Carl Bruecken >>> wrote: >>> > >>> > Are there any estimates as to when a fix for this will be checked >>> into >>> > trunk? >>> > >>> > Coincidentally, has anyone tracked down the issue? >>> > >>> > I'm experiencing same issue with nightly build from a week ago. >>> > >>> > Thank You >>> > >>> >>> >>> >>> -- >>> Jonathan Ellis >>> Project Chair, Apache Cassandra >>> co-founder of Riptano, the source for professional Cassandra support >>> http://riptano.com >>> >> > -- *Andres March* amarch@qualcomm.com Qualcomm Internet Services