I believe the problem here is with Pelops.   The batch_mutate does a 
flush and the pelops flush is not required.   Removing the flush from 
pelops fixes the issue.

By chance is anyone that has seen this error using Pelops?


On 8/31/10 6:54 PM, Carl Bruecken wrote:
> I've made some progress on narrowing this down and am able to 
> reproduce easily.   I am using pelops as a client and I configured the 
> policy in pelops to only establish 1 connection to a cassandra node.  
> I'm able to step through the pelops code line by line and see the 
> resulting thrift transport logging in cassandra.   Seems that flushing 
> the transport causes the unwanted TTransportConnection in the server 
> and subsequent closing of the connection.   The connection should stay 
> open after flushing.   When there are many connection established the 
> behaviour seems intermittent and many operations succeed.
>
>
>
> Here are the details
>
> 1) The trigger from the client side is when the framed transport is 
> flushed.
>                conn.getAPI().batch_mutate(convertedBatch, cLevel);
>                 // Flush connection
>                 conn.flush();
>
> 2) In CustomTThreadPoolServer.java in Cassandra I modified the code to 
> log TTransportExceptions.
>
>         catch (TTransportException ttx) {
>         LOGGER.error("Transport exception", ttx);
>     } catch (TException tx) {
>         LOGGER.error("Thrift error occurred during processing of 
> message.", tx);
>     } catch (Exception x) {
>         LOGGER.error("Error occurred during processing of message.", x);
>     }
>
>
> 3) Here is the exception that is ignored in cassandra.   Flushing the 
> transport causes the server to believe the client has closed the 
> connection.
>
> org.apache.thrift.transport.TTransportException: Cannot read. Remote 
> side has closed. Tried to read 4 bytes, but only got 0 bytes.
>     at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>     at 
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:369)
>     at 
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:295)
>     at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:202)
>     at 
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2487)
>     at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:637)
>
> 4) The next batch mutate to this connection caused the exception in 
> the client
>
>  WARN [main] 2010-08-31 18:40:06,749 Operand.java (line 72) Operation 
> failed as result of network exception. Connection must be destroyed.  
> See cause for details...
> org.apache.thrift.transport.TTransportException: 
> java.net.SocketException: Connection reset
>     at 
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>     at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>     at 
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>     at 
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>     at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>     at 
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:369)
>     at 
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:295)
>     at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:202)
>     at 
> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:905)
>     at 
> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:889)
>     at org.scale7.cassandra.pelops.Mutator$1.execute(Mutator.java:42)
>     at org.scale7.cassandra.pelops.Mutator$1.execute(Mutator.java:38)
>     at org.scale7.cassandra.pelops.Operand.tryOperation(Operand.java:53)
>     at org.scale7.cassandra.pelops.Mutator.execute(Mutator.java:49)
>     at com.aol.data.c7.App.doWork(App.java:41)
>     at com.aol.data.c7.App.main(App.java:77)
> Caused by: java.net.SocketException: Connection reset
>     at java.net.SocketInputStream.read(SocketInputStream.java:168)
>     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>     at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>     at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>     at 
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>     ... 15 more
>
>
>
>
> On 8/31/10 4:04 PM, Jonathan Ellis wrote:
>>
>> No, I don't know that anyone has reproduced that.  TTransportException
>> always means "something went wrong on the thrift side" in my
>> experience, it shouldn't be cassandra-version specific.
>>
>> On Tue, Aug 31, 2010 at 12:53 PM, Carl Bruecken
>> <carl.bruecken@corp.aol.com> wrote:
>> >
>> >  Are there any estimates as to when a fix for this will be checked into
>> > trunk?
>> >
>> > Coincidentally, has anyone tracked down the issue?
>> >
>> >  I'm experiencing same issue with nightly build from a week ago.
>> >
>> > Thank You
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>