incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Senthil, Athinanthny X. -ND" <Athinanthny.X.Senthil....@disney.com>
Subject Hector Could not flush transport error
Date Sat, 08 Feb 2014 00:05:34 GMT
We get the below error in app logs when it's trying to hit DC which doesn't get traffic in
multi DC cluster. But other DC where all traffic hits seems to be responding without any issues.
App read/write requests are timing out. Hector client version is 1.0.5. On cassandra side
, we are getting socket timeouts. Configs are all same in both DC and on app side as well?

We expanded this cluster by adding 3 nodes to 2 DC . It has 6 nodes in each DC.  App did not
recognize new nodes until it was explicitly defined in property file. Since then we are seeing
the issue/error below. This DC acts like failover DC and this is only place where get socket
timeout errors in cassandra layer in all nodes.

Do we need to change any config ? Will increasing this value thrift_framed_transport_size_in_mb:
15 help?

Tried enabling assandra.useSocketKeepalive=true, timeout still occurs. cassandraThriftSocketTimeout=0
in app property


2014-02-06 10:38:57,936 -0800 [Hector.me.prettyprint.cassandra.connection.NodeAutoDiscoverService-1]
ERROR me.prettyprint.cassandra.connection.client.HThriftClient - Could not flush transport
(to be expected if the pool is shutting
down) in close for client: CassandraClient<:9160-10220>
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:98)
at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:26)
at me.prettyprint.cassandra.connection.HConnectionManager.closeClient(HConnectionManager.java:323)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:272)
at me.prettyprint.cassandra.service.AbstractCluster.describeKeyspaces(AbstractCluster.java:136)
at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.discoverNodes(NodeAutoDiscoverService.java:97)
at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.doAddNodes(NodeAutoDiscoverService.java:68)
at me.prettyprint.cassandra.connection.NodeAutoDiscoverService$QueryRing.run(NodeAutoDiscoverService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)

Mime
View raw message