I was thinking, it's only the one function that it has happened to so far with the other functions unaffected. My understanding is that the hector client pool should be shared across functions meaning that other functions should also be affected but they are not.

Here is the connection configuration that we are using currently in Spring 3.0:

    <bean id="cassandraHostConfigurator" class="me.prettyprint.cassandra.service.CassandraHostConfigurator">

        <!--  constructor-arg ref="cassandraConnectionString"/ -->

        <property name="hosts" value="cassandradevrk1:9393,cassandradevrk2:9393"/>

        <property name="maxIdle" value="-1"/>

        <property name="maxWaitTimeWhenExhausted" value="-1"/>

        <property name="cassandraThriftSocketTimeout" value="0"/>

    </bean>




On Tue, Aug 2, 2011 at 11:13 AM, Anthony Ikeda <anthony.ikeda.dev@gmail.com> wrote:
The link (which I may be misreading) is http://groups.google.com/group/hector-users/browse_thread/thread/8d7004b6f85a0f2e

It's only started happening today and happened on 2 occassions (8:43 and 10:21) performing the same function (querying a column family).

It seems to be trying to access a connection on one of the servers 

The client accesses the first node:
2011-08-02 08:43:06,541 ERROR [me.prettyprint.cassandra.connection.HThriftClient] - Could not flush transport (to be expected if the pool is shutting down) in close for client: CassandraClient<cassandradevrk1:9393-33>
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
...
2011-08-02 08:43:06,544 WARN [me.prettyprint.cassandra.connection.HConnectionManager] - Could not fullfill request on this host CassandraClient<cassandradevrk1:9393-33>
...
2011-08-02 08:43:06,543 ERROR [me.prettyprint.cassandra.connection.HConnectionManager] - MARK HOST AS DOWN TRIGGERED for host cassandradevrk1(10.130.202.34):9393 2011-08-02 08:43:06,543 ERROR [me.prettyprint.cassandra.connection.HConnectionManager] - Pool state on shutdown: <ConcurrentCassandraClientPoolByHost>:{cassandradevrk1(10.130.202.34):9393}; IsActive?: true; Active: 1; Blocked: 0; Idle: 15; NumBeforeExhausted: 49 2011-08-02 08:43:06,543 ERROR [me.prettyprint.cassandra.connection.ConcurrentHClientPool] - Shutdown triggered on <ConcurrentCassandraClientPoolByHost>:{cassandradevrk1(10.130.202.34):9393} 2011-08-02 08:43:06,544 ERROR [me.prettyprint.cassandra.connection.ConcurrentHClientPool] - Shutdown complete on <ConcurrentCassandraClientPoolByHost>:{cassandradevrk1(10.130.202.34):9393} 2011-08-02 08:43:06,544 INFO [me.prettyprint.cassandra.connection.CassandraHostRetryService] - Host detected as down was added to retry queue: cassandradevrk1(10.130.202.34):9393 2011-08-02 08:43:06,544 WARN [me.prettyprint.cassandra.connection.HConnectionManager] - Could not fullfill request on this host CassandraClient<
cassandradevrk1:9393-33> 2011-08-02 08:43:06,544 WARN [me.prettyprint.cassandra.connection.HConnectionManager] - Exception: me.prettyprint.hector.api.exceptions.HectorTransportException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset


Then it appears to try the second node and fails:
2011-08-02 08:43:06,556 INFO [me.prettyprint.cassandra.connection.HConnectionManager] - Client CassandraClient<cassandradevrk1:9393-33> released to inactive or dead pool. Closing.
2011-08-02 08:43:06,557 ERROR [me.prettyprint.cassandra.connection.HThriftClient] - Could not flush transport (to be expected if the pool is shutting down) in close for client: CassandraClient<cassandradevrk2:9393-49>
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
2011-08-02 08:43:06,558 ERROR [me.prettyprint.cassandra.connection.HConnectionManager] - MARK HOST AS DOWN TRIGGERED for host cassandradevrk2(10.130.202.35):9393
2011-08-02 08:43:06,559 ERROR [me.prettyprint.cassandra.connection.HConnectionManager] - Pool state on shutdown: <ConcurrentCassandraClientPoolByHost>:{cassandradevrk2(10.130.202.35):9393}; IsActive?: true; Active: 1; Blocked: 0; Idle: 15; NumBeforeExhausted: 49
2011-08-02 08:43:06,559 ERROR [me.prettyprint.cassandra.connection.ConcurrentHClientPool] - Shutdown triggered on <ConcurrentCassandraClientPoolByHost>:{cassandradevrk2(10.130.202.35):9393}
2011-08-02 08:43:06,559 ERROR [me.prettyprint.cassandra.connection.ConcurrentHClientPool] - Shutdown complete on <ConcurrentCassandraClientPoolByHost>:{cassandradevrk2(10.130.202.35):9393}
2011-08-02 08:43:06,559 INFO [me.prettyprint.cassandra.connection.CassandraHostRetryService] - Host detected as down was added to retry queue: cassandradevrk2(10.130.202.35):9393
2011-08-02 08:43:06,560 WARN [me.prettyprint.cassandra.connection.HConnectionManager] - Could not fullfill request on this host CassandraClient<cassandradevrk2:9393-49>
2011-08-02 08:43:06,560 WARN [me.prettyprint.cassandra.connection.HConnectionManager] - Exception: 
me.prettyprint.hector.api.exceptions.HectorTransportException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
The process is the same at 10:21.

Are the exceptions related to any external events (e.g. node restarts, network issues...)?
Not that I'm aware, unless there are firewall timeouts between the application and the node servers. Let me find out. The cassandra log files have no errors reported.

What versions of Hector and Cassandra are you running?
Cassandra 0.8.1, Hector 0.8.0-1




On Tue, Aug 2, 2011 at 10:37 AM, Jim Ancona <jim@anconafamily.com> wrote:
On Tue, Aug 2, 2011 at 4:36 PM, Anthony Ikeda
<anthony.ikeda.dev@gmail.com> wrote:
> I'm not sure if this is a problem with Hector or with Cassandra.
> We seem to be seeing broken pipe issues with our connections on the client
> side (Exception below). A bit of googling finds possibly a problem with the
> amount of data we are trying to store, although I'm certain our datasets are
> not all that large.

I'm not sure what you're referring to here. Large requests could lead
to timeouts, but that's not what you're seeing here. Could you link to
the page you're referencing?

> A nodetool ring command doesn't seem to present any downed nodes:
> Address         DC          Rack        Status State   Load            Owns
>    Token
>
>   153951716904446304929228999025275230571
> 10.130.202.34   datacenter1 rack1       Up     Normal  470.74 KB
> 79.19%  118538200848404459763384037192174096102
> 10.130.202.35   datacenter1 rack1       Up     Normal  483.63 KB
> 20.81%  153951716904446304929228999025275230571
>
> There are no errors in the cassandra server logs.
>
> Are there any particular timeouts on connections that we need to be aware
> of? Or perhaps configure on the Cassandra nodes? Is this purely and issue
> with the Hector API configuration?

There is a server side timeout (rpc_timeout_in_ms in cassandra.yaml)
and a Hector client-side timeout
(CassandraHostConfigurator.cassandraThriftSocketTimeout). But again,
the "Broken pipe" error is not a timeout, it indicates that something
happened to the underlying network socket. For example you will see
those when a server node is restarted.

Some questions that might help troubleshoot this:
How often are these occurring?
Does this affect both nodes in the cluster or just one?
Are the exceptions related to any external events (e.g. node restarts,
network issues...)?
What versions of Hector and Cassandra are you running?

Keep in mind that failures like this will normally be retried by
Hector, resulting in no loss of data. For that reason, I think that
exception is logged as a warning in the newest Hector versions.

We've seen something similar, but more catastrophic because it affects
connectivity to the entire cluster, not just a single node. See this
post for more details: http://goo.gl/hrgkw So far we haven't
identified the cause.

Jim

> Anthony
>
> 2011-08-02 08:43:06,541 ERROR
> [me.prettyprint.cassandra.connection.HThriftClient] - Could not flush
> transport (to be expected if the pool is shutting down) in close for client:
> CassandraClient<cassandradevrk1:9393-33>
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Broken pipe
>       at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
>       at
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
>       at
> me.prettyprint.cassandra.connection.HThriftClient.close(HThriftClient.java:85)
>       at
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:232)
>       at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131)
>       at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl.getSlice(KeyspaceServiceImpl.java:289)
>       at
> me.prettyprint.cassandra.model.thrift.ThriftSliceQuery$1.doInKeyspace(ThriftSliceQuery.java:53)
>       at
> me.prettyprint.cassandra.model.thrift.ThriftSliceQuery$1.doInKeyspace(ThriftSliceQuery.java:49)
>       at
> me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
>       at
> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
>       at
> me.prettyprint.cassandra.model.thrift.ThriftSliceQuery.execute(ThriftSliceQuery.java:48)
>       at
> com.wsgc.services.registry.persistenceservice.impl.cassandra.strategy.read.StandardFindRegistryPersistenceStrategy.findRegistryByProfileId(StandardFindRegistryPersistenceStrategy.java:237)
>       at
> com.wsgc.services.registry.persistenceservice.impl.cassandra.strategy.read.StandardFindRegistryPersistenceStrategy.execute(StandardFindRegistryPersistenceStrategy.java:277)
>       at
> com.wsgc.services.registry.registryservice.impl.service.StandardRegistryService.getRegistriesByProfileId(StandardRegistryService.java:327)
>       at
> com.wsgc.services.registry.webapp.impl.RegistryServicesController.getRegistriesByProfileId(RegistryServicesController.java:247)
>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>       at java.lang.reflect.Method.invoke(Method.java:597)
>       at
> org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
>       at
> org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
>       at
> org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
>       at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
>       at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>       at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>       at
> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>       at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>       at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>       at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>       at
> org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
>       at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>       at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>       at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>       at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>       at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>       at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
>       at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>       at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>       at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>       at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>       at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
>       at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
>       at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
>       at
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
>       at
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
>       at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
>       at java.lang.Thread.run(Thread.java:662)
> Caused by: java.net.SocketException: Broken pipe
>       at java.net.SocketOutputStream.socketWrite0(Native Method)
>       at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>       at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>       at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
>       ... 47 more
> 2011-08-02 08:43:06,543 ERROR
> [me.prettyprint.cassandra.connection.HConnectionManager] - MARK HOST AS DOWN
> TRIGGERED for host cassandradevrk1(10.130.202.34):9393
> 2011-08-02 08:43:06,543 ERROR
> [me.prettyprint.cassandra.connection.HConnectionManager] - Pool state on
> shutdown:
> <ConcurrentCassandraClientPoolByHost>:{cassandradevrk1(10.130.202.34):9393};
> IsActive?: true; Active: 1; Blocked: 0; Idle: 15; NumBeforeExhausted: 49
> 2011-08-02 08:43:06,543 ERROR
> [me.prettyprint.cassandra.connection.ConcurrentHClientPool] - Shutdown
> triggered on
> <ConcurrentCassandraClientPoolByHost>:{cassandradevrk1(10.130.202.34):9393}
> 2011-08-02 08:43:06,544 ERROR
> [me.prettyprint.cassandra.connection.ConcurrentHClientPool] - Shutdown
> complete on
> <ConcurrentCassandraClientPoolByHost>:{cassandradevrk1(10.130.202.34):9393}
> 2011-08-02 08:43:06,544 INFO
> [me.prettyprint.cassandra.connection.CassandraHostRetryService] - Host
> detected as down was added to retry queue:
> cassandradevrk1(10.130.202.34):9393
> 2011-08-02 08:43:06,544 WARN
> [me.prettyprint.cassandra.connection.HConnectionManager] - Could not
> fullfill request on this host CassandraClient<cassandradevrk1:9393-33>
>