From user-return-19561-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Aug 5 06:14:45 2011 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9A65578FA for ; Fri, 5 Aug 2011 06:14:45 +0000 (UTC) Received: (qmail 87827 invoked by uid 500); 5 Aug 2011 06:14:40 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 87655 invoked by uid 500); 5 Aug 2011 06:14:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 87645 invoked by uid 99); 5 Aug 2011 06:14:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Aug 2011 06:14:17 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=SPF_NEUTRAL,TO_NO_BRKTS_PCNT X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [173.201.192.170] (HELO p3plwbeout13-05.prod.phx3.secureserver.net) (173.201.192.170) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 05 Aug 2011 06:14:09 +0000 Received: (qmail 20067 invoked from network); 5 Aug 2011 06:13:47 -0000 Received: from unknown (HELO localhost) (10.6.247.5) by p3plwbeout13-05.prod.phx3.secureserver.net with SMTP; 5 Aug 2011 06:13:32 -0000 Received: (qmail 9049 invoked by uid 99); 5 Aug 2011 06:13:32 -0000 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" X-Originating-IP: 178.197.254.3 User-Agent: Web-Based Email 5.5.14 Message-Id: <20110804231331.050558854cd8e3604bbb7dcb1d74d208.4412c733c4.wbe@email13.secureserver.net> From: "Tim Snyder" To: user@cassandra.apache.org Subject: RE: Trying to find the problem with a broken pipe Date: Thu, 04 Aug 2011 23:13:31 -0700 Mime-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org I no longer get the error on the loader program. The steps I took to fix=0A= it are increasing the thrift_max_message_length_in_mb msg length,=0Astoppin= g cassandra, blowing away the prior data store, and then=0Arestarting cassa= ndra. =0A=0ATim=0A=0A=0A-------- Original Message --------=0ASubject: Re: T= rying to find the problem with a broken pipe=0AFrom: aaron morton =0ADate: Fri, August 05, 2011 12:58 am=0ATo: user@cassandra= .apache.org=0A=0AIt's probably a network thing. =0A=0AThe only thing I can = think of in cassandra is=0Athrift_max_message_length_in_mb in the config. T= hat config setting will=0Aresult in a TException thrown on the server side = (i think), not sure if=0Athat makes the server kill the socket. I would hop= e the error returns to=0Athe client. =0A=0APerhaps check the server log. = =0A=0ACheers=0A =0A-----------------=0AAaron Morton=0AFreelance Cassandra D= eveloper=0A@aaronmorton=0Ahttp://www.thelastpickle.com=0A=0AOn 4 Aug 2011, = at 23:05, Tim Snyder wrote:=0A=0A> I am getting the same problem (Broken Pi= pe) on a loader program, after=0A> about 8 million read, write pairs. I am = pushing serialized objects into=0A> a column with the program, the object i= t seems to be doing it on is much=0A> larger than the prior objects, so I a= m wondering if it is possibly a=0A> column size streaming issue through the= thrift api? I am using Cassandra=0A> 0.8.0 and Hector 0.8.0-1=0A> =0A> Tim= =0A> =0A> -------- Original Message --------=0A> Subject: Re: Trying to fin= d the problem with a broken pipe=0A> From: Anthony Ikeda =0A> Date: Tue, August 02, 2011 10:43 pm=0A> To: user@cassandra.= apache.org=0A> =0A>> Very interesting. After the second host goes down do y= ou see=0A>> "me.prettyprint.hector.api.exceptions.HectorException: All host= pools=0A>> marked down. Retry burden pushed out to client"?=0A> =0A> No, t= he last message is:=0A> 2011-08-02 08:43:06,561 INFO=0A> [me.prettyprint.ca= ssandra.connection.HConnectionManager] - Client=0A> CassandraClient released to inactive or dead=0A> pool. Closing.> Does you= r client recover after a period of time?=0A> =0A> =0A> =0A> The application= seems to be fine for now but my concern is the=0A> connection pooling as w= ell - I mean do we have one pool or multiple?=0A> I'll post to the Hector u= ser group about the pooling because the=0A> incident seems so isolated. We = also have our infrastructure team looking=0A> into the communication betwee= n the application server and the cassandra=0A> nodes.=0A> =0A> =0A> So far = it's still a mystery.=0A> =0A> =0A> =0A> =0A> =0A> On Tue, Aug 2, 2011 at 1= :25 PM, Jim Ancona wrote:=0A> On Tue, Aug 2, 2011 at= 6:13 PM, Anthony Ikeda=0A> wrote:=0A> =0A>> = The link (which I may be misreading)=0A>> is=0A> http://groups.google.com/g= roup/hector-users/browse_thread/thread/8d7004b6f85a0f2e=0A> =0A> =0A> I had= n't found that one, but I doubt that our issue is related to that.=0A> =0A>= =0A>> It's only started happening today and happened on 2 occassions (8:43= =0A> and=0A>> 10:21) performing the same function (querying a column family= ).=0A>> It seems to be trying to access a connection on one of the servers= =0A>> The client accesses the first node:=0A>> =0A>> 2011-08-02 08:43:06,54= 1 ERROR=0A>> [me.prettyprint.cassandra.connection.HThriftClient] - Could no= t flush=0A>> transport (to be expected if the pool is shutting down) in clo= se for=0A> client:=0A>> CassandraClient=0A>> org.a= pache.thrift.transport.TTransportException:=0A> java.net.SocketException:= =0A>> Broken pipe=0A>> =0A>> ...=0A>> 2011-08-02 08:43:06,544 WARN=0A>> [me= .prettyprint.cassandra.connection.HConnectionManager] - Could not=0A>> full= fill request on this host=0A> CassandraClient=0A>>= =0A>> ...=0A>> =0A>> 2011-08-02 08:43:06,543 ERROR=0A>> [me.prettyprint.ca= ssandra.connection.HConnectionManager] - MARK HOST=0A> AS DOWN=0A>> TRIGGER= ED for host cassandradevrk1(10.130.202.34):9393=0A>> 2011-08-02 08:43:06,54= 3 ERROR=0A>> [me.prettyprint.cassandra.connection.HConnectionManager] - Poo= l state=0A> on=0A>> shutdown:=0A>> =0A> :{cassandradevrk1(10.130.202.34):9393};=0A>> IsActive?: true; Active: 1; = Blocked: 0; Idle: 15; NumBeforeExhausted:=0A> 49=0A>> 2011-08-02 08:43:06,5= 43 ERROR=0A>> [me.prettyprint.cassandra.connection.ConcurrentHClientPool] -= =0A> Shutdown=0A>> triggered on=0A>> =0A> :{cassandradevrk1(10.130.202.34):9393}=0A>> 2011-08-02 08:43:06,544 ERR= OR=0A>> [me.prettyprint.cassandra.connection.ConcurrentHClientPool] -=0A> S= hutdown=0A>> complete on=0A>> =0A> :{c= assandradevrk1(10.130.202.34):9393}=0A>> 2011-08-02 08:43:06,544 INFO=0A>> = [me.prettyprint.cassandra.connection.CassandraHostRetryService] -=0A> Host= =0A>> detected as down was added to retry queue:=0A>> cassandradevrk1(10.13= 0.202.34):9393=0A>> 2011-08-02 08:43:06,544 WARN=0A>> [me.prettyprint.cassa= ndra.connection.HConnectionManager] - Could not=0A>> fullfill request on th= is host CassandraClient<=0A>> cassandradevrk1:9393-33>=0A>> 2011-08-02 08:4= 3:06,544 WARN=0A>> [me.prettyprint.cassandra.connection.HConnectionManager]= - Exception:=0A>> me.prettyprint.hector.api.exceptions.HectorTransportExce= ption:=0A>> org.apache.thrift.transport.TTransportException:=0A> java.net.S= ocketException:=0A>> Connection reset=0A>> =0A>> =0A>> Then it appears to t= ry the second node and fails:=0A>> =0A>> 2011-08-02 08:43:06,556 INFO=0A>> = [me.prettyprint.cassandra.connection.HConnectionManager] - Client=0A>> Cass= andraClient released to inactive or dead=0A> pool.= =0A>> Closing.=0A>> 2011-08-02 08:43:06,557 ERROR=0A>> [me.prettyprint.cass= andra.connection.HThriftClient] - Could not flush=0A>> transport (to be exp= ected if the pool is shutting down) in close for=0A> client:=0A>> Cassandra= Client=0A>> =0A>> org.apache.thrift.transport.TTra= nsportException:=0A> java.net.SocketException:=0A>> Broken pipe=0A>> =0A>> = 2011-08-02 08:43:06,558 ERROR=0A>> [me.prettyprint.cassandra.connection.HCo= nnectionManager] - MARK HOST=0A> AS DOWN=0A>> TRIGGERED for host cassandrad= evrk2(10.130.202.35):9393=0A>> 2011-08-02 08:43:06,559 ERROR=0A>> [me.prett= yprint.cassandra.connection.HConnectionManager] - Pool state=0A> on=0A>> sh= utdown:=0A>> =0A> :{cassandradevrk2(10= .130.202.35):9393};=0A>> IsActive?: true; Active: 1; Blocked: 0; Idle: 15; = NumBeforeExhausted:=0A> 49=0A>> 2011-08-02 08:43:06,559 ERROR=0A>> [me.pret= typrint.cassandra.connection.ConcurrentHClientPool] -=0A> Shutdown=0A>> tri= ggered on=0A>> =0A> :{cassandradevrk2(= 10.130.202.35):9393}=0A>> 2011-08-02 08:43:06,559 ERROR=0A>> [me.prettyprin= t.cassandra.connection.ConcurrentHClientPool] -=0A> Shutdown=0A>> complete = on=0A>> =0A> :{cassandradevrk2(10.130.= 202.35):9393}=0A>> 2011-08-02 08:43:06,559 INFO=0A>> [me.prettyprint.cassan= dra.connection.CassandraHostRetryService] -=0A> Host=0A>> detected as down = was added to retry queue:=0A>> cassandradevrk2(10.130.202.35):9393=0A>> 201= 1-08-02 08:43:06,560 WARN=0A>> [me.prettyprint.cassandra.connection.HConnec= tionManager] - Could not=0A>> fullfill request on this host=0A> CassandraCl= ient=0A>> 2011-08-02 08:43:06,560 WARN=0A>> [me.pr= ettyprint.cassandra.connection.HConnectionManager] - Exception:=0A>> me.pre= ttyprint.hector.api.exceptions.HectorTransportException:=0A>> org.apache.th= rift.transport.TTransportException:=0A> java.net.SocketException:=0A>> Conn= ection reset=0A> =0A> =0A> =0A> Very interesting. After the second host goe= s down do you see=0A> "me.prettyprint.hector.api.exceptions.HectorException= : All host pools=0A> marked down. Retry burden pushed out to client"?=0A> = =0A> Does your client recover after a period of time?=0A> =0A>> =0A>> The p= rocess is the same at 10:21.=0A>> Are the exceptions related to any externa= l events (e.g. node=0A>> restarts, network issues...)?=0A>> Not that I'm aw= are, unless there are firewall timeouts between the=0A>> application and th= e node servers. Let me find out. The cassandra log=0A> files=0A>> have no e= rrors reported.=0A>> What versions of Hector and Cassandra are you running?= =0A>> Cassandra 0.8.1, Hector 0.8.0-1=0A> =0A> =0A> Our issue is occurring = with Cassandra 0.7.8 and Hector 0.7-30. We plan=0A> to deploy Hector 0.7-31= this week and to turn on useSocketKeepalive.=0A> Are you using that? We're= also using tcpdump to capture packets when=0A> failures occur to see if th= ere are anomalies in the network traffic.=0A> =0A> Jim=0A> =0A> =0A> =0A>> = =0A>> =0A>> =0A>> On Tue, Aug 2, 2011 at 10:37 AM, Jim Ancona =0A> wrote:=0A>>> =0A>>> On Tue, Aug 2, 2011 at 4:36 PM, Anthony I= keda=0A>>> wrote:=0A>>>> I'm not sure if this= is a problem with Hector or with Cassandra.=0A>>>> We seem to be seeing br= oken pipe issues with our connections on=0A> the=0A>>>> client=0A>>>> side = (Exception below). A bit of googling finds possibly a problem=0A> with=0A>>= >> the=0A>>>> amount of data we are trying to store, although I'm certain o= ur=0A> datasets=0A>>>> are=0A>>>> not all that large.=0A>>> =0A>>> I'm not = sure what you're referring to here. Large requests could=0A> lead=0A>>> to = timeouts, but that's not what you're seeing here. Could you link=0A> to=0A>= >> the page you're referencing?=0A>>> =0A>>>> A nodetool ring command doesn= 't seem to present any downed nodes:=0A>>>> Address DC Rack Status State Lo= ad=0A>>>> Owns=0A>>>> Token=0A>>>> =0A>>>> 15395171690444630492922899902527= 5230571=0A>>>> 10.130.202.34 datacenter1 rack1 Up Normal 470.74 KB=0A>>>> 7= 9.19% 118538200848404459763384037192174096102=0A>>>> 10.130.202.35 datacent= er1 rack1 Up Normal 483.63 KB=0A>>>> 20.81% 1539517169044463049292289990252= 75230571=0A>>>> =0A>>>> There are no errors in the cassandra server logs.= =0A>>>> =0A>>>> Are there any particular timeouts on connections that we ne= ed to=0A> be=0A>>>> aware=0A>>>> of? Or perhaps configure on the Cassandra = nodes? Is this purely=0A> and=0A>>>> issue=0A>>>> with the Hector API confi= guration?=0A>>> =0A>>> There is a server side timeout (rpc_timeout_in_ms in= cassandra.yaml)=0A>>> and a Hector client-side timeout=0A>>> (CassandraHos= tConfigurator.cassandraThriftSocketTimeout). But again,=0A>>> the "Broken p= ipe" error is not a timeout, it indicates that=0A> something=0A>>> happened= to the underlying network socket. For example you will see=0A>>> those whe= n a server node is restarted.=0A>>> =0A>>> Some questions that might help t= roubleshoot this:=0A>>> How often are these occurring?=0A>>> Does this affe= ct both nodes in the cluster or just one?=0A>>> Are the exceptions related = to any external events (e.g. node=0A> restarts,=0A>>> network issues...)?= =0A>>> What versions of Hector and Cassandra are you running?=0A>>> =0A>>> = Keep in mind that failures like this will normally be retried by=0A>>> Hect= or, resulting in no loss of data. For that reason, I think that=0A>>> excep= tion is logged as a warning in the newest Hector versions.=0A>>> =0A>>> We'= ve seen something similar, but more catastrophic because it=0A> affects=0A>= >> connectivity to the entire cluster, not just a single node. See this=0A>= >> post for more details: http://goo.gl/hrgkw So far we haven't=0A>>> ident= ified the cause.=0A>>> =0A>>> Jim=0A>>> =0A>>>> Anthony=0A>>>> =0A>>>> 2011= -08-02 08:43:06,541 ERROR=0A>>>> [me.prettyprint.cassandra.connection.HThri= ftClient] - Could not=0A> flush=0A>>>> transport (to be expected if the poo= l is shutting down) in close=0A> for=0A>>>> client:=0A>>>> CassandraClient<= cassandradevrk1:9393-33>=0A>>>> org.apache.thrift.transport.TTransportExcep= tion:=0A>>>> java.net.SocketException:=0A>>>> Broken pipe=0A>>>> at=0A>>>> = =0A>>>> =0A> org.apache.thrift.transport.TIOStreamTransport.write(TIOStream= Transport.java:147)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.thrift.transpo= rt.TFramedTransport.flush(TFramedTransport.java:156)=0A>>>> at=0A>>>> =0A>>= >> =0A> me.prettyprint.cassandra.connection.HThriftClient.close(HThriftClie= nt.java:85)=0A>>>> at=0A>>>> =0A>>>> =0A> me.prettyprint.cassandra.connecti= on.HConnectionManager.operateWithFailover(HConnectionManager.java:232)=0A>>= >> at=0A>>>> =0A>>>> =0A> me.prettyprint.cassandra.service.KeyspaceServiceI= mpl.operateWithFailover(KeyspaceServiceImpl.java:131)=0A>>>> at=0A>>>> =0A>= >>> =0A> me.prettyprint.cassandra.service.KeyspaceServiceImpl.getSlice(Keys= paceServiceImpl.java:289)=0A>>>> at=0A>>>> =0A>>>> =0A> me.prettyprint.cass= andra.model.thrift.ThriftSliceQuery$1.doInKeyspace(ThriftSliceQuery.java:53= )=0A>>>> at=0A>>>> =0A>>>> =0A> me.prettyprint.cassandra.model.thrift.Thrif= tSliceQuery$1.doInKeyspace(ThriftSliceQuery.java:49)=0A>>>> at=0A>>>> =0A>>= >> =0A> me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspa= ceAndMeasure(KeyspaceOperationCallback.java:20)=0A>>>> at=0A>>>> =0A>>>> = =0A> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKe= yspace.java:85)=0A>>>> at=0A>>>> =0A>>>> =0A> me.prettyprint.cassandra.mode= l.thrift.ThriftSliceQuery.execute(ThriftSliceQuery.java:48)=0A>>>> at=0A>>>= > =0A>>>> =0A> com.wsgc.services.registry.persistenceservice.impl.cassandra= .strategy.read.StandardFindRegistryPersistenceStrategy.findRegistryByProfil= eId(StandardFindRegistryPersistenceStrategy.java:237)=0A>>>> at=0A>>>> =0A>= >>> =0A> com.wsgc.services.registry.persistenceservice.impl.cassandra.strat= egy.read.StandardFindRegistryPersistenceStrategy.execute(StandardFindRegist= ryPersistenceStrategy.java:277)=0A>>>> at=0A>>>> =0A>>>> =0A> com.wsgc.serv= ices.registry.registryservice.impl.service.StandardRegistryService.getRegis= triesByProfileId(StandardRegistryService.java:327)=0A>>>> at=0A>>>> =0A>>>>= =0A> com.wsgc.services.registry.webapp.impl.RegistryServicesController.get= RegistriesByProfileId(RegistryServicesController.java:247)=0A>>>> at sun.re= flect.NativeMethodAccessorImpl.invoke0(Native=0A> Method)=0A>>>> at=0A>>>> = =0A>>>> =0A> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:39)=0A>>>> at=0A>>>> =0A>>>> =0A> sun.reflect.DelegatingMethodA= ccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)=0A>>>> at java.lan= g.reflect.Method.invoke(Method.java:597)=0A>>>> at=0A>>>> =0A>>>> =0A> org.= springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHand= lerMethod(HandlerMethodInvoker.java:175)=0A>>>> at=0A>>>> =0A>>>> =0A> org.= springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.i= nvokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)=0A>>>> at=0A>>>= > =0A>>>> =0A> org.springframework.web.servlet.mvc.annotation.AnnotationMet= hodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)=0A>>>> at= =0A>>>> =0A>>>> =0A> org.springframework.web.servlet.DispatcherServlet.doDi= spatch(DispatcherServlet.java:774)=0A>>>> at=0A>>>> =0A>>>> =0A> org.spring= framework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:71= 9)=0A>>>> at=0A>>>> =0A>>>> =0A> org.springframework.web.servlet.FrameworkS= ervlet.processRequest(FrameworkServlet.java:644)=0A>>>> at=0A>>>> =0A>>>> = =0A> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServle= t.java:549)=0A>>>> at=0A> javax.servlet.http.HttpServlet.service(HttpServle= t.java:617)=0A>>>> at=0A> javax.servlet.http.HttpServlet.service(HttpServle= t.java:717)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.catalina.core.Applicat= ionFilterChain.internalDoFilter(ApplicationFilterChain.java:290)=0A>>>> at= =0A>>>> =0A>>>> =0A> org.apache.catalina.core.ApplicationFilterChain.doFilt= er(ApplicationFilterChain.java:206)=0A>>>> at=0A>>>> =0A>>>> =0A> org.sprin= gframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMet= hodFilter.java:77)=0A>>>> at=0A>>>> =0A>>>> =0A> org.springframework.web.fi= lter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)=0A>>>> at= =0A>>>> =0A>>>> =0A> org.apache.catalina.core.ApplicationFilterChain.intern= alDoFilter(ApplicationFilterChain.java:235)=0A>>>> at=0A>>>> =0A>>>> =0A> o= rg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh= ain.java:206)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.catalina.core.Standa= rdWrapperValve.invoke(StandardWrapperValve.java:233)=0A>>>> at=0A>>>> =0A>>= >> =0A> org.apache.catalina.core.StandardContextValve.invoke(StandardContex= tValve.java:191)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.catalina.authenti= cator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)=0A>>>> at=0A>>>>= =0A>>>> =0A> org.apache.catalina.core.StandardHostValve.invoke(StandardHos= tValve.java:127)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.catalina.valves.E= rrorReportValve.invoke(ErrorReportValve.java:102)=0A>>>> at=0A>>>> =0A>>>> = =0A> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv= e.java:109)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.catalina.connector.Coy= oteAdapter.service(CoyoteAdapter.java:298)=0A>>>> at=0A>>>> =0A> org.apache= .jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)=0A>>>> at=0A>>>= > =0A> org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)= =0A>>>> at=0A>>>> org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.j= ava:774)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.jk.common.ChannelSocket.p= rocessConnection(ChannelSocket.java:703)=0A>>>> at=0A>>>> =0A>>>> =0A> org.= apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:89= 6)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.tomcat.util.threads.ThreadPool$= ControlRunnable.run(ThreadPool.java:690)=0A>>>> at java.lang.Thread.run(Thr= ead.java:662)=0A>>>> Caused by: java.net.SocketException: Broken pipe=0A>>>= > at java.net.SocketOutputStream.socketWrite0(Native Method)=0A>>>> at=0A>>= >> =0A> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)= =0A>>>> at=0A> java.net.SocketOutputStream.write(SocketOutputStream.java:13= 6)=0A>>>> at=0A>>>> =0A>>>> =0A> org.apache.thrift.transport.TIOStreamTrans= port.write(TIOStreamTransport.java:145)=0A>>>> ... 47 more=0A>>>> 2011-08-0= 2 08:43:06,543 ERROR=0A>>>> [me.prettyprint.cassandra.connection.HConnectio= nManager] - MARK=0A> HOST AS=0A>>>> DOWN=0A>>>> TRIGGERED for host cassandr= adevrk1(10.130.202.34):9393=0A>>>> 2011-08-02 08:43:06,543 ERROR=0A>>>> [me= .prettyprint.cassandra.connection.HConnectionManager] - Pool=0A> state on= =0A>>>> shutdown:=0A>>>> =0A>>>> =0A> = :{cassandradevrk1(10.130.202.34):9393};=0A>>>> IsActive?: true; Active: 1; = Blocked: 0; Idle: 15;=0A> NumBeforeExhausted: 49=0A>>>> 2011-08-02 08:43:06= ,543 ERROR=0A>>>> [me.prettyprint.cassandra.connection.ConcurrentHClientPoo= l] -=0A> Shutdown=0A>>>> triggered on=0A>>>> =0A>>>> =0A> :{cassandradevrk1(10.130.202.34):9393}=0A>>>> 2011-08-0= 2 08:43:06,544 ERROR=0A>>>> [me.prettyprint.cassandra.connection.Concurrent= HClientPool] -=0A> Shutdown=0A>>>> complete on=0A>>>> =0A>>>> =0A> :{cassandradevrk1(10.130.202.34):9393}=0A>>>> = 2011-08-02 08:43:06,544 INFO=0A>>>> [me.prettyprint.cassandra.connection.Ca= ssandraHostRetryService] -=0A> Host=0A>>>> detected as down was added to re= try queue:=0A>>>> cassandradevrk1(10.130.202.34):9393=0A>>>> 2011-08-02 08:= 43:06,544 WARN=0A>>>> [me.prettyprint.cassandra.connection.HConnectionManag= er] - Could=0A> not=0A>>>> fullfill request on this host=0A> CassandraClien= t=0A>>>> =0A>> =0A>> =0A>=0A