Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 942286350 for ; Thu, 4 Aug 2011 13:06:32 +0000 (UTC) Received: (qmail 93707 invoked by uid 500); 4 Aug 2011 13:06:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 93606 invoked by uid 500); 4 Aug 2011 13:06:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 93598 invoked by uid 99); 4 Aug 2011 13:06:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Aug 2011 13:06:28 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=SPF_NEUTRAL,TO_NO_BRKTS_PCNT X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [173.201.192.162] (HELO p3plwbeout13-02.prod.phx3.secureserver.net) (173.201.192.162) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 04 Aug 2011 13:06:23 +0000 Received: (qmail 27888 invoked from network); 4 Aug 2011 13:06:02 -0000 Received: from unknown (HELO localhost) (10.6.247.3) by p3plwbeout13-02.prod.phx3.secureserver.net with SMTP; 4 Aug 2011 13:06:02 -0000 Received: (qmail 16624 invoked by uid 99); 4 Aug 2011 13:06:02 -0000 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" X-Originating-IP: 178.197.254.3 User-Agent: Web-Based Email 5.5.14 Message-Id: <20110804060559.050558854cd8e3604bbb7dcb1d74d208.9ced5c3d31.wbe@email13.secureserver.net> From: "Tim Snyder" To: user@cassandra.apache.org Subject: RE: Trying to find the problem with a broken pipe Date: Thu, 04 Aug 2011 06:05:59 -0700 Mime-Version: 1.0 I am getting the same problem (Broken Pipe) on a loader program, after=0Aab= out 8 million read, write pairs. I am pushing serialized objects into=0Aa c= olumn with the program, the object it seems to be doing it on is much=0Alar= ger than the prior objects, so I am wondering if it is possibly a=0Acolumn = size streaming issue through the thrift api? I am using Cassandra=0A0.8.0 a= nd Hector 0.8.0-1=0A=0ATim=0A=0A-------- Original Message --------=0ASubjec= t: Re: Trying to find the problem with a broken pipe=0AFrom: Anthony Ikeda = =0ADate: Tue, August 02, 2011 10:43 pm=0ATo: u= ser@cassandra.apache.org=0A=0A> Very interesting. After the second host goe= s down do you see=0A> "me.prettyprint.hector.api.exceptions.HectorException= : All host pools=0A > marked down. Retry burden pushed out to client"?=0A= =0ANo, the last message is:=0A2011-08-02 08:43:06,561 INFO=0A[me.prettyprin= t.cassandra.connection.HConnectionManager] - Client=0ACassandraClient released to inactive or dead=0Apool. Closing.> Does you= r client recover after a period of time?=0A=0A=0A =0AThe application seems = to be fine for now but my concern is the=0Aconnection pooling as well - I m= ean do we have one pool or multiple?=0AI'll post to the Hector user group a= bout the pooling because the=0Aincident seems so isolated. We also have our= infrastructure team looking=0Ainto the communication between the applicati= on server and the cassandra=0Anodes.=0A=0A=0ASo far it's still a mystery.= =0A=0A=0A=0A=0A=0AOn Tue, Aug 2, 2011 at 1:25 PM, Jim Ancona wrote:=0A On Tue, Aug 2, 2011 at 6:13 PM, Anthony Ikeda=0A wrote:=0A =0A> The link (which I may be misreading)= =0A > is=0Ahttp://groups.google.com/group/hector-users/browse_thread/thread= /8d7004b6f85a0f2e=0A =0A =0AI hadn't found that one, but I doubt that our i= ssue is related to that.=0A =0A=0A > It's only started happening today and = happened on 2 occassions (8:43=0Aand=0A > 10:21) performing the same functi= on (querying a column family).=0A > It seems to be trying to access a conne= ction on one of the servers=0A > The client accesses the first node:=0A >= =0A > 2011-08-02 08:43:06,541 ERROR=0A > [me.prettyprint.cassandra.connecti= on.HThriftClient] - Could not flush=0A > transport (to be expected if the p= ool is shutting down) in close for=0Aclient:=0A > CassandraClient=0A > org.apache.thrift.transport.TTransportException:=0Ajav= a.net.SocketException:=0A > Broken pipe=0A >=0A > ...=0A > 2011-08-02 08:43= :06,544 WARN=0A > [me.prettyprint.cassandra.connection.HConnectionManager] = - Could not=0A > fullfill request on this host=0ACassandraClient=0A >=0A > ...=0A >=0A > 2011-08-02 08:43:06,543 ERROR=0A > [= me.prettyprint.cassandra.connection.HConnectionManager] - MARK HOST=0AAS DO= WN=0A > TRIGGERED for host cassandradevrk1(10.130.202.34):9393=0A > 2011-08= -02 08:43:06,543 ERROR=0A > [me.prettyprint.cassandra.connection.HConnectio= nManager] - Pool state=0Aon=0A > shutdown:=0A >=0A:{cassandradevrk1(10.130.202.34):9393};=0A > IsActive?: true; A= ctive: 1; Blocked: 0; Idle: 15; NumBeforeExhausted:=0A49=0A > 2011-08-02 08= :43:06,543 ERROR=0A > [me.prettyprint.cassandra.connection.ConcurrentHClien= tPool] -=0AShutdown=0A > triggered on=0A >=0A:{cassandradevrk1(10.130.202.34):9393}=0A > 2011-08-02 08:43:06,544 = ERROR=0A > [me.prettyprint.cassandra.connection.ConcurrentHClientPool] -=0A= Shutdown=0A > complete on=0A >=0A:{cas= sandradevrk1(10.130.202.34):9393}=0A > 2011-08-02 08:43:06,544 INFO=0A > [m= e.prettyprint.cassandra.connection.CassandraHostRetryService] -=0AHost=0A >= detected as down was added to retry queue:=0A > cassandradevrk1(10.130.202= .34):9393=0A > 2011-08-02 08:43:06,544 WARN=0A > [me.prettyprint.cassandra.= connection.HConnectionManager] - Could not=0A > fullfill request on this ho= st CassandraClient<=0A > cassandradevrk1:9393-33>=0A > 2011-08-02 08:43:06,= 544 WARN=0A > [me.prettyprint.cassandra.connection.HConnectionManager] - Ex= ception:=0A > me.prettyprint.hector.api.exceptions.HectorTransportException= :=0A > org.apache.thrift.transport.TTransportException:=0Ajava.net.SocketEx= ception:=0A > Connection reset=0A >=0A >=0A > Then it appears to try the se= cond node and fails:=0A >=0A > 2011-08-02 08:43:06,556 INFO=0A > [me.pretty= print.cassandra.connection.HConnectionManager] - Client=0A > CassandraClien= t released to inactive or dead=0Apool.=0A > Closin= g.=0A > 2011-08-02 08:43:06,557 ERROR=0A > [me.prettyprint.cassandra.connec= tion.HThriftClient] - Could not flush=0A > transport (to be expected if the= pool is shutting down) in close for=0Aclient:=0A > CassandraClient=0A >=0A > org.apache.thrift.transport.TTransportException= :=0Ajava.net.SocketException:=0A > Broken pipe=0A >=0A > 2011-08-02 08:43:0= 6,558 ERROR=0A > [me.prettyprint.cassandra.connection.HConnectionManager] -= MARK HOST=0AAS DOWN=0A > TRIGGERED for host cassandradevrk2(10.130.202.35)= :9393=0A > 2011-08-02 08:43:06,559 ERROR=0A > [me.prettyprint.cassandra.con= nection.HConnectionManager] - Pool state=0Aon=0A > shutdown:=0A >=0A:{cassandradevrk2(10.130.202.35):9393};=0A > = IsActive?: true; Active: 1; Blocked: 0; Idle: 15; NumBeforeExhausted:=0A49= =0A > 2011-08-02 08:43:06,559 ERROR=0A > [me.prettyprint.cassandra.connecti= on.ConcurrentHClientPool] -=0AShutdown=0A > triggered on=0A >=0A:{cassandradevrk2(10.130.202.35):9393}=0A > 2011-= 08-02 08:43:06,559 ERROR=0A > [me.prettyprint.cassandra.connection.Concurre= ntHClientPool] -=0AShutdown=0A > complete on=0A >=0A:{cassandradevrk2(10.130.202.35):9393}=0A > 2011-08-02 08:43:= 06,559 INFO=0A > [me.prettyprint.cassandra.connection.CassandraHostRetrySer= vice] -=0AHost=0A > detected as down was added to retry queue:=0A > cassand= radevrk2(10.130.202.35):9393=0A > 2011-08-02 08:43:06,560 WARN=0A > [me.pre= ttyprint.cassandra.connection.HConnectionManager] - Could not=0A > fullfill= request on this host=0ACassandraClient=0A > 2011-= 08-02 08:43:06,560 WARN=0A > [me.prettyprint.cassandra.connection.HConnecti= onManager] - Exception:=0A > me.prettyprint.hector.api.exceptions.HectorTra= nsportException:=0A > org.apache.thrift.transport.TTransportException:=0Aja= va.net.SocketException:=0A > Connection reset=0A =0A =0A=0AVery interesting= . After the second host goes down do you see=0A "me.prettyprint.hector.api.= exceptions.HectorException: All host pools=0A marked down. Retry burden pus= hed out to client"?=0A =0A Does your client recover after a period of time?= =0A =0A >=0A > The process is the same at 10:21.=0A > Are the exceptions re= lated to any external events (e.g. node=0A > restarts, network issues...)?= =0A > Not that I'm aware, unless there are firewall timeouts between the=0A= > application and the node servers. Let me find out. The cassandra log=0Af= iles=0A > have no errors reported.=0A > What versions of Hector and Cassand= ra are you running?=0A > Cassandra 0.8.1, Hector 0.8.0-1=0A =0A =0AOur issu= e is occurring with Cassandra 0.7.8 and Hector 0.7-30. We plan=0A to deploy= Hector 0.7-31 this week and to turn on useSocketKeepalive.=0A Are you usin= g that? We're also using tcpdump to capture packets when=0A failures occur = to see if there are anomalies in the network traffic.=0A =0A Jim=0A =0A=0A = =0A >=0A >=0A >=0A > On Tue, Aug 2, 2011 at 10:37 AM, Jim Ancona =0Awrote:=0A >>=0A >> On Tue, Aug 2, 2011 at 4:36 PM, Anthony I= keda=0A >> wrote:=0A >> > I'm not sure if thi= s is a problem with Hector or with Cassandra.=0A >> > We seem to be seeing = broken pipe issues with our connections on=0Athe=0A >> > client=0A >> > sid= e (Exception below). A bit of googling finds possibly a problem=0Awith=0A >= > > the=0A >> > amount of data we are trying to store, although I'm certain= our=0Adatasets=0A >> > are=0A >> > not all that large.=0A >>=0A >> I'm not= sure what you're referring to here. Large requests could=0Alead=0A >> to t= imeouts, but that's not what you're seeing here. Could you link=0Ato=0A >> = the page you're referencing?=0A >>=0A >> > A nodetool ring command doesn't = seem to present any downed nodes:=0A >> > Address DC Rack = Status State Load=0A >> > Owns=0A >> > Token=0A >> >=0A >> > = 153951716904446304929228999025275230571=0A >> > 10.130.202.34 datacenter= 1 rack1 Up Normal 470.74 KB=0A >> > 79.19% 11853820084840445976= 3384037192174096102=0A >> > 10.130.202.35 datacenter1 rack1 Up = Normal 483.63 KB=0A >> > 20.81% 153951716904446304929228999025275230571= =0A >> >=0A >> > There are no errors in the cassandra server logs.=0A >> >= =0A >> > Are there any particular timeouts on connections that we need to= =0Abe=0A >> > aware=0A >> > of? Or perhaps configure on the Cassandra nodes= ? Is this purely=0Aand=0A >> > issue=0A >> > with the Hector API configurat= ion?=0A >>=0A >> There is a server side timeout (rpc_timeout_in_ms in cassa= ndra.yaml)=0A >> and a Hector client-side timeout=0A >> (CassandraHostConfi= gurator.cassandraThriftSocketTimeout). But again,=0A >> the "Broken pipe" e= rror is not a timeout, it indicates that=0Asomething=0A >> happened to the = underlying network socket. For example you will see=0A >> those when a serv= er node is restarted.=0A >>=0A >> Some questions that might help troublesho= ot this:=0A >> How often are these occurring?=0A >> Does this affect both n= odes in the cluster or just one?=0A >> Are the exceptions related to any ex= ternal events (e.g. node=0Arestarts,=0A >> network issues...)?=0A >> What v= ersions of Hector and Cassandra are you running?=0A >>=0A >> Keep in mind t= hat failures like this will normally be retried by=0A >> Hector, resulting = in no loss of data. For that reason, I think that=0A >> exception is logged= as a warning in the newest Hector versions.=0A >>=0A >> We've seen somethi= ng similar, but more catastrophic because it=0Aaffects=0A >> connectivity t= o the entire cluster, not just a single node. See this=0A >> post for more = details: http://goo.gl/hrgkw So far we haven't=0A >> identified the cause.= =0A >>=0A >> Jim=0A >>=0A >> > Anthony=0A >> >=0A >> > 2011-08-02 08:43:06,= 541 ERROR=0A >> > [me.prettyprint.cassandra.connection.HThriftClient] - Cou= ld not=0Aflush=0A >> > transport (to be expected if the pool is shutting do= wn) in close=0Afor=0A >> > client:=0A >> > CassandraClient=0A >> > org.apache.thrift.transport.TTransportException:=0A >> > j= ava.net.SocketException:=0A >> > Broken pipe=0A >> > at=0A >> >=0A >>= >=0Aorg.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTranspor= t.java:147)=0A >> > at=0A >> >=0A >> >=0Aorg.apache.thrift.transport.= TFramedTransport.flush(TFramedTransport.java:156)=0A >> > at=0A >> >= =0A >> >=0Ame.prettyprint.cassandra.connection.HThriftClient.close(HThriftC= lient.java:85)=0A >> > at=0A >> >=0A >> >=0Ame.prettyprint.cassandra.= connection.HConnectionManager.operateWithFailover(HConnectionManager.java:2= 32)=0A >> > at=0A >> >=0A >> >=0Ame.prettyprint.cassandra.service.Key= spaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131)=0A >> > = at=0A >> >=0A >> >=0Ame.prettyprint.cassandra.service.KeyspaceService= Impl.getSlice(KeyspaceServiceImpl.java:289)=0A >> > at=0A >> >=0A >> = >=0Ame.prettyprint.cassandra.model.thrift.ThriftSliceQuery$1.doInKeyspace(T= hriftSliceQuery.java:53)=0A >> > at=0A >> >=0A >> >=0Ame.prettyprint.= cassandra.model.thrift.ThriftSliceQuery$1.doInKeyspace(ThriftSliceQuery.jav= a:49)=0A >> > at=0A >> >=0A >> >=0Ame.prettyprint.cassandra.model.Key= spaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.jav= a:20)=0A >> > at=0A >> >=0A >> >=0Ame.prettyprint.cassandra.model.Exe= cutingKeyspace.doExecute(ExecutingKeyspace.java:85)=0A >> > at=0A >> = >=0A >> >=0Ame.prettyprint.cassandra.model.thrift.ThriftSliceQuery.execute(= ThriftSliceQuery.java:48)=0A >> > at=0A >> >=0A >> >=0Acom.wsgc.servi= ces.registry.persistenceservice.impl.cassandra.strategy.read.StandardFindRe= gistryPersistenceStrategy.findRegistryByProfileId(StandardFindRegistryPersi= stenceStrategy.java:237)=0A >> > at=0A >> >=0A >> >=0Acom.wsgc.servic= es.registry.persistenceservice.impl.cassandra.strategy.read.StandardFindReg= istryPersistenceStrategy.execute(StandardFindRegistryPersistenceStrategy.ja= va:277)=0A >> > at=0A >> >=0A >> >=0Acom.wsgc.services.registry.regis= tryservice.impl.service.StandardRegistryService.getRegistriesByProfileId(St= andardRegistryService.java:327)=0A >> > at=0A >> >=0A >> >=0Acom.wsgc= .services.registry.webapp.impl.RegistryServicesController.getRegistriesByPr= ofileId(RegistryServicesController.java:247)=0A >> > at sun.reflect.N= ativeMethodAccessorImpl.invoke0(Native=0AMethod)=0A >> > at=0A >> >= =0A >> >=0Asun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor= Impl.java:39)=0A >> > at=0A >> >=0A >> >=0Asun.reflect.DelegatingMeth= odAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)=0A >> > a= t java.lang.reflect.Method.invoke(Method.java:597)=0A >> > at=0A >> >= =0A >> >=0Aorg.springframework.web.bind.annotation.support.HandlerMethodInv= oker.invokeHandlerMethod(HandlerMethodInvoker.java:175)=0A >> > at=0A= >> >=0A >> >=0Aorg.springframework.web.servlet.mvc.annotation.AnnotationMe= thodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:= 421)=0A >> > at=0A >> >=0A >> >=0Aorg.springframework.web.servlet.mvc= .annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAd= apter.java:409)=0A >> > at=0A >> >=0A >> >=0Aorg.springframework.web.= servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)=0A >> > = at=0A >> >=0A >> >=0Aorg.springframework.web.servlet.DispatcherServlet.= doService(DispatcherServlet.java:719)=0A >> > at=0A >> >=0A >> >=0Aor= g.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServ= let.java:644)=0A >> > at=0A >> >=0A >> >=0Aorg.springframework.web.se= rvlet.FrameworkServlet.doGet(FrameworkServlet.java:549)=0A >> > at=0A= javax.servlet.http.HttpServlet.service(HttpServlet.java:617)=0A >> > = at=0Ajavax.servlet.http.HttpServlet.service(HttpServlet.java:717)=0A >> > = at=0A >> >=0A >> >=0Aorg.apache.catalina.core.ApplicationFilterChain.i= nternalDoFilter(ApplicationFilterChain.java:290)=0A >> > at=0A >> >= =0A >> >=0Aorg.apache.catalina.core.ApplicationFilterChain.doFilter(Applica= tionFilterChain.java:206)=0A >> > at=0A >> >=0A >> >=0Aorg.springfram= ework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFi= lter.java:77)=0A >> > at=0A >> >=0A >> >=0Aorg.springframework.web.fi= lter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)=0A >> > = at=0A >> >=0A >> >=0Aorg.apache.catalina.core.ApplicationFilterChain.in= ternalDoFilter(ApplicationFilterChain.java:235)=0A >> > at=0A >> >=0A= >> >=0Aorg.apache.catalina.core.ApplicationFilterChain.doFilter(Applicatio= nFilterChain.java:206)=0A >> > at=0A >> >=0A >> >=0Aorg.apache.catali= na.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)=0A >> > = at=0A >> >=0A >> >=0Aorg.apache.catalina.core.StandardContextValve.in= voke(StandardContextValve.java:191)=0A >> > at=0A >> >=0A >> >=0Aorg.= apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.ja= va:563)=0A >> > at=0A >> >=0A >> >=0Aorg.apache.catalina.core.Standar= dHostValve.invoke(StandardHostValve.java:127)=0A >> > at=0A >> >=0A >= > >=0Aorg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j= ava:102)=0A >> > at=0A >> >=0A >> >=0Aorg.apache.catalina.core.Standa= rdEngineValve.invoke(StandardEngineValve.java:109)=0A >> > at=0A >> >= =0A >> >=0Aorg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapte= r.java:298)=0A >> > at=0A >> >=0Aorg.apache.jk.server.JkCoyoteHandler= .invoke(JkCoyoteHandler.java:190)=0A >> > at=0A >> >=0Aorg.apache.jk.= common.HandlerRequest.invoke(HandlerRequest.java:291)=0A >> > at=0A >= > > org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)=0A >>= > at=0A >> >=0A >> >=0Aorg.apache.jk.common.ChannelSocket.processCon= nection(ChannelSocket.java:703)=0A >> > at=0A >> >=0A >> >=0Aorg.apac= he.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)= =0A >> > at=0A >> >=0A >> >=0Aorg.apache.tomcat.util.threads.ThreadPo= ol$ControlRunnable.run(ThreadPool.java:690)=0A >> > at java.lang.Thre= ad.run(Thread.java:662)=0A >> > Caused by: java.net.SocketException: Broken= pipe=0A >> > at java.net.SocketOutputStream.socketWrite0(Native Meth= od)=0A >> > at=0A >> >=0Ajava.net.SocketOutputStream.socketWrite(Sock= etOutputStream.java:92)=0A >> > at=0Ajava.net.SocketOutputStream.writ= e(SocketOutputStream.java:136)=0A >> > at=0A >> >=0A >> >=0Aorg.apach= e.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)=0A= >> > ... 47 more=0A >> > 2011-08-02 08:43:06,543 ERROR=0A >> > [me.p= rettyprint.cassandra.connection.HConnectionManager] - MARK=0AHOST AS=0A >> = > DOWN=0A >> > TRIGGERED for host cassandradevrk1(10.130.202.34):9393=0A >>= > 2011-08-02 08:43:06,543 ERROR=0A >> > [me.prettyprint.cassandra.connecti= on.HConnectionManager] - Pool=0Astate on=0A >> > shutdown:=0A >> >=0A >> >= =0A:{cassandradevrk1(10.130.202.34):93= 93};=0A >> > IsActive?: true; Active: 1; Blocked: 0; Idle: 15;=0ANumBeforeE= xhausted: 49=0A >> > 2011-08-02 08:43:06,543 ERROR=0A >> > [me.prettyprint.= cassandra.connection.ConcurrentHClientPool] -=0AShutdown=0A >> > triggered = on=0A >> >=0A >> >=0A:{cassandradevrk1= (10.130.202.34):9393}=0A >> > 2011-08-02 08:43:06,544 ERROR=0A >> > [me.pre= ttyprint.cassandra.connection.ConcurrentHClientPool] -=0AShutdown=0A >> > c= omplete on=0A >> >=0A >> >=0A:{cassand= radevrk1(10.130.202.34):9393}=0A >> > 2011-08-02 08:43:06,544 INFO=0A >> > = [me.prettyprint.cassandra.connection.CassandraHostRetryService] -=0AHost=0A= >> > detected as down was added to retry queue:=0A >> > cassandradevrk1(10= .130.202.34):9393=0A >> > 2011-08-02 08:43:06,544 WARN=0A >> > [me.prettypr= int.cassandra.connection.HConnectionManager] - Could=0Anot=0A >> > fullfill= request on this host=0ACassandraClient=0A >> >=0A= >=0A >=0A