Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 940192EFE for ; Wed, 27 Apr 2011 13:56:44 +0000 (UTC) Received: (qmail 51972 invoked by uid 500); 27 Apr 2011 13:56:42 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 51953 invoked by uid 500); 27 Apr 2011 13:56:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 51945 invoked by uid 99); 27 Apr 2011 13:56:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 13:56:42 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of colin.wook.park@gmail.com designates 209.85.212.44 as permitted sender) Received: from [209.85.212.44] (HELO mail-vw0-f44.google.com) (209.85.212.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 13:56:36 +0000 Received: by vws12 with SMTP id 12so1538498vws.31 for ; Wed, 27 Apr 2011 06:56:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=svMovgdaGOCeA29x5fd1ce3llA/NvgDV1cS7ItlUlVA=; b=k49Dn+a0GPKZA+hQnZOMb6einESc8MyGJ/OJxzZ8hLKV3RCFY4SaJw8pPo/mWxIumo 1SyQT/dNUjfqRNI8RIb/dBxmm5TRJDNP9vT3pc6TwCP9IPM3oBssmUw/9HrPNXkXYZ68 YB+C63NDbXM/5YO6XW3Xwn82HS7auYHeMFwJk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=kaE3h+afaEiSjgTJZ3ki8Hc6C0gPuGw4G2RU0NTj8XoT7KR/rJ7h7EyfNlpvKM+op5 tzan6npONhE0rFJflE2rQy1cJ8KIeEUEKh4ZPtpmaEVaxnPPWPt/mAphYI9hk1WHWTYA 6X1D87wR0rNB6GJZunEgnsMBDkAAqBoOtorh4= MIME-Version: 1.0 Received: by 10.52.18.14 with SMTP id s14mr261725vdd.164.1303912575114; Wed, 27 Apr 2011 06:56:15 -0700 (PDT) Received: by 10.220.61.79 with HTTP; Wed, 27 Apr 2011 06:56:15 -0700 (PDT) Date: Wed, 27 Apr 2011 22:56:15 +0900 Message-ID: Subject: Thrift client thread is locked. (TSocket is initialized with _timeout) From: =?EUC-KR?B?udq/67/t?= To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf3054a12fdba04804a1e6cd28 --20cf3054a12fdba04804a1e6cd28 Content-Type: text/plain; charset=ISO-8859-1 Hello. I have a problem with thrift client socket. Server (0.7.4) - 6 nodes cluster - reboot 1 node(EC2 instance) suddenly. Client (hector-core-0.7.0-22, libthrift-0.5) - hector's cassandraThriftSocketTimeout option is set to 30000ms and *It initiated TSocket with same timeout(Socket.setSoTimeout(timeout_) value.* I'm getting locked thread of cassandra thrift client after rebooting an arbitrary node. stack below does not disappear for hours. It seems Socket.setSoTimeout() is not working and I can not figure out the reason. "New I/O server worker #1-43" prio=10 tid=0x00007f36488ea000 nid=0x2884 runnable [0x00007f363b634000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.javaservice.lwst.SocketWrapper$InputStreamWrap.read(Unknown Source) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) - locked <0x00000006c09caaf8> (a java.io.BufferedInputStream) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:906) at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:890) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:93) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:88) at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:89) at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:142) at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129) at me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:100) at me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(KeyspaceServiceImpl.java:106) at me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.java:149) at me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.java:146) at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:65) at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:146) ........... at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:116) ........... at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:522) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:506) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:443) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Best reguards. Colin. --20cf3054a12fdba04804a1e6cd28 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Hello. I have a problem with thrift client socket.


Server (0.7.4)
- 6 nodes cluster
- reboot= 1 node(EC2 instance) suddenly.


Client (hector-core-0.7.0-22, libthrift-0.5)
- hector&= #39;s cassandraThriftSocketTimeout option is set to 30000ms and It= initiated TSocket with same timeout(Socket.setSoTimeout(timeout_) value.



I'm getting locked th= read of cassandra thrift client after rebooting an arbitrary node.
stack below does not disappear for hours.
It seems Socket.setSo= Timeout() is not working and I can not figure out the reason.



"New I/O server worker #1-= 43" prio=3D10 tid=3D0x00007f36488ea000 nid=3D0x2884 runnable [0x00007f= 363b634000]
=A0=A0 java.lang.Thread.State: RUNNABLE
=A0= =A0 =A0 =A0 =A0at java.net.SocketInputStream.socketRead0(Native Method)
=A0=A0 =A0 =A0 =A0at java.net.SocketInputStream.read(SocketInputStream= .java:129)
=A0=A0 =A0 =A0 =A0at com.javaservice.lwst.SocketWrappe= r$InputStreamWrap.read(Unknown Source)
=A0=A0 =A0 =A0 =A0at java.= io.BufferedInputStream.fill(BufferedInputStream.java:218)
=A0=A0 =A0 =A0 =A0at java.io.BufferedInputStream.read1(BufferedInputSt= ream.java:258)
=A0=A0 =A0 =A0 =A0at java.io.BufferedInputStream.r= ead(BufferedInputStream.java:317)
=A0=A0 =A0 =A0 =A0- locked <= 0x00000006c09caaf8> (a java.io.BufferedInputStream)
=A0=A0 =A0 =A0 =A0at org.apache.thrift.transport.TIOStreamTransport.re= ad(TIOStreamTransport.java:127)
=A0=A0 =A0 =A0 =A0at org.apache.t= hrift.transport.TTransport.readAll(TTransport.java:84)
=A0=A0 =A0= =A0 =A0at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTr= ansport.java:129)
=A0=A0 =A0 =A0 =A0at org.apache.thrift.transport.TFramedTransport.read= (TFramedTransport.java:101)
=A0=A0 =A0 =A0 =A0at org.apache.thrif= t.transport.TTransport.readAll(TTransport.java:84)
=A0=A0 =A0 =A0= =A0at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.j= ava:378)
=A0=A0 =A0 =A0 =A0at org.apache.thrift.protocol.TBinaryProtocol.readI3= 2(TBinaryProtocol.java:297)
=A0=A0 =A0 =A0 =A0at org.apache.thrif= t.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
=
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.thrift.Cassandra$Client.recv= _batch_mutate(Cassandra.java:906)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.thrift.Cassandra$Client.batc= h_mutate(Cassandra.java:890)
=A0=A0 =A0 =A0 =A0at me.prettyprint.= cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:93= )
=A0=A0 =A0 =A0 =A0at me.prettyprint.cassandra.service.KeyspaceS= erviceImpl$1.execute(KeyspaceServiceImpl.java:88)
=A0=A0 =A0 =A0 =A0at me.prettyprint.cassandra.service.Operation.execut= eAndSetResult(Operation.java:89)
=A0=A0 =A0 =A0 =A0at me.prettypr= int.cassandra.connection.HConnectionManager.operateWithFailover(HConnection= Manager.java:142)
=A0=A0 =A0 =A0 =A0at me.prettyprint.cassandra.service.KeyspaceServiceI= mpl.operateWithFailover(KeyspaceServiceImpl.java:129)
=A0=A0 =A0 = =A0 =A0at me.prettyprint.cassandra.service.KeyspaceServiceImpl.batchMutate(= KeyspaceServiceImpl.java:100)
=A0=A0 =A0 =A0 =A0at me.prettyprint.cassandra.service.KeyspaceServiceI= mpl.batchMutate(KeyspaceServiceImpl.java:106)
=A0=A0 =A0 =A0 =A0a= t me.prettyprint.cassandra.model.MutatorImpl$2.doInKeyspace(MutatorImpl.jav= a:149)
=A0=A0 =A0 =A0 =A0at me.prettyprint.cassandra.model.MutatorImpl$2.doInKeysp= ace(MutatorImpl.java:146)
=A0=A0 =A0 =A0 =A0at me.prettyprint.cas= sandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOpera= tionCallback.java:20)
=A0=A0 =A0 =A0 =A0at me.prettyprint.cassandra.model.ExecutingKeyspace.= doExecute(ExecutingKeyspace.java:65)
=A0=A0 =A0 =A0 =A0at me.pret= typrint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:146)
=A0=A0 =A0 =A0 =A0...........
=A0=A0 =A0 =A0 =A0at org.jboss.netty.channel.SimpleChannelUpstreamHand= ler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
=A0=A0 = =A0 =A0 =A0at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(D= efaultChannelPipeline.java:545)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.channel.DefaultChannelPipeline$De= faultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.handler.codec.http.HttpChunkAg= gregator.messageReceived(HttpChunkAggregator.java:116)
=A0=A0 =A0 =A0 =A0...........
=A0=A0 =A0 =A0 =A0at org.jboss= .netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUps= treamHandler.java:80)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.channe= l.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.channel.DefaultChannelPipeline$De= faultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.channel.Channels.fireMessageRe= ceived(Channels.java:302)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.handler.codec.replay.ReplayingDec= oder.unfoldAndfireMessageReceived(ReplayingDecoder.java:522)
=A0= =A0 =A0 =A0 =A0at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cal= lDecode(ReplayingDecoder.java:506)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.handler.codec.replay.ReplayingDec= oder.messageReceived(ReplayingDecoder.java:443)
=A0=A0 =A0 =A0 = =A0at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(S= impleChannelUpstreamHandler.java:80)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.channel.DefaultChannelPipeline.se= ndUpstream(DefaultChannelPipeline.java:545)
=A0=A0 =A0 =A0 =A0at = org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelP= ipeline.java:540)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.channel.Channels.fireMessageRecei= ved(Channels.java:274)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.chann= el.Channels.fireMessageReceived(Channels.java:261)
=A0=A0 =A0 =A0= =A0at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349= )
=A0=A0 =A0 =A0 =A0at org.jboss.netty.channel.socket.nio.NioWorker.proc= essSelectedKeys(NioWorker.java:280)
=A0=A0 =A0 =A0 =A0at org.jbos= s.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
=A0= =A0 =A0 =A0 =A0at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRen= amingRunnable.java:108)
=A0=A0 =A0 =A0 =A0at org.jboss.netty.util.internal.DeadLockProofWorker= $1.run(DeadLockProofWorker.java:44)
=A0=A0 =A0 =A0 =A0at java.uti= l.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)=
=A0=A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Wor= ker.run(ThreadPoolExecutor.java:908)
=A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662)
=



Best reguards.
= Colin.
--20cf3054a12fdba04804a1e6cd28--