Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 84568 invoked from network); 22 Dec 2009 19:35:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 22 Dec 2009 19:35:18 -0000 Received: (qmail 28741 invoked by uid 500); 22 Dec 2009 19:35:16 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 28731 invoked by uid 500); 22 Dec 2009 19:35:16 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 28722 invoked by uid 99); 22 Dec 2009 19:35:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Dec 2009 19:35:15 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of richiesgr@gmail.com designates 209.85.220.216 as permitted sender) Received: from [209.85.220.216] (HELO mail-fx0-f216.google.com) (209.85.220.216) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Dec 2009 19:35:07 +0000 Received: by fxm8 with SMTP id 8so6130581fxm.27 for ; Tue, 22 Dec 2009 11:34:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=kcvscs9cd0zbWRgFDsgGck8oTybZwNY9eIRBUqGYTbM=; b=t56DwQvTE1AimwaT+XNepZ9GuDdSeOMoyg8jdIYjEMjoUEr/3DZaqA/3uSXbEJCJkq cZ3cNUBrWQemlK7kE83YGMrtqgYzv03cbyyW/iibQSxMTd2qj68c1e65xTxYSZ7A9Xml Pfs0rH23TzkvA3rP7urbscoC5pbmcqPKw/CAo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=mwhZzd/LbOnvmMaJ50iQovlY6riCD1A7nuVG95ttrSYScyLPA4yb/Eyi41sBIhivXY xPBQSf72QZE0TvmMe0LjZZLVaXydlGn3qo+tiH3BzC+lTMJGRidDdpn94kfKsK8lDpYz XMM4l4R7oZeAY1yIC2oWVt8dMDV32Cpgx9ojU= MIME-Version: 1.0 Received: by 10.239.141.143 with SMTP id c15mr930161hba.47.1261510486653; Tue, 22 Dec 2009 11:34:46 -0800 (PST) In-Reply-To: <766B5A29D28DA442AB229AAEE2AFC44507D7B91508@SEAMBX.corp.real.com> References: <468b21170912220147q4c3b0726p8cb7a1fa81c0ec74@mail.gmail.com> <468b21170912220422u4ce138f2pe112f10b57b626fc@mail.gmail.com> <468b21170912220442k681b8980w1fd3f0b40207481a@mail.gmail.com> <468b21170912220601q756d0b59rb041dff7b8e4d1e6@mail.gmail.com> <766B5A29D28DA442AB229AAEE2AFC44507D7B91508@SEAMBX.corp.real.com> Date: Tue, 22 Dec 2009 21:34:46 +0200 Message-ID: <468b21170912221134p43ca71e7rfe71834a87e12789@mail.gmail.com> Subject: Re: MultiThread Client problem with thrift From: Richard Grossman To: cassandra-user@incubator.apache.org Content-Type: multipart/alternative; boundary=001485f6d4846ffd8e047b564ce7 X-Virus-Checked: Checked by ClamAV on apache.org --001485f6d4846ffd8e047b564ce7 Content-Type: text/plain; charset=ISO-8859-1 When I try to reuse socket to make multiple thrift operation in multithread I get always exception On Tue, Dec 22, 2009 at 9:28 PM, Brian Burruss wrote: > i don't close the connection to a server unless i get exceptions. and when > i close the connection i try a new server in the cluster just to keep the > connections spread across the cluster. > > should i be closing them? if the connection is closed by client or server > i'll just reconnect. > > ________________________________________ > From: Ran Tavory [rantav@gmail.com] > Sent: Tuesday, December 22, 2009 9:50 AM > To: cassandra-user@incubator.apache.org > Subject: Re: MultiThread Client problem with thrift > > I don't know how keeping the connections open affects at scale. I suppose > if you have 10 to 1 ratio of cassandra clients to cassandra server (probably > a typical ratio) then you may be using too much server resources > > On Tue, Dec 22, 2009 at 4:46 PM, matthew hawthorne > wrote: > On Tue, Dec 22, 2009 at 9:10 AM, Ran Tavory rantav@gmail.com>> wrote: > > Not at expert in this field, but I think what you want is use a > connection > > pool and NOT close the connections - reuse them. Only idle connections > are > > released after, say 1sec. Also, with a connection pool it's easy > > to throttle the application, you can tell the pool to block if all 50 > > connections, or how many you define are allowed. > > I did something very similar to this. A difference in my approach is > that I did not release idle connections after a specific time period, > instead I performed a liveness check on each connection after > obtaining it from the pool, like this: > > // get client connection from pool > Cassandra.Client client = > > try { > client.getInputProtocol().getTransport().flush(); > } catch (TTransportException e) { > // connection is invalid, obtain new connection > } > > It seemed to work during my testing, not sure if the thrift specifics > are 100% correct (meaning I'm not sure if the catch block will work > for all situations involving stale or expired connections). > > -matt > > > > On Tue, Dec 22, 2009 at 4:01 PM, Richard Grossman > > > wrote: > >> > >> So I can't use it. > >> But I've make my own connection pool. This are not fix nothing because > the > >> problem is lower than even java. In fact the socket is closed and java > >> consider it as close but the system keep the Socket in the state > TIME_WAIT. > >> Then the port used is actually still in use. > >> So my question is that is there people that manage to open multiple > >> connection and ride off the TIME_WAIT. No matter in which language PHP > or > >> Python etc... > >> Thanks > >> On Tue, Dec 22, 2009 at 2:55 PM, Ran Tavory rantav@gmail.com>> wrote: > >>> > >>> I don't have a 0.5.0-beta2 version, no. It's not too difficult to add > it, > >>> but I haven't done so myself, I'm using 0.4.2 > >>> > >>> On Tue, Dec 22, 2009 at 2:42 PM, Richard Grossman > > >>> wrote: > >>>> > >>>> Yes of course but do you have updated to cassandra 0.5.0-beta2 ? > >>>> > >>>> On Tue, Dec 22, 2009 at 2:30 PM, Ran Tavory rantav@gmail.com>> wrote: > >>>>> > >>>>> Would connection pooling work for you? > >>>>> This Java client http://code.google.com/p/cassandra-java-client/ has > >>>>> connection pooling. > >>>>> I haven't put the client under stress yet so I can't testify, but > this > >>>>> may be a good solution for you > >>>>> > >>>>> On Tue, Dec 22, 2009 at 2:22 PM, Richard Grossman < > richiesgr@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>> I agree it's solve my problem but can give a bigger one. > >>>>>> The problem is I can't succeed to prevent opening a lot of > connection > >>>>>> > >>>>>> On Tue, Dec 22, 2009 at 1:51 PM, Jaakko > > >>>>>> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> I don't know the particulars of java implementation, but if it > works > >>>>>>> the same way as Unix native socket API, then I would not recommend > >>>>>>> setting linger to zero. > >>>>>>> > >>>>>>> SO_LINGER option with zero value will cause TCP connection to be > >>>>>>> aborted immediately as soon as the socket is closed. That is, (1) > >>>>>>> remaining data in the send buffer will be discarded, (2) no proper > >>>>>>> disconnect handshake and (3) receiving end will get TCP reset. > >>>>>>> > >>>>>>> Sure this will avoid TIME_WAIT state, but TIME_WAIT is our friend > and > >>>>>>> is there to avoid packets from old connection being delivered to > new > >>>>>>> incarnation of the connection. Instead of avoiding the state, the > >>>>>>> application should be changed so that TIME_WAIT will not be a > >>>>>>> problem. > >>>>>>> How many open files you can see when the exception happens? Might > be > >>>>>>> that you're out of file descriptors. > >>>>>>> > >>>>>>> -Jaakko > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Dec 22, 2009 at 8:17 PM, Richard Grossman > >>>>>>> > wrote: > >>>>>>> > Hi > >>>>>>> > To all is interesting I've found a solution seems not recommended > >>>>>>> > but > >>>>>>> > working. > >>>>>>> > When opening a Socket set this: > >>>>>>> > tSocket.getSocket().setReuseAddress(true); > >>>>>>> > tSocket.getSocket().setSoLinger(true, 0); > >>>>>>> > it's prevent to have a lot of connection TIME_WAIT state but not > >>>>>>> > recommended. > >>>>>>> > > >>>>>> > >>>>> > >>>> > >>> > >> > > > > > > --001485f6d4846ffd8e047b564ce7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
When I try to reuse socket to make multiple thrift operati= on in multithread I get always exception

= On Tue, Dec 22, 2009 at 9:28 PM, Brian Burruss <bburruss@real.com> wrote:
i don't close= the connection to a server unless i get exceptions. =A0and when i close th= e connection i try a new server in the cluster just to keep the connections= spread across the cluster.

should i be closing them? =A0if the connection is closed by client or serve= r i'll just reconnect.

________________________________________
From: Ran Tavory [rantav@gmail.com]=
Sent: Tuesday, December 22, 2009 9:50 AM
To: cassandra-user@i= ncubator.apache.org
Subject: Re: MultiThread Client problem with thrift

I don't know how keeping the connections open affects at scale. I suppo= se if you have 10 to 1 ratio of cassandra clients to cassandra server (prob= ably a typical ratio) then you may be using too much server resources

On Tue, Dec 22, 2009 at 4:46 PM, matthew hawthorne = <mhawthorne@gmail.com<mai= lto:mhawthorne@gmail.com>>= ; wrote:
On Tue, Dec 22, 2009 at 9:10 AM, Ran Tavory <rantav@gmail.com<mailto:rantav@gmail.com>> wrote:
> Not at expert in this field, but I think what you want is use a connec= tion
> pool and NOT close the connections - reuse them. Only idle connections= are
> released after, say 1sec. Also, with a connection pool it's easy > to throttle the application, you can tell the pool to block if all 50<= br> > connections, or how many you define are allowed.

I did something very similar to this. =A0A difference in my approach is
that I did not release idle connections after a specific time period,
instead I performed a liveness check on each connection after
obtaining it from the pool, like this:

// get client connection from pool
Cassandra.Client client =3D

try {
=A0client.getInputProtocol().getTransport().flush();
} catch (TTransportException e) {
=A0// connection is invalid, obtain new connection
}

It seemed to work during my testing, not sure if the thrift specifics
are 100% correct (meaning I'm not sure if the catch block will work
for all situations involving stale or expired connections).

-matt


> On Tue, Dec 22, 2009 at 4:01 PM, Richard Grossman <richiesgr@gmail.com<mailto:richiesgr@gmail.com>>
> wrote:
>>
>> So I can't use it.
>> But I've make my own connection pool. This are not fix nothing= because the
>> problem is lower than even java. In fact the socket is closed and = java
>> consider it as close but the system keep the Socket in the =A0stat= e TIME_WAIT.
>> Then the port used is actually still in use.
>> So my question is that is there people that manage to open multipl= e
>> connection and ride off the TIME_WAIT. No matter in which language= PHP or
>> Python etc...
>> Thanks
>> On Tue, Dec 22, 2009 at 2:55 PM, Ran Tavor= y <rantav@gmail.com<mailto:rantav@gmail.com>> wrote:
>>>
>>> I don't have a 0.5.0-beta2 version, no. It's not too d= ifficult to add it,
>>> but I haven't done so myself, I'm using 0.4.2
>>>
>>> On Tue, Dec 22, 2009 at 2:42 PM, Richard Grossman <richiesgr@gmail.com<mailto:richiesgr@gmail.com>>
>>> wrote:
>>>>
>>>> Yes of course but do you have updated to cassandra 0.5.0-b= eta2 ?
>>>>
>>>> On Tue, Dec 22, 2009 at 2:30 PM, R= an Tavory <rantav@gmail.com<m= ailto:rantav@gmail.com>> wrot= e:
>>>>>
>>>>> Would connection pooling work for you?
>>>>> This Java client http://code.google.com/p/cassandra= -java-client/ has
>>>>> connection pooling.
>>>>> I haven't put the client under stress yet so I can= 't testify, but this
>>>>> may be a good solution for you
>>>>>
>>>>> On Tue, Dec 22, 2009 at 2:22 PM, Richard Grossma= n <richiesgr@gmail.com<mai= lto:richiesgr@gmail.com>><= br>
>>>>> wrote:
>>>>>>
>>>>>> I agree it's solve my problem but can give a b= igger one.
>>>>>> The problem is I can't succeed to prevent open= ing a lot of connection
>>>>>>
>>>>>> On Tue, Dec 22, 2009 at 1:51 PM, Jaakko <= rosvopaallikko@gmail.com<= ;mailto:rosvopaallikko@gmail.co= m>>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I don't know the particulars of java imple= mentation, but if it works
>>>>>>> the same way as Unix native socket API, then I= would not recommend
>>>>>>> setting linger to zero.
>>>>>>>
>>>>>>> SO_LINGER option with zero value will cause TC= P connection to be
>>>>>>> aborted immediately as soon as the socket is c= losed. That is, (1)
>>>>>>> remaining data in the send buffer will be disc= arded, (2) no proper
>>>>>>> disconnect handshake and (3) receiving end wil= l get TCP reset.
>>>>>>>
>>>>>>> Sure this will avoid TIME_WAIT state, but TIME= _WAIT is our friend and
>>>>>>> is there to avoid packets from old connection = being delivered to new
>>>>>>> incarnation of the connection. Instead of avoi= ding the state, the
>>>>>>> application should be changed so that TIME_WAI= T will not be a
>>>>>>> problem.
>>>>>>> How many open files you can see when the excep= tion happens? Might be
>>>>>>> that you're out of file descriptors.
>>>>>>>
>>>>>>> -Jaakko
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Dec 22, 2009 at 8:17 PM, Richard Gross= man
>>>>>>> <richiesgr@gmail.com<mailto:richiesgr@gmail.com>> wrote: >>>>>>> > Hi
>>>>>>> > To all is interesting I've found a so= lution seems not recommended
>>>>>>> > but
>>>>>>> > working.
>>>>>>> > When opening a Socket set this:
>>>>>>> > =A0 =A0tSocket.getSocket().setReuseAddres= s(true);
>>>>>>> > =A0 =A0tSocket.getSocket().setSoLinger(tr= ue, 0);
>>>>>>> > it's prevent to have a lot of connect= ion TIME_WAIT state but not
>>>>>>> > recommended.
>>>>>>> >
>>>>>>
>>>>>
>>>>
>>>
>>
>
>


--001485f6d4846ffd8e047b564ce7--