Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 82941 invoked from network); 22 Dec 2009 15:28:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 22 Dec 2009 15:28:49 -0000 Received: (qmail 367 invoked by uid 500); 22 Dec 2009 15:28:48 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 356 invoked by uid 500); 22 Dec 2009 15:28:48 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 347 invoked by uid 99); 22 Dec 2009 15:28:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Dec 2009 15:28:48 +0000 X-ASF-Spam-Status: No, hits=0.1 required=5.0 tests=AWL,BAYES_00,HTML_MESSAGE X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.220.216] (HELO mail-fx0-f216.google.com) (209.85.220.216) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Dec 2009 15:28:38 +0000 Received: by fxm8 with SMTP id 8so5876442fxm.27 for ; Tue, 22 Dec 2009 07:28:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.223.15.86 with SMTP id j22mr11792766faa.47.1261495694609; Tue, 22 Dec 2009 07:28:14 -0800 (PST) In-Reply-To: References: <468b21170912220147q4c3b0726p8cb7a1fa81c0ec74@mail.gmail.com> <468b21170912220317k1ea6a8aagba24d847161fca42@mail.gmail.com> <468b21170912220422u4ce138f2pe112f10b57b626fc@mail.gmail.com> <468b21170912220442k681b8980w1fd3f0b40207481a@mail.gmail.com> <468b21170912220601q756d0b59rb041dff7b8e4d1e6@mail.gmail.com> Date: Tue, 22 Dec 2009 17:28:14 +0200 Message-ID: Subject: Re: MultiThread Client problem with thrift From: Richard Grossman To: cassandra-user@incubator.apache.org Content-Type: multipart/alternative; boundary=001517447f0cc34d43047b52dab6 --001517447f0cc34d43047b52dab6 Content-Type: text/plain; charset=ISO-8859-1 Ok I got this, of course this problem can be solved but lowered the load of the server by whatever you want : connection pool or Thread management less agressive. It's not my goal I would like to keep the server under high pressure. Finally I managed to find a solution by lowered the TIME_WAIT connection status on my machine. need to make the adjustement to every machine. It's system related on mac os x it's here : http://www.brianp.net/2008/10/03/changing-the-length-of-the-time_wait-state-on-mac-os-x/ on linux it's easier to find. Thanks On Tue, Dec 22, 2009 at 4:46 PM, matthew hawthorne wrote: > On Tue, Dec 22, 2009 at 9:10 AM, Ran Tavory wrote: > > Not at expert in this field, but I think what you want is use a > connection > > pool and NOT close the connections - reuse them. Only idle connections > are > > released after, say 1sec. Also, with a connection pool it's easy > > to throttle the application, you can tell the pool to block if all 50 > > connections, or how many you define are allowed. > > I did something very similar to this. A difference in my approach is > that I did not release idle connections after a specific time period, > instead I performed a liveness check on each connection after > obtaining it from the pool, like this: > > // get client connection from pool > Cassandra.Client client = > > try { > client.getInputProtocol().getTransport().flush(); > } catch (TTransportException e) { > // connection is invalid, obtain new connection > } > > It seemed to work during my testing, not sure if the thrift specifics > are 100% correct (meaning I'm not sure if the catch block will work > for all situations involving stale or expired connections). > > -matt > > > > On Tue, Dec 22, 2009 at 4:01 PM, Richard Grossman > > wrote: > >> > >> So I can't use it. > >> But I've make my own connection pool. This are not fix nothing because > the > >> problem is lower than even java. In fact the socket is closed and java > >> consider it as close but the system keep the Socket in the state > TIME_WAIT. > >> Then the port used is actually still in use. > >> So my question is that is there people that manage to open multiple > >> connection and ride off the TIME_WAIT. No matter in which language PHP > or > >> Python etc... > >> Thanks > >> On Tue, Dec 22, 2009 at 2:55 PM, Ran Tavory wrote: > >>> > >>> I don't have a 0.5.0-beta2 version, no. It's not too difficult to add > it, > >>> but I haven't done so myself, I'm using 0.4.2 > >>> > >>> On Tue, Dec 22, 2009 at 2:42 PM, Richard Grossman > > >>> wrote: > >>>> > >>>> Yes of course but do you have updated to cassandra 0.5.0-beta2 ? > >>>> > >>>> On Tue, Dec 22, 2009 at 2:30 PM, Ran Tavory wrote: > >>>>> > >>>>> Would connection pooling work for you? > >>>>> This Java client http://code.google.com/p/cassandra-java-client/ has > >>>>> connection pooling. > >>>>> I haven't put the client under stress yet so I can't testify, but > this > >>>>> may be a good solution for you > >>>>> > >>>>> On Tue, Dec 22, 2009 at 2:22 PM, Richard Grossman < > richiesgr@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>> I agree it's solve my problem but can give a bigger one. > >>>>>> The problem is I can't succeed to prevent opening a lot of > connection > >>>>>> > >>>>>> On Tue, Dec 22, 2009 at 1:51 PM, Jaakko > >>>>>> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> I don't know the particulars of java implementation, but if it > works > >>>>>>> the same way as Unix native socket API, then I would not recommend > >>>>>>> setting linger to zero. > >>>>>>> > >>>>>>> SO_LINGER option with zero value will cause TCP connection to be > >>>>>>> aborted immediately as soon as the socket is closed. That is, (1) > >>>>>>> remaining data in the send buffer will be discarded, (2) no proper > >>>>>>> disconnect handshake and (3) receiving end will get TCP reset. > >>>>>>> > >>>>>>> Sure this will avoid TIME_WAIT state, but TIME_WAIT is our friend > and > >>>>>>> is there to avoid packets from old connection being delivered to > new > >>>>>>> incarnation of the connection. Instead of avoiding the state, the > >>>>>>> application should be changed so that TIME_WAIT will not be a > >>>>>>> problem. > >>>>>>> How many open files you can see when the exception happens? Might > be > >>>>>>> that you're out of file descriptors. > >>>>>>> > >>>>>>> -Jaakko > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Dec 22, 2009 at 8:17 PM, Richard Grossman > >>>>>>> wrote: > >>>>>>> > Hi > >>>>>>> > To all is interesting I've found a solution seems not recommended > >>>>>>> > but > >>>>>>> > working. > >>>>>>> > When opening a Socket set this: > >>>>>>> > tSocket.getSocket().setReuseAddress(true); > >>>>>>> > tSocket.getSocket().setSoLinger(true, 0); > >>>>>>> > it's prevent to have a lot of connection TIME_WAIT state but not > >>>>>>> > recommended. > >>>>>>> > > >>>>>> > >>>>> > >>>> > >>> > >> > > > > > --001517447f0cc34d43047b52dab6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Ok I got this, of course this problem can be solved but lo= wered the load of the server by whatever you want : connection pool or Thre= ad management less agressive. It's not my goal I would like to keep the= server under high pressure.

Finally I managed to find a solution by lowered the TIME_WAIT connectio= n status on my machine. need to make the adjustement to every machine. It&#= 39;s system related on mac os x it's here : h= ttp://www.brianp.net/2008/10/03/changing-the-length-of-the-time_wait-state-= on-mac-os-x/

on linux it's easier to find.

Thanks

On Tue, Dec 22, 2009 at 4:46 PM, matthew hawthorne <mhawthorne@gmail.com&g= t; wrote:
On Tue, Dec 22, 2009 at 9:10 AM, Ran Tavory <rantav@gmail.com> wrote:
> Not at expert in this field, but I think what you want is use a connec= tion
> pool and NOT close the connections - reuse them. Only idle connections= are
> released after, say 1sec. Also, with a connection pool it's easy > to=A0throttle=A0the application, you can tell the pool to block if all= 50
> connections, or how many you define are allowed.

I did something very similar to this. =A0A difference in my approach = is
that I did not release idle connections after a specific time period,
instead I performed a liveness check on each connection after
obtaining it from the pool, like this:

// get client connection from pool
Cassandra.Client client =3D

try {
=A0client.getInputProtocol().getTransport().flush();
} catch (TTransportException e) {
=A0// connection is invalid, obtain new connection
}

It seemed to work during my testing, not sure if the thrift specifics
are 100% correct (meaning I'm not sure if the catch block will work
for all situations involving stale or expired connections).

-matt


> On Tue, Dec 22, 2009 at 4:01 PM, Richard Grossman <richiesgr@gmail.com>
> wrote:
>>
>> So I can't use it.
>> But I've make my own connection pool. This are not fix nothing= because the
>> problem is lower than even java. In fact the socket is closed and = java
>> consider it as close but the system keep the Socket in the =A0stat= e TIME_WAIT.
>> Then the port used is actually still in use.
>> So my question is that is there people that manage to open multipl= e
>> connection and ride off the TIME_WAIT. No matter in which language= PHP or
>> Python etc...
>> Thanks
>> On Tue, Dec 22, 2009 at 2:55 PM, Ran Tavory <rantav@gmail.com> wrote:
>>>
>>> I don't have a 0.5.0-beta2 version, no. It's not too d= ifficult to add it,
>>> but I haven't done so myself, I'm using 0.4.2
>>>
>>> On Tue, Dec 22, 2009 at 2:42 PM, Richard Grossman <richiesgr@gmail.com>
>>> wrote:
>>>>
>>>> Yes of course but do you have updated to cassandra 0.5.0-b= eta2 ?
>>>>
>>>> On Tue, Dec 22, 2009 at 2:30 PM, Ran Tavory <rantav@gmail.com> wrote:
>>>>>
>>>>> Would connection pooling work for you?
>>>>> This Java client=A0http://code.google.com/p/cassand= ra-java-client/ has
>>>>> connection pooling.
>>>>> I haven't put the client under stress yet so I can= 't testify, but this
>>>>> may be a good solution for you
>>>>>
>>>>> On Tue, Dec 22, 2009 at 2:22 PM, Richard Grossman <= richiesgr@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> I agree it's solve my problem but can give a b= igger one.
>>>>>> The problem is I can't succeed to prevent open= ing a lot of connection
>>>>>>
>>>>>> On Tue, Dec 22, 2009 at 1:51 PM, Jaakko <rosvopaallikko@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I don't know the particulars of java imple= mentation, but if it works
>>>>>>> the same way as Unix native socket API, then I= would not recommend
>>>>>>> setting linger to zero.
>>>>>>>
>>>>>>> SO_LINGER option with zero value will cause TC= P connection to be
>>>>>>> aborted immediately as soon as the socket is c= losed. That is, (1)
>>>>>>> remaining data in the send buffer will be disc= arded, (2) no proper
>>>>>>> disconnect handshake and (3) receiving end wil= l get TCP reset.
>>>>>>>
>>>>>>> Sure this will avoid TIME_WAIT state, but TIME= _WAIT is our friend and
>>>>>>> is there to avoid packets from old connection = being delivered to new
>>>>>>> incarnation of the connection. Instead of avoi= ding the state, the
>>>>>>> application should be changed so that TIME_WAI= T will not be a
>>>>>>> problem.
>>>>>>> How many open files you can see when the excep= tion happens? Might be
>>>>>>> that you're out of file descriptors.
>>>>>>>
>>>>>>> -Jaakko
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Dec 22, 2009 at 8:17 PM, Richard Gross= man
>>>>>>> <ric= hiesgr@gmail.com> wrote:
>>>>>>> > Hi
>>>>>>> > To all is=A0interesting I've found a = solution seems not recommended
>>>>>>> > but
>>>>>>> > working.
>>>>>>> > When opening a Socket set this:
>>>>>>> > =A0 =A0tSocket.getSocket().setReuseAddres= s(true);
>>>>>>> > =A0 =A0tSocket.getSocket().setSoLinger(tr= ue, 0);
>>>>>>> > it's prevent to have a lot of connect= ion TIME_WAIT state but not
>>>>>>> > recommended.
>>>>>>> >
>>>>>>
>>>>>
>>>>
>>>
>>
>
>

--001517447f0cc34d43047b52dab6--