cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erich Nachbar <er...@nachbar.biz>
Subject Re: read latency creaping up
Date Tue, 15 Dec 2009 17:17:17 GMT
One thing I have seen in the past when benchmarking is that the client
GC settings are equally important.
When the client GC pauses, to collect all those small short lived
objects, it can make the server look bad.

I'd use similar GC settings (with a smaller heap) then I'd use on the
server (e.g. parallel GC).

On Mon, Dec 14, 2009 at 6:03 PM, Brian Burruss <bburruss@real.com> wrote:
> I plan to investigate that next.  There isn't a straightforward "disconnect" that I
could easily see.
>
> Ian Holsman <ian@holsman.net> wrote:
>
>
> can you make it so that the client restarts the connection every 30m or so ?
> It could be an issue in thrift or something with long-lived connections.
>
> On Dec 15, 2009, at 10:16 AM, Brian Burruss wrote:
>
>> i agree.  i don't know anything about thrift, and i don't know how it keeps connections
open or manages resources from a client or server perspective, but this situation suggests
that maybe killing the clients is forcing the server to free something.
>>
>> how's that sound :)
>>
>> ________________________________________
>> From: Jonathan Ellis [jbellis@gmail.com]
>> Sent: Monday, December 14, 2009 3:12 PM
>> To: cassandra-user@incubator.apache.org
>> Subject: Re: read latency creaping up
>>
>> hmm, me neither
>>
>> but, I can't think how restarting the client would, either :)
>>
>> On Mon, Dec 14, 2009 at 4:59 PM, Brian Burruss <bburruss@real.com> wrote:
>>> Well not sure how that would affect he latency as reported by the Cassandra server
using nodeprobe cfstats
>>>
>>> Jonathan Ellis <jbellis@gmail.com> wrote:
>>>
>>>
>>> possibly the clients are running into memory pressure?
>>>
>>> On Mon, Dec 14, 2009 at 4:27 PM, Brian Burruss <bburruss@real.com> wrote:
>>>> thx, i'm actually the "B. Todd Burruss" in that thread ..  we changed our
email system and well now, i'm just Brian .. long story.
>>>>
>>>> anyway, in this case it isn't compaction pendings as i can kill the clients
and immediately restart and the latency is back to a reasonable number.  i'm still investigating.
>>>>
>>>> thx!
>>>> ________________________________________
>>>> From: Eric Evans [eevans@rackspace.com]
>>>> Sent: Monday, December 14, 2009 8:23 AM
>>>> To: cassandra-user@incubator.apache.org
>>>> Subject: RE: read latency creaping up
>>>>
>>>> On Sun, 2009-12-13 at 13:18 -0800, Brian Burruss wrote:
>>>>> if this isn't a known issue, lemme do some more investigating.  my
>>>>> test client becomes "more random" with reads as time progresses, so
>>>>> possibly this is what causes the latency issue.  however, all that
>>>>> being said, the performance really becomes bad after a while.
>>>>
>>>> Have a look at the following thread:
>>>>
>>>> http://thread.gmane.org/gmane.comp.db.cassandra.user/1402
>>>>
>>>>
>>>> --
>>>> Eric Evans
>>>> eevans@rackspace.com
>>>>
>>>>
>>>
>
> --
> Ian Holsman
> Ian@Holsman.net
>
>
>
>

Mime
View raw message