Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 17544 invoked from network); 17 Nov 2009 02:26:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 17 Nov 2009 02:26:38 -0000 Received: (qmail 86528 invoked by uid 500); 17 Nov 2009 02:26:38 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 86505 invoked by uid 500); 17 Nov 2009 02:26:38 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 86496 invoked by uid 99); 17 Nov 2009 02:26:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 02:26:38 +0000 X-ASF-Spam-Status: No, hits=-2.6 required=5.0 tests=BAYES_00,HTML_MESSAGE X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ikatkov@gmail.com designates 209.85.218.210 as permitted sender) Received: from [209.85.218.210] (HELO mail-bw0-f210.google.com) (209.85.218.210) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 02:26:35 +0000 Received: by bwz2 with SMTP id 2so8522233bwz.20 for ; Mon, 16 Nov 2009 18:26:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type; bh=p5Mqa4JnyqoHa/SiPtio2jkn8zEpowVa0riyl7w3vFA=; b=oBhuLSRBXYnkp2h7ykB/ufHT8N6X7cBZSnA/UHIoeGJeYNCDpjq9KyT0nOVGlDpR/T g1Awdv32cKsVItJmSP3RRcDBfeX8ulaSkP6hpsSOVzgOZOVKGF3FVRbd+xP0BzJlVgpk wmk7z3LdkpPJIgGzlCcPfTTtE/HV7NTUyC5GQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=KH3vu5CDI37F+P3NJikqI8V5FEMU4DV87BKHhUZD/gBZv17SH9OEGl3XRHl8ZnZWNE sDzOhI2kCaenoTjS/S0ZWtxAo2BgS+Wpq6zB5dI1xFYzF2A/35hYI1WK4pRRJ6m/L2Gq 9aQzaE7A39uWqBW7VNQ8Di/EAo9a4obrsYXjQ= MIME-Version: 1.0 Received: by 10.223.144.82 with SMTP id y18mr1291972fau.74.1258424773631; Mon, 16 Nov 2009 18:26:13 -0800 (PST) In-Reply-To: <35bb42690911161812q7ce42990v340fc99b413c2134@mail.gmail.com> References: <35bb42690911092025l109b871exa58ff629d624e299@mail.gmail.com> <35bb42690911101149i18fcc590v1cbc2ba9b2b99356@mail.gmail.com> <35bb42690911101153y3a998431se86a64613f31b030@mail.gmail.com> <35bb42690911160946pb37f763x52666a890ded9a91@mail.gmail.com> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C842D4E4A@GVW0432EXB.americas.hpqcorp.net> <35bb42690911161013y3ee067cao637c189e751fea49@mail.gmail.com> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C842D4E8B@GVW0432EXB.americas.hpqcorp.net> <35bb42690911161812q7ce42990v340fc99b413c2134@mail.gmail.com> From: Igor Katkov Date: Mon, 16 Nov 2009 21:25:53 -0500 Message-ID: <23b1e84e0911161825w6adae420n935f3dbdb52b2325@mail.gmail.com> Subject: Re: Timeout Exception To: cassandra-user@incubator.apache.org, chris@chriswere.com Content-Type: multipart/alternative; boundary=0023545bd7609bdfa0047887d965 --0023545bd7609bdfa0047887d965 Content-Type: text/plain; charset=ISO-8859-1 On most resolvable hardware (for Cassandra) JVM will be running in server mode by default. http://java.sun.com/j2se/1.5.0/docs/guide/vm/server-class.html On Mon, Nov 16, 2009 at 9:12 PM, Chris Were wrote: > Reading more on JVM GC led me to investigate the java -server flag ( > http://stackoverflow.com/questions/198577/real-differences-between-java-server-and-java-client > ) > > From what I can see cassandra's startup scripts don't invoke this mode, or > did I miss it? > > Chris. > > > On Mon, Nov 16, 2009 at 10:33 AM, Freeman, Tim wrote: > >> You'll have to stop the swapping somehow. Maybe you can install more >> memory, maybe you can run Cassandra smaller, maybe you can get some other >> process on the machine to be smaller or on some other machine, maybe you can >> move Cassandra to some other machine with more available physical memory. >> >> >> >> I don't have experience with running Cassandra smaller than the >> recommended size, so one of those options might not work. >> >> >> >> Caching database information in swapped-out pages usually isn't a win. To >> a first approximation, you need an I/O to fetch the swapped-out page, but >> you'd need an I/O anyway to get the information from the database. Swapping >> on modern machines usually isn't a win in general -- Memory got bigger and >> CPU's got faster in the last decade, but disks didn't get much faster. >> >> >> >> Tim Freeman >> Email: tim.freeman@hp.com >> Desk in Palo Alto: (650) 857-2581 >> Home: (408) 774-1298 >> Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and >> Thursday; call my desk instead.) >> >> >> >> *From:* Chris Were [mailto:chris.were@gmail.com] >> *Sent:* Monday, November 16, 2009 10:13 AM >> *To:* cassandra-user@incubator.apache.org >> *Subject:* Re: Timeout Exception >> >> >> >> Hi Tim, >> >> >> >> Thanks for the great pointers. >> >> >> >> si, so are regularly in the 100-2000 range. I'll need to Google more about >> what these mean etc, but are you effectively saying to tell cassandra to use >> less memory? Cassandra is the only Java App running on the server. >> >> >> >> Cheers, >> >> Chris >> >> On Mon, Nov 16, 2009 at 9:59 AM, Freeman, Tim wrote: >> >> I'm running 0.4.1. I used to get timeouts, then I changed my timeout from >> 5 seconds to 30 seconds and I get no more timeouts. The relevant line from >> storage-conf.xml is: >> >> >> >> 30000 >> >> >> >> The maximum latency is often just over 5 seconds in the worst case when I >> fetch thousands of records, so default timeout of 5 seconds happens to be a >> little bit too low for me. My records are ~100Kbytes each. You may get >> different results if your records are much larger or much smaller. >> >> >> >> The other issue I was having a few days ago was that the machine was page >> faulting so garbage collections were taking forever. Some GC's took 20 >> minutes in another Java process. I didn't have verbose:gc turned on in >> Cassandra so I'm not sure what the score was there, but there's little >> reason to expect it to be qualitatively better, since it's pretty random >> which process gets some of its pages swapped out. On a Linux machine, run >> "vmstat 5" when your machine is loaded and if you see numbers greater than 0 >> in the "si" and "so" columns in rows after the first, tell one of your Java >> processes to take less memory. >> >> >> >> Tim Freeman >> Email: tim.freeman@hp.com >> Desk in Palo Alto: (650) 857-2581 >> Home: (408) 774-1298 >> Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and >> Thursday; call my desk instead.) >> >> >> >> *From:* Chris Were [mailto:chris.were@gmail.com] >> *Sent:* Monday, November 16, 2009 9:47 AM >> *To:* Jonathan Ellis >> *Cc:* cassandra-user@incubator.apache.org >> *Subject:* Re: Timeout Exception >> >> >> >> I turned on debug logging for a few days and timeouts happened across >> pretty much all requests. I couldn't see any particular request that was >> consistently the problem. >> >> >> >> After some experimenting it seems that shutting down cassandra and >> restarting resolves the problem. Once it hits the JVM memory limit however, >> the timeouts start again. I have read the page on MemTable thresholds and >> have tried thresholds of 32MB, 64MB and 128MB with no noticeable difference. >> Cassandra is set to use 7GB of memory. I have 12 CF's, however only 6 of >> those have lots of data. >> >> >> >> Cheers, >> >> Chris >> >> On Tue, Nov 10, 2009 at 11:55 AM, Jonathan Ellis >> wrote: >> >> if you're timing out doing a slice on 10 columns w/ 10% cpu used, >> something is broken >> >> is it consistent as to which keys this happens on? try turning on >> debug logging and seeing where the latency is coming from. >> >> >> On Tue, Nov 10, 2009 at 1:53 PM, Chris Were wrote: >> > >> > On Tue, Nov 10, 2009 at 11:50 AM, Jonathan Ellis >> wrote: >> >> >> >> On Tue, Nov 10, 2009 at 1:49 PM, Chris Were >> wrote: >> >> > Maybe... but it's not just multigets, it also happens when retreiving >> >> > one >> >> > row with get_slice. >> >> >> >> how many of the 3M columns are you trying to slice at once? >> > >> > Sorry, I must have mixed up the terminology. >> > There's ~3M keys, but less than 10 columns in each. The get_slice calls >> are >> > to retreive all the columns (10) for a given key. >> >> >> >> >> > > --0023545bd7609bdfa0047887d965 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On most resolvable hardware (for Cassandra) JVM will be running in server m= ode by default.
http://java.sun.com/j2se/1.5.0/docs/guide/vm/server-clas= s.html

On Mon, Nov 16, 2009 at 9:12 PM, Chris Were = <chris.were@gm= ail.com> wrote:
Reading more on JVM GC led me to investigate the java -server flag (http://stackoverflow.com/qu= estions/198577/real-differences-between-java-server-and-java-client)

From what I can see cassandra's startup scripts don= 't invoke this mode, or did I miss it?

Chris.


On Mon, Nov 16, 2009 at 10:33 AM, Freeman, Tim <tim.freeman@hp.com>= ; wrote:

You'll have to stop the swapping somehow.=A0 Maybe you can instal= l more memory, maybe you can run Cassandra smaller, maybe you can get some ot= her process on the machine to be smaller or on some other machine, maybe you ca= n move Cassandra to some other machine with more available physical memory.= =A0

=A0

I don't have experience with running Cassandra smaller than the recommended size, so one of those options might not work.

=A0

Caching database information in swapped-out pages usually isn't a win.=A0 To a first approximation, you need an I/O to fetch the swapped-ou= t page, but you'd need an I/O anyway to get the information from the data= base.=A0 Swapping on modern machines usually isn't a win in general -- Memory go= t bigger and CPU's got faster in the last decade, but disks didn't get much = faster.

=A0

Tim Freeman
Email: tim.freeman@= hp.com
Desk in Palo Alto: (650) 857-2581
Home: (408) 774-1298
Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and Thursday; call my desk instead.)

=A0

From:= Chris Were [mailto:chris.wer= e@gmail.com]
Sent: Monday, November 16, 2009 10:13 AM
To: cassandra-user@incubator.apache.org
Subject: Re: Timeout Exception

=A0

Hi Tim,

=A0

Thanks for the great pointers.

=A0

si, so are regularly in the 100-2000 range. I'll= need to Google more about what these mean etc, but are you effectively saying to te= ll cassandra to use less memory? Cassandra is the only Java App running on the server.

=A0

Cheers,

Chris

On Mon, Nov 16, 2009 at 9:59 AM, Freeman, Tim <tim.freeman@hp.com> wrote:

I'm running 0.4.1.=A0 I used to get timeouts, then I changed my timeout from 5 seconds to 30 seconds and I get = no more timeouts.=A0 The relevant line from storage-conf.xml is:

=A0

=A0 <RpcTimeoutInMillis>30000</RpcTimeoutInMillis>

=A0

The maximum latency is often just over 5 seconds in the worst case when I fetch thousands of records, so default tim= eout of 5 seconds happens to be a little bit too low for me.=A0 My records are ~100Kbytes each.=A0 You may get different results if your records are much larger or much smaller.

=A0

The other issue I was having a few days ago was that the machine was page faulting so garbage collections were taki= ng forever.=A0 Some GC's took 20 minutes in another Java process.=A0 I didn't have verbose:gc turned on in Cassandra so I'm not sure what = the score was there, but there's little reason to expect it to be qualitatively b= etter, since it's pretty random which process gets some of its pages swapped out.=A0 On a Linux machine, run "vmstat 5" when your machine is loaded and if you see numbers greater than 0 in the "si" and "so" columns in rows after the first, tell one of your Java proce= sses to take less memory.

=A0

Tim Freeman
Email: tim.freeman@= hp.com
Desk in Palo Alto: (650) 857-2581
Home: (408) 774-1298
Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and Thursday; call my desk instead.)

=A0

From:= Chris Were [mailto:chri= s.were@gmail.com]
Sent: Monday, November 16, 2009 9:47 AM
To: Jonathan Ellis
Cc: cassandra-user@incubator.apache.org
Subject: Re: Timeout Exception

=A0

I turned on debug logging for a few days and timeouts happened across pretty = much all requests. I couldn't see any particular request that was consistent= ly the problem.

=A0

After some experimenting it seems that shutting down cassandra and restarting resolves the problem. Once it hits the JVM memory limit however, the timeou= ts start again. I have read the page on MemTable thresholds and have tried thresholds of 32MB, 64MB and 128MB with no noticeable difference. Cassandra= is set to use 7GB of memory. I have 12 CF's, however only 6 of those have = lots of data.

=A0

Cheers,

Chris

On Tue, Nov 10, 2009 at 11:55 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

if you're timing out doing a slice on 10 columns w/ 10% cpu used,
something is broken

is it consistent as to which keys this happens on? =A0try turning on
debug logging and seeing where the latency is coming from.


On Tue, Nov 10, 2009 at 1:53 PM, Chris Were <chris.were@gmail.com> wrote:
>
> On Tue, Nov 10, 2009 at 11:50 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> On Tue, Nov 10, 2009 at 1:49 PM, Chris Were <chris.were@gmail.com> wrote:
>> > Maybe... but it's not just multigets, it also happens whe= n retreiving
>> > one
>> > row with get_slice.
>>
>> how many of the 3M columns are you trying to slice at once?
>
> Sorry, I must have mixed up the terminology.
> There's ~3M keys, but less than 10 columns in each. The get_slice = calls are
> to retreive all the columns (10) for a given key.

=A0

=A0



--0023545bd7609bdfa0047887d965--