Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.216.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=JSfC0JAtIP8ngxTPScrDFebczyt8GnIUtAHYNSMs+Qx+2tYlSjlnwquxsUl2c/QqGv
         +vsUeuNj4A321xwwJxzG0PdfvCoddE3b5I8cw4UzK/ACnYLJ/2zeKW+0JVfC/R/cAj1T
         HREJGJn5OSlkuDqfqVIYijFDGBAQ0NzdCGUCk=
MIME-Version: 1.0
In-Reply-To: <4C9D2015.40002@pdf.com>
References: <4C940305.7070905@pdf.com>
	<AANLkTi=px8iT04FjePgjOrutSRoc8FuNu-BHVDTpKDtW@mail.gmail.com>
	<4C9805FD.6000802@pdf.com>
	<AANLkTin8+zVsx++32SFR18opXU35eN0b0M=Xmcm6PDeH@mail.gmail.com>
	<4C9D2015.40002@pdf.com>
Date: Fri, 24 Sep 2010 15:42:20 -0700
Message-ID: <AANLkTinEckmcRF4J1jZRsJQEmpv_efbaU19w-nK80KGF@mail.gmail.com>
Subject: Re: 0.7 memory usage problem
From: Jonathan Ellis <jbellis@gmail.com>
To: user <user@cassandra.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

looks like you're OOMing trying to compact a very large row. solution:
smaller rows, or larger heap.

On Fri, Sep 24, 2010 at 3:03 PM, Alaa Zubaidi <alaa.zubaidi@pdf.com> wrote:
> =A0Nothing is working, after disabling swap entirely, the heap is not
> exhausted but Cassandra crashed with out of memory error.
> I even slowed down the reading and writing, tried access mode=3Dstandard
> nothing is working.
> am I using Cassandra wrong or is it Windows?
>
> Any help is appreciated?
> Alaa
>
> On 9/21/2010 6:56 PM, Jonathan Ellis wrote:
>>
>> Disabling swap entirely is usually the easiest fix, yes.
>>
>> On Mon, Sep 20, 2010 at 8:10 PM, Alaa Zubaidi<alaa.zubaidi@pdf.com>
>> =A0wrote:
>>>
>>> =A0Thanks Peter,
>>> I decreased the heap size, it did not help, however, it delayed the
>>> problem.
>>> I noticed that its swapping, so, do you think that I should set windows
>>> to
>>> Not to swap?
>>>
>>> Do you think its related to this issue?
>>> https://issues.apache.org/jira/browse/CASSANDRA-1014
>>>
>>> Thanks,
>>> Alaa
>>>
>>> On 9/18/2010 7:04 AM, Peter Schuller wrote:
>>>>>
>>>>> I see a spike in heap memory usage on Node 2 where it goes from aroun=
d
>>>>> 1G
>>>>> to
>>>>> 6GB (max) in less than an hour, and then goes our of memory.
>>>>> There are some errors in the log file that are reported by other
>>>>> people,
>>>>> but
>>>>> I don't think that these errors are the reason, because it use to
>>>>> happen
>>>>> even before I have seen them.
>>>>>
>>>>> Can someone help me understand whats going on??
>>>>
>>>> Only partially:
>>>>
>>>> 54 seconds ParNew GC:s are wild and crazy:
>>>>
>>>> =A0 =A0INFO [GC inspection] 2010-09-17 14:53:59,403 GCInspector.java (=
line
>>>> 129) GC for ParNew: 54095 ms, 53297952 reclaimed leaving 4712568360
>>>> used; max is 6563430400
>>>>
>>>> Is the machine swapping?
>>>>
>>>> I noticed there is hinted hand-off activity going on. Maybe that is a
>>>> result of nodes dropping in and out due to swapping. In any case, you
>>>> definitely don't want to have the machine swapping to death. I'm not
>>>> sure what the best way is to avoid this on Windows, other than
>>>> decreasing heap size.
>>>>
>>>> The repeated exceptions in your log aren't normal as far as I know. I
>>>> don't remember, but IIRC the UTF-8 encoding issues can be caused by
>>>> changing partitioner after inserting data (but I'm not sure).
>>>>
>>>> With respect to memory use, you don't seem to be inserting so much
>>>> data for bloom filters and sstable index samples to be a problem.
>>>> Memtable flushing could cause problems if they're happening too slowly
>>>> (maybe plausible with swapping) - except that the stage statistics
>>>> don't indicate there are lots of memtables in memory waiting to be
>>>> flushed, so that shouldn't be it.
>>>>
>>>> Hinted handoff maybe, but I don't remember whether hinted handoff has
>>>> the potential to accumulate data in RAM (no time to check now).
>>>>
>>>> Regardless, I'd recommend fixing any swapping issues you have before
>>>> trying to draw conclusions about performance. And you don't want those
>>>> exceptions in your logs.
>>>>
>>> --
>>> Alaa Zubaidi
>>> PDF Solutions, Inc.
>>> 333 West San Carlos Street, Suite 700
>>> San Jose, CA 95110 =A0USA
>>> Tel: 408-283-5639 (or 408-280-7900 x5639)
>>> fax: 408-938-6479
>>> email: alaa.zubaidi@pdf.com
>>>
>>>
>>>
>>
>>
>
> --
> Alaa Zubaidi
> PDF Solutions, Inc.
> 333 West San Carlos Street, Suite 700
> San Jose, CA 95110 =A0USA
> Tel: 408-283-5639 (or 408-280-7900 x5639)
> fax: 408-938-6479
> email: alaa.zubaidi@pdf.com
>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com