cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Two questions : Server crash during compaction and UnavailableException
Date Tue, 03 Aug 2010 12:16:59 GMT
If you have a crash log you should post at least the header rather
than playing 20 questions with us.  But if it's not OOM then it's
likely to be a bug in the JVM, so upgrading is probably your best
option.

On Tue, Aug 3, 2010 at 3:49 AM, Ilun Ahn <prozect.mail@gmail.com> wrote:
> No, I don't think the direct cause is out of heap space. It didn't left any
> heap dump file with the option -XX:+HeapDumpOnOutOfMemoryError.
> My system.log for the last minute is as follows(many GC occurs):
> INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,254
> HintedHandOffManager.java (line 153) Started hinted handoff for endPoint
> /10.25.32.36
> INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,255
> HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows to
> endpoint /10.25.32.36
> INFO [GC inspection] 2010-08-02 20:34:01,919 GCInspector.java (line 110) GC
> for ParNew: 269 ms, 11161808 reclaimed leaving 5068544312 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:34:14,985 GCInspector.java (line 110) GC
> for ParNew: 208 ms, 12326736 reclaimed leaving 4044195008 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:34:15,986 GCInspector.java (line 110) GC
> for ParNew: 208 ms, 12283112 reclaimed leaving 2005777224 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:35:45,834 GCInspector.java (line 110) GC
> for ParNew: 229 ms, 13074080 reclaimed leaving 5374833480 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:35:46,836 GCInspector.java (line 110) GC
> for ParNew: 203 ms, 12529824 reclaimed leaving 5321733432 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:37:01,624 GCInspector.java (line 110) GC
> for ParNew: 206 ms, 11029656 reclaimed leaving 4473650352 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:38:19,064 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 4501 ms, 1057548400 reclaimed leaving 2461458096
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:41:20,446 GCInspector.java (line 110) GC
> for ParNew: 218 ms, 15072720 reclaimed leaving 5345683640 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:41:23,453 GCInspector.java (line 110) GC
> for ParNew: 234 ms, 16818048 reclaimed leaving 3937902088 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:42:15,229 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 8015 ms, 739534984 reclaimed leaving 3550138024
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:42:42,444 GCInspector.java (line 110) GC
> for ParNew: 203 ms, 14218928 reclaimed leaving 4398967608 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:42:43,565 GCInspector.java (line 110) GC
> for ParNew: 203 ms, 12274600 reclaimed leaving 1989854648 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:43:01,801 GCInspector.java (line 110) GC
> for ParNew: 212 ms, 10183184 reclaimed leaving 2337034168 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:44:01,215 GCInspector.java (line 110) GC
> for ParNew: 218 ms, 10402368 reclaimed leaving 4334140184 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:44:35,623 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 4424 ms, 3101007888 reclaimed leaving 2459621048
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:45:32,089 GCInspector.java (line 110) GC
> for ParNew: 227 ms, 27109720 reclaimed leaving 5410486832 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:45:35,095 GCInspector.java (line 110) GC
> for ParNew: 203 ms, 28235832 reclaimed leaving 3580093424 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:46:01,203 GCInspector.java (line 110) GC
> for ParNew: 257 ms, 12257744 reclaimed leaving 3469012312 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:46:51,060 GCInspector.java (line 110) GC
> for ParNew: 222 ms, 18473064 reclaimed leaving 5320004640 used; max is
> 8719630336
>
>
>
>
>
> and the JRE crash log show that current thread is COMPACTION-POOL:1.
>
>
>
> ps. Sorry for unnecessary message. my mistake.
>
>
>
>
> 2010/8/3 Ilun Ahn <prozect.mail@gmail.com>
>>
>>
>> 2010/8/2 Peter Schuller <peter.schuller@infidyne.com>
>>>
>>> > First, Cassandra suddenly dies during compaction. Java core dump says
>>> > that
>>> > the last thread run was  "COMPACTION-POOL:1".
>>> > I suspect that my business logic could lead size of columns in a column
>>> > family per a row to be greater than two gigabytes. (but i couldn't
>>> > confirm
>>> > it yet)
>>>
>>> Are you running out of memory (java heap)? If you're running cassandra
>>> with default options, it will be running with
>>> -XX:+HeapDumpOnOutOfMemoryError
>>>
>>> Have you checked the cassandra system.log for garbage collection
>>> messages? What is in the last minute or two of logs?
>>>
>>> --
>>> / Peter Schuller
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message