cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yatong Zhang <bluefl...@gmail.com>
Subject Re: hs_err_pid3013.log, out of memory?
Date Thu, 18 Sep 2014 01:35:59 GMT
@Chris Lohfink I have 16G memory per node, all the other settings are
default

@J. Ryan Earl I am not sure. I am using the default settings.

But I've found out it might be because some settings in '/etc/sysctl.conf'.
I am still testing it

On Wed, Sep 17, 2014 at 11:46 AM, J. Ryan Earl <oss@jryanearl.us> wrote:

> Are you using JNA?  Did you adjust your memlock limit?
>
> On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink <clohfink@blackbirdit.com>
> wrote:
>
>> How much memory does your system have? How much memory is system
>> utilizing before starting Cassandra (use command free)? What are the heap
>> setting it tries to use?
>>
>> Chris
>>
>> On Sep 15, 2014, at 8:16 PM, Yatong Zhang <blueflycn@gmail.com> wrote:
>>
>> It's during the startup. I tried to upgrade cassandra from 2.0.7 to
>> 2.0.10, but looks like cassandra could not start again. Also I found the
>> following log at '/var/log/messages':
>>
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more
>>> than 120 seconds.
>>> Sep 16 09:06:59 storage6 kernel:      Tainted: G
>>> --------------- H  2.6.32-431.el6.x86_64 #1
>>> Sep 16 09:06:59 storage6 kernel: "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Sep 16 09:06:59 storage6 kernel: java          D 0000000000000003     0
>>> 4971      1 0x00000080
>>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082
>>> ffffffff81ed4ff0 ffff8803b4f01540
>>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370
>>> ffff88042b591ca0 ffff8803b4f01540
>>> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8
>>> 000000000000fbc8 ffff8803b4f01af8
>>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ?
>>> exit_robust_list+0x90/0x160
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>]
>>> do_group_exit+0x58/0xd0
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>]
>>> get_signal_to_deliver+0x1f6/0x460
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>]
>>> do_signal+0x75/0x800
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ?
>>> wake_up_new_task+0xd9/0x130
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ?
>>> do_fork+0x13d/0x480
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ?
>>> sys_futex+0x7b/0x170
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>]
>>> do_notify_resume+0x90/0xc0
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>]
>>> int_signal+0x12/0x17
>>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more
>>> than 120 seconds.
>>> Sep 16 09:06:59 storage6 kernel:      Tainted: G
>>> --------------- H  2.6.32-431.el6.x86_64 #1
>>> Sep 16 09:06:59 storage6 kernel: "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Sep 16 09:06:59 storage6 kernel: java          D 0000000000000000     0
>>> 4972      1 0x00000080
>>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082
>>> ffffffff81ed6d78 ffff8803b4cf1500
>>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370
>>> ffff8803b4d7fca0 ffff8803b4cf1500
>>> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8
>>> 000000000000fbc8 ffff8803b4cf1ab8
>>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ?
>>> exit_robust_list+0x90/0x160
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ?
>>> wake_up_state+0x10/0x20
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>]
>>> do_group_exit+0x58/0xd0
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>]
>>> get_signal_to_deliver+0x1f6/0x460
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>]
>>> do_signal+0x75/0x800
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ?
>>> __switch_to+0x1ac/0x320
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ?
>>> thread_return+0x4e/0x76e
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ?
>>> sys_futex+0x7b/0x170
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>]
>>> do_notify_resume+0x90/0xc0
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>]
>>> int_signal+0x12/0x17
>>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more
>>> than 120 seconds.
>>>
>>
>>
>> On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rcoli@eventbrite.com>
>> wrote:
>>
>>> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <blueflycn@gmail.com>
>>> wrote:
>>>
>>>> I just encountered an error which left a log '/hs_err_pid3013.log'. So
>>>> is there a way to solve this?
>>>>
>>>> # There is insufficient memory for the Java Runtime Environment to
>>>>> continue.
>>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>>>>> committing reserved memory.
>>>>>
>>>>
>>> Use less heap memory?
>>>
>>> You haven't specified under which circumstances this occurred, so I can
>>> only conjecture that it is likely being caused by "writing too fast".
>>>
>>> Write more slowly.
>>>
>>> =Rob
>>>
>>>
>>
>>
>

Mime
View raw message