ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Gura <ag...@apache.org>
Subject Re: Ignit Cache Stopped
Date Fri, 17 Feb 2017 17:26:27 GMT
>From GC logs at the end of files I see Full GC pauses like this:

2017-02-17T04:29:22.118-0800: 21122.643: [Full GC (Allocation Failure)
 10226M->8526M(10G), 26.8952036 secs]
   [Eden: 0.0B(512.0M)->0.0B(536.0M) Survivors: 0.0B->0.0B Heap:
10226.0M(10.0G)->8526.8M(10.0G)], [Metaspace:
77592K->77592K(1120256K)]

Your heap is exhausted. During GC discovery doesn't receive heart
betas and nodes stopped due to segmentation. Please check your nodes'
logs for NODE_SEGMENTED pattern. If it is your case try to tune GC or
reduce load on GC (see for details [1])

[1] https://apacheignite.readme.io/docs/jvm-and-system-tuning

On Fri, Feb 17, 2017 at 6:35 PM, Anil <anilklce@gmail.com> wrote:
> Hi Andrey,
>
> The queyr execution time is very high when limit 10000+250 .
>
> 10 GB of heap memory for both client and servers. I have attached the gc
> logs of 4 servers. Could you please take a look ? thanks.
>
>
> On 17 February 2017 at 20:52, Anil <anilklce@gmail.com> wrote:
>>
>> Hi Andrey,
>>
>> I checked GClogs  and everything looks good.
>>
>> Thanks
>>
>> On 17 February 2017 at 20:45, Andrey Gura <agura@apache.org> wrote:
>>>
>>> Anil,
>>>
>>> IGNITE-4003 isn't related with your problem.
>>>
>>> I think that nodes are going out of topology due to long GC pauses.
>>> You can easily check this using GC logs.
>>>
>>> On Fri, Feb 17, 2017 at 6:04 PM, Anil <anilklce@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > We noticed whenever long running queries fired, nodes are going out of
>>> > topology and entire ignite cluster is down.
>>> >
>>> > In my case, a filter criteria could get 5L records. So each API request
>>> > could fetch 250 records. When page number is getting increased the
>>> > query
>>> > execution time is high and entire cluster is down
>>> >
>>> >  https://issues.apache.org/jira/browse/IGNITE-4003 related to this ?
>>> >
>>> > Can we set seperate thread pool for queries executions, compute jobs
>>> > and
>>> > other services instead of common public thread pool ?
>>> >
>>> > Thanks
>>> >
>>> >
>>
>>
>

Mime
View raw message