Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
From: Dmitry Karachentsev <dkarachentsev@gridgain.com>
To: dev@ignite.apache.org
Subject: Re: Grid hang on compute
References: <25c3ae8a-a663-cec7-fdd1-81d31644aa90@gridgain.com>
 <CAGcMBHhiFaaANRSvEQPkofvu=3PSoW90zYkwTVzeim1VhmN34w@mail.gmail.com>
 <CA+0=VoXfQ-Gik0YNPYVDq7QjiO19QaE3gNr3S8ySVvKWRH=iLw@mail.gmail.com>
Message-ID: <5df5f009-ad8a-0a1a-0755-00be301896b5@gridgain.com>
Date: Thu, 8 Dec 2016 10:34:14 +0300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.5.1
MIME-Version: 1.0
In-Reply-To: <CA+0=VoXfQ-Gik0YNPYVDq7QjiO19QaE3gNr3S8ySVvKWRH=iLw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Thu, 08 Dec 2016 07:34:22 -0000

Thanks!

Opened a ticket to support Yakov's proposal.
https://issues.apache.org/jira/browse/IGNITE-4395

On 08.12.2016 4:03, Dmitriy Setrakyan wrote:
> Is there any way we can detect this and prevent from happening? Or perhaps
> start rejecting jobs if they can potentially block the system?
>
> On Wed, Dec 7, 2016 at 8:11 AM, Yakov Zhdanov <yzhdanov@apache.org> wrote:
>
>> Proper solution here is to have communication backpressure per policy -
>> SYSTEM or PUBLIC, but not single point as it is now. I think we can achieve
>> this having two queues per communication session or (which looks a bit
>> easier to implement) to have separate connections.
>>
>> As a workaround you can increase the limit. Setting it to 0 may lead to a
>> potential OOME on sender or receiver sides.
>>
>> --Yakov
>>
>> 2016-12-07 20:35 GMT+07:00 Dmitry Karachentsev <dkarachentsev@gridgain.com
>>> :
>>> Igniters!
>>>
>>> Recently faced with arguable issue, it looks like a bug. Scenario is
>>> following:
>>>
>>> 1) Start two data nodes with some cache.
>>>
>>> 2) From one node in async mode post some big number of jobs to another.
>>> That jobs do some cache operations.
>>>
>>> 3) Grid hangs almost immediately and all threads are sleeping except
>>> public ones, they are waiting for response.
>>>
>>> This happens because all cache and job messages are queued on
>>> communication and limited with default number (1024). It looks like jobs
>>> are waiting for cache responses that could not be received due to this
>>> limit. It's hard to diagnose and looks not convenient (as I know we have
>> no
>>> limitation in docs for using cache ops from compute jobs).
>>>
>>> So, my question is. Should we try to solve that or, may be, it's enough
>> to
>>> update documentation with recommendation to disable queue limit for such
>>> cases?
>>>
>>>