ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Threadpools and .WithExecute() for C# clients
Date Tue, 23 Jul 2019 19:38:14 GMT
Looping in the dev list.

Pavel, Igor and other C# maintainers, this looks like a valuable extension
of our C# APIs. Shouldn't this be a quick addition to Ignite?

-
Denis


On Mon, Jul 22, 2019 at 3:22 PM Raymond Wilson <raymond_wilson@trimble.com>
wrote:

> Alexandr,
>
> If .WithExecute is not planned to be made available in the C# client, what
> is the plan to support custom thread pools from the C# side of things?
>
> Thanks,
> Raymond.
>
>
> On Thu, Jul 18, 2019 at 9:28 AM Raymond Wilson <raymond_wilson@trimble.com>
> wrote:
>
>> The source of inbound requests into Server A is from client applications.
>>
>> Server B is really a cluster of servers that are performing clustered
>> transformations and computations across a data set.
>>
>> I originally used IComputeJob and similar functions which work very well
>> but have the restriction that they return the entire result set from a
>> Server B node in a single response. These result sets can be large (100's
>> of megabytes and larger), which makes life pretty hard for Server A if it
>> has to field multiple incoming responses of this size. So, these types of
>> requests progressively send responses back (using Ignite messaging) to
>> Server A using the Ignite messaging fabric. As Server A receives each part
>> of the overall response it processes it according the business rules
>> relevant to the request.
>>
>> The cluster config and numbers of nodes are not really material to this.
>>
>> Raymond.
>>
>> On Thu, Jul 18, 2019 at 12:26 AM Alexandr Shapkin <lexwert@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> Can you share a more detailed use case, please?
>>>
>>>
>>>
>>> Right now it’s not clear why do you need a messaging fabric.
>>>
>>> If you are interesting in a progress tracking, then you could try a
>>> CacheAPI or QueryContinious, for example.
>>>
>>>
>>>
>>> What are the sources of inbound requests? Is it a client requests?
>>>
>>>
>>>
>>> What is your cluster config? How many nodes do you have for your
>>> distributed computations?
>>>
>>>
>>>
>>> *From: *Raymond Wilson <raymond_wilson@trimble.com>
>>> *Sent: *Wednesday, July 17, 2019 1:49 PM
>>> *To: *user <user@ignite.apache.org>
>>> *Subject: *Re: Threadpools and .WithExecute() for C# clients
>>>
>>>
>>>
>>> Hi Alexandr,
>>>
>>>
>>>
>>> To summarise from the original thread, say I have server A that accepts
>>> requests. It contacts server B in order to help processing those requests.
>>> Server B sends in-progress results to server A using the Ignite messaging
>>> fabric. If the thread pool in server A is saturated with inbound requests,
>>> then there are no available threads to service the messaging fabric traffic
>>> from server B to server A resulting in a deadlock condition.
>>>
>>>
>>>
>>> In the original discussion it was suggested creating a custom thread
>>> pool to handle the Server B to Server A traffic would resolve it.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Raymond.
>>>
>>>
>>>
>>> On Wed, Jul 17, 2019 at 9:48 PM Alexandr Shapkin <lexwert@gmail.com>
>>> wrote:
>>>
>>> Hi, Raymond!
>>>
>>>
>>>
>>> As far as I can see, there are no plans for porting custom executors
>>> configuration in .NET client right now [1].
>>>
>>>
>>>
>>> Please, remind, why do you need a separate pool instead of a default
>>> PublicPool?
>>>
>>>
>>>
>>> [1] - https://issues.apache.org/jira/browse/IGNITE-6566
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From: *Raymond Wilson <raymond_wilson@trimble.com>
>>> *Sent: *Wednesday, July 17, 2019 10:58 AM
>>> *To: *user <user@ignite.apache.org>
>>> *Subject: *Threadpools and .WithExecute() for C# clients
>>>
>>>
>>>
>>> Some time ago I ran into and issue with thread pool exhaustion and
>>> deadlocking in AI 2.2.
>>>
>>>
>>>
>>> This is the original thread:
>>> http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html
>>>
>>>
>>>
>>>
>>> At the time .WithExecutor() was not implemented in the C# client so
>>> there was little option but to expand the size of the public thread pool
>>> sufficiently to prevent the deadlocking.
>>>
>>>
>>>
>>> We have been revisiting this issue and see that .WithExecutor() is not
>>> supported in the AI 2.7.5 client.
>>>
>>>
>>>
>>> Can this be supported in the C# client, or is there a workaround in the
>>> .Net environment? that does not require this capability?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Raymond.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message