ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (IGNITE-4699) Introduce custom configurable executors.
Date Fri, 17 Mar 2017 15:09:41 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930100#comment-15930100
] 

Vladimir Ozerov edited comment on IGNITE-4699 at 3/17/17 3:09 PM:
------------------------------------------------------------------

[~sharpler],
I believe we should proceed with original solution. Upper bound is important. Consider that
you start 1000 of heavy long-running tasks. In this case after the timeout you mention expires,
system will have 1000 threads heavily contenting for machine resources. This is why maximum
number of threads should stay.
Asking user to tell us whether he have nested tasks or not is bad idea either. If you have
more or less complex application on top of the grid with some level of abstractions and asynchrony,
it is nearly impossible to know whether you have nested call or not unless you propagate this
information explicitly all over the code, making it hard to maintain and tightly bound to
Ignite internals. 
Also we do not want to precisely control subtasking on our own because we will have to propagate
execution chains to our messages and to various thread-local mechanics, what will make our
engine less efficient.

My proposal based on positive experience of Akka [1] and Hazelcast [2] products. It is easy
to implement and easy to use. Another advantage is more fine grained resource control. For
example, user will be able to say "these types should consume no more than N cores on this
machine", thus having a way to prevent starvation of one tasks due to burst of others.

[1] http://doc.akka.io/docs/akka/current/scala/dispatchers.html
[2] http://docs.hazelcast.org/docs/3.5/manual/html/execserviceconfiguration.html


was (Author: vozerov):
[~sharpler],
I believe we should proceed with original solution. Upper bound is important. Consider that
you start 1000 of heavy long-running tasks. In this case after the timeout you mention expires,
system will have 1000 threads heavily contenting for machine resources. This is why maximum
number of threads should stay.
Asking user to tell us whether he have nested tasks or not is bad idea either. If you have
more or less complex application on top of the grid with some level of abstractions and asynchrony,
it is nearly impossible to know whether you have nested call or not unless you propagate this
information explicitly all over the code, making it hard to maintain and tightly bound to
Ignite internals. 
Also we do not want to precisely control subtasking on our own because we will have to propagate
execution chains to our messages and to various thread-local mechanics, what will make you
engine less efficient.

My proposal based on positive experience of Akka [1] and Hazelcast [2] products. It is easy
to implement and easy to use. Another advantage is more fine grained resource control. For
example, user will be able to say "these types should consume no more than N cores on this
machine", thus having a way to prevent starvation of one tasks due to burst of others.

[1] http://doc.akka.io/docs/akka/current/scala/dispatchers.html
[2] http://docs.hazelcast.org/docs/3.5/manual/html/execserviceconfiguration.html

> Introduce custom configurable executors.
> ----------------------------------------
>
>                 Key: IGNITE-4699
>                 URL: https://issues.apache.org/jira/browse/IGNITE-4699
>             Project: Ignite
>          Issue Type: Task
>          Components: compute
>            Reporter: Vladimir Ozerov
>            Assignee: Alexander Menshikov
>             Fix For: 2.0
>
>
> We need to provide a way to configure custom thread pools for user compute tasks. 
> Proposed API:
> 1) Config
> {code}
> class ExecutorConfiguration {
>     String name;
>     int size;
> }
> {code}
> 2) Choosing executor for compute task:
> {code}
> IgniteCompute compute = Ignite.compute().withExecutor("my_exec");
> {code}
> 3) Accessing raw executor (could be required for proper execution of future listeners):
> {code}
> ThreadPoolExecutor exec = ignite.compute().localExecutor("my_exec");
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message