river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Creswell <dan.cresw...@gmail.com>
Subject Re: TaskManager
Date Fri, 01 Jul 2011 06:38:37 GMT
On 1 July 2011 03:22, Gregg Wonderly <gregg@wonderly.org> wrote:
> On 6/30/2011 9:02 PM, Patricia Shanahan wrote:
>>
>> I'm having trouble squaring this with the description of
>> ThreadPoolExecutor. If I'm reading it right, it *can* be initialized to
>> set a low limit on the maximum number of threads, but can equally well
>> be set to have up to Integer.MAX_VALUE threads, effectively unlimited.
>
> Yes, this is the case.  There are some other subtle behaviors which we just
> need to study to make sure that the "documented behavior" is what we need.
>
>> There are two forms of TaskManager behavior that I do not see an easy
>> way to reproduce using a simple ThreadPoolExecutor, runAfter and
>> loadLevel.
>>
>> runAfter my be better handled by the class issuing the tasks.
>
> There is also the Fork-Join framework which does allow some "arrangement" of
> order.  I'm not sure this is a great fit right now, but it is something to
> consider I think, especially for "notifications", "commits" and some other
> things that have two level ordering "function->tasks".
>
>> loadLevel permits specification of a ratio between threads and tasks, so
>> that threads are only created up to e.g. one third of the number of
>> tasks. I'm not sure that is a really good idea. It means leaving tasks
>> to wait even if we could create another thread.
>

Yes, but tasks are waiting because you're choosing to limit resources.
That always happens. The choices are to increase the limit and utilise
more processor (assuming you have it) or have fewer tasks (often a
function of user-load so needs admission control etc to make that
stick).

> I don't like this thought at all in a distributed system.  A vast majority
> of tasks really need to be invoked immediately to maintain a reasonable
> chance at throughput.
>

Equally a vast majority of tasks can only have a reasonable chance of
throughput in the presence of sufficient resource. The point of
TaskManager is to control resource consumption (including processor
utilisation) which naturally asserts a limit on throughput.

> Gregg
>
>> Patricia
>>
>>
>>
>> On 6/29/2011 7:06 AM, Christopher Dolan wrote:
>>>
>>> Just remember, we discussed this a year ago
>>> (http://mail-archives.apache.org/mod_mbox/incubator-river-dev/201006.mbo
>>> x/%3C4C12147A.8030601@zeus.net.au%3E) and some of us concluded that
>>> ThreadPoolExecutor has rather non-intuitive runtime behavior because
>>> it's optimized to not cause OutOfMemory problems. Not everyone had a
>>> problem with that behavior, but I'm not a fan.
>>>
>>> Chris
>>>
>>> P.S. can someone add a link to the Incubator mail archives to the web
>>> page? They are quite hard to find...
>>> http://mail-archives.apache.org/mod_mbox/incubator-river-dev/
>>> http://mail-archives.apache.org/mod_mbox/incubator-river-user/
>>> http://mail-archives.apache.org/mod_mbox/incubator-river-commits/
>>>
>>>
>>> -----Original Message-----
>>> From: Peter Firmstone [mailto:jini@zeus.net.au]
>>> Sent: Sunday, June 26, 2011 5:18 PM
>>> To: dev@river.apache.org
>>> Subject: Re: TaskManager
>>>
>>>
>>>> I think the way to look at this is to examine the non-trivial runAfter
>>>> implementations. If they have common elements that can be helped by
>>>
>>> some
>>>>
>>>> utility classes, write those classes. If not, deal with each
>>>> individually.
>>>>
>>>> We are now committed to 1.5, so if we did not have runAfter we could
>>>> replace TaskManager with java.util.concurrent.ThreadPoolExecutor.
>>>
>>> That's probably where we should be headed, since it reduces our
>>> maintenance burden.
>>>
>>> Peter.
>>>
>>
>>
>
>

Mime
View raw message