river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patricia Shanahan <p...@acm.org>
Subject Re: A new implementation of TaskManager
Date Wed, 07 Jul 2010 19:24:19 GMT
Peter Firmstone wrote:
> Patricia Shanahan wrote:
>> Peter Firmstone wrote:
>>> So it would seem that the runAfter() method is superfluous and 
>>> unnecessary in most, if not all cases?  Especially if the caller is 
>>> making the decision about when tasks are placed on the queue.
>>> We do need to confirm that in our current implementations, the 
>>> callers are taking responsibility for the queue placement.  Then try 
>>> to discovery why we needed to have a dependency check by the 
>>> TaskManager.
>>> It sounds like for absolute concurrency the design needs to be 
>>> totally revised, perhaps the responsibility of determining if the 
>>> Task is ready to be executed should be placed solely on the Task 
>>> implementation itself, but not by passing in the list of all Tasks on 
>>> the queue, but with a simple:
>> Do you have a lot of bug reports indicating the concurrency design is 
>> basically broken?
> No, not really, don't think it's full of bugs, just not as concurrent as 
> it should be, I'd like to see it plateau under pressure, rather than 
> degrade significantly.   I'm also a bit puzzled why the caller, 
> TaskManager and Task all share some responsibility (but not all 
> responsibility) for managing dependency ordering correctness.

I agree with the "plateau under pressure" objective, and think 
degradation should be investigated. That said, there are some things 
that may be beyond our control. For example, the current tendency in 
operating system memory management is to keep time slicing all runnable 
threads even if it means thrashing, rather than picking a subset that 
can run, not crawl, and periodically changing the subset.

>> In a few months, when I have had time to learn my way around both the 
>> specification and the implementation, I would be happy to participate 
>> in a liveness and ordering review. I've never done it for software, 
>> but some of the intellectual tools used for proving memory order 
>> properties in large multiprocessor designs were borrowed from 
>> distributed software engineering.
> Sounds like you've got some good experiences to share.

Yes, but please remember I want to function primarily as a Java 
programmer. I have spent decades as a project leader, or architect, or 
most recently as an academic researcher. Been there, done that. 
Programming is the part of computing I enjoy most, I'm retired, and I'm 
doing this for fun.

>> I am opposed to moving the queuing and run after organization work 
>> from TaskManager to its callers. 
> I understand, if based on the work involved for each implementing 
> class.  To reduce the work substantially, a class implementing Task 
> (eg:: TaskHelper) could be provided for the developer implementing a 
> Task to encapsulate, to delegate the dependency management to 
> internally, rather than extending an abstract class.  This reduces the 
> work back to mostly implementing one method, run(), but leaves the 
> implementer free to do something else too. 
> It does have some merit:
>    * Very small lists and references are fast, with small lock windows
>      and good concurrency.
>    * The caller doesn't talk to TaskManager, only Task, coupling is
>      reduced.

The caller still needs to control which tasks share a set of threads, 
and the characteristics of the set of threads, such as the rules for the 
number of threads in the set. How would this work if the caller does not 
create and talk to a TaskManager or equivalent?

>    * The caller doesn't have to be responsible for the safety of
>      placing Tasks on a queue.

Wouldn't the caller would have to be fully responsible, in the sense of 
not adding a Task until it can be run in any order relative to all tasks 
in the queue?

>    * Any number of initiators can add dependencies to a Task.
>    * TaskManager can assume all Tasks received are ready to run.
>    * The Task implementer can encapsulate a TaskHelper, to delegate
>      responsibility, for dependencies and correctness.
>> It is work that has to be done, and the choice is between doing it in 
>> one class, or doing it in each class that needs to both permit some 
>> parallelism and yet maintain some ordering among tasks. It is far less 
>> work to make it both efficient and correct if it only needs to be done 
>> in one class.
> But I digress, it's only a suggestion and it might not be the optimum 
> solution, so feel free to do something else.

I think there is an approach that gives us the immediate benefits of 
fixing the obvious performance issues in TaskManager without eliminating 
the possibility of moving in the direction you are suggesting if it 
seems desirable later.


View raw message