brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aled Sage <>
Subject Re: [PROPOSAL] Controlling effectors concurrency
Date Thu, 12 Jan 2017 09:49:35 GMT
Hi all,

Great discussion - really like the direction this is going of using 
tasks in yaml blueprints.

However, it feels like we've launched into discussing a complex use-case 
(concurrency control) without having first discussed what yaml 
blueprints would look like for simpler tasks (*). I suggest we discuss 
that in another proposal thread, and get agreement of what the YAML 
should look like. Let's focus on the yaml blueprint author - i.e. reach 
agreement on some yaml examples, without worrying too much about how 
it's implemented under the covers.

I'd like the examples to show an entire blueprint - including defining 
effectors, sensors, feeds, etc.


Svet, for the "callback behaviour for effectors" I think that would be a 
powerful/advanced feature. If multiple callbacks were registered (e.g. 
in the super-type and in the sub-type, when extending existing 
blueprints), the order the callbacks are called could get tricky for 
blueprint authors. Maybe we live with that. Maybe the task-based 
approach will give a more elegant way to control it, making the 
callbacks redundant.

If it helps to solve your immediate problem cleanly, I'm fine with us 
adding support for it. But I worry that we'd want to deprecate it 
shortly after task-based yaml is supported.


(*) Sorry if I've forgotten about some previous discussion on the 
mailing list - if I have, then please point us at it again!

On 11/01/2017 17:45, Svetoslav Neykov wrote:
> Sam I think that's a great approach to composing tasks and mixing them with semaphores.
>> sem --id brooklyn --fg "$REMOVE_OLD_CMD ; $ADD_NEW_CMD"
> Alex, I like that, gives you much cleaner semantics than flock.
>> effectors:
>>    start:
>>       my-pre-launch:
>>         task:  { acquire-semaphore: ... }
>>         run-before: launch
> Geoff, I have some reservations re init.d style naming as well. Mostly due to being too
fragile when having multiple layers of blueprints. I'm going to suggest one more alternative,
somehow similar to before/after. Instead of declaring when to run a task (before/after another
task), just override the parent task, do the work you need to and reference the parent task
> effectors:
>    start:
>       launch: [my-custom-launch, $super.launch]
>       my-custom-launch:
>       - ssh: ....
> Anyway this is getting sidetracked. I agree that the task based approach is a nice place
to be longer term. Might go with extending the latches shorter term - need to think some more
about it.
> What do people think about introducing the callback behaviour for effectors though? I
see it as an orthogonal concept to what's possible with composing tasks. It's a middle ground
for implementing the semaphores where the latches are too limited and "semaphores as tasks"
are still not here. Will make life easier for some more complex scenarios where one needs
to sync with/influence other entities (examples in previous emails).
> Svet.
>> On 11.01.2017 г., at 18:50, Alex Heneveld <>
>> Svet-
>> On 11/01/2017 15:55, Svetoslav Neykov wrote:
>>>> PS - as a near-term option if needed we could extend SoftwareProcess LATCH
to do something special if the config/sensor it is given is a "Semaphore" type
>>> What do you think the behaviour should be here - releasing the semaphore after
the corresponding step completes or at the end of the wrapping effector? I think this is defined
by the blueprint authors. And making this configurable adds even more complexity. Instead
could invest in developing the above functionality.
>> I'd probably release the latch sempahore right after the phase but I agree it's arbitrary
and don't like that.  Only suggested it if we need something very quick as it's isolated and
probably fairly easy in the code.
>> Note code exists and is run implicitly I think during the Install step (otherwise
things like yum complain), along with I think the class is called SemaphoreWithOwner.
>> I'd LOVE to invest in the task-based functionality.  Code-wise it's not that far
away with the type registry and YOML, but it needs a couple more people to become familiar
with it!
>> Lastly I meant to say -- someone mentioned flock but the bash *sem* command (aka
parallel) is awesome though not as well known as it should be.  This is an example used in
>> sudo yum install parallel
>> REMOVE_OLD_CMD="sed -i /server.*=/d zoo.cfg"
>> ADD_NEW_CMD="cat >> zoo.cfg << EOF
>> server.0=NEW
>> server.1=NEW
>> EOF
>> "
>> sem --id brooklyn --fg "$REMOVE_OLD_CMD ; $ADD_NEW_CMD"
>> It runs the commands in the quotes on the last line while acquiring the semaphore
with the given name "brooklyn" on the machine.  You can configure semaphore counts and timeouts
>> Best
>> Alex

View raw message