brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aled Sage (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BROOKLYN-356) The sensor Transformer enricher fails to resolve its targetValue indeterministically.
Date Wed, 12 Oct 2016 19:51:20 GMT

    [ https://issues.apache.org/jira/browse/BROOKLYN-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569708#comment-15569708
] 

Aled Sage commented on BROOKLYN-356:
------------------------------------

I talked this over with [~svet]. We've done a very short-term improvement of increasing the
timeout. I'm looking at a longer term fix where we don't execute so many tasks in different
threads to get the value.

As Svet described, the {{Transformer}} executes a task to get the value (with a timeout, because
the value might not be available yet). It is trying to do a non-blocking lookup of the value,
but using the task api that requires the work be executed in another thread. It therefore
uses a timeout to terminate the lookup if it hasn't returned promptly, but that obviously
gives a race condition.

A better solution for such a non-blocking lookup is to ask for the value immediately, and
to get back a {{Maybe}} (similar to guava's {{Optional}}, but accepts null values). Each DSL
(formatString, attributeWhenReady, etc) is of type {{DeferredSupplier}}. I'm adding another
interface ({{ImmediateSupplier}} with a method {{Maybe<T> getImmediately()}}) that we'll
use to get the value immediately, if we don't want to wait for it. However, it is a bit fiddly
(e.g. the DSL execution doesn't know the context entity, so I'm making some (small) changes
to the subscription context as well).

Even longer term, we could look at reversing the responsibilities: if the value is an {{attributeWhenReady}}
then that could do the subscription itself, and call back to the transformer when the value
becomes available (or changes). Also, by using callbacks more, we can change some of the places
that we do blocking calls (so we improve our thread usage).

Below is the current sequence of task being executed (and the thread's stacktrace for each),
when executing {{TransformerEnricherWithDslTest}} and the transformer retrieving the value:

{noformat}
task=Task[waiting on port]@J5UlSQ04; thread=Thread[brooklyn-execmanager-S0LtDxD8-9,5,main];
submitTime=1476272773046; startTime=1476272773046; queuedTime=-1
task=Task[formatting '%d' with 1 task]@lNzNRuj1; thread=Thread[brooklyn-execmanager-S0LtDxD8-11,5,main];
submitTime=1476272773045; startTime=1476272773045; queuedTime=-1
task=Task[formatting '%s' with 1 task]@ExQmzZBq; thread=Thread[brooklyn-execmanager-S0LtDxD8-10,5,main];
submitTime=1476272773045; startTime=1476272773045; queuedTime=-1
task=Task[Resolving dependent value]@wQxaXZ77; thread=Thread[brooklyn-execmanager-S0LtDxD8-7,5,main];
submitTime=1476272773024; startTime=1476272773025; queuedTime=-1
task=Task[LSM.publishInitialValue(Application[9ftxjzc5], Sensor: port (java.lang.Integer));[subscription-delivery-entity-pf9ftxjzc5[Application[9ftxjzc5]]]]@C1UYl8il;
thread=Thread[brooklyn-execmanager-S0LtDxD8-1,5,main]; submitTime=1476272738192; startTime=1476272738192;
queuedTime=-1

task=Task[waiting on port]@J5UlSQ04; thread=Thread[brooklyn-execmanager-S0LtDxD8-9,5,main]
"brooklyn-execmanager-S0LtDxD8-9" daemon prio=5 tid=0x00007facf22ab800 nid=0x660b at breakpoint[0x0000700001be7000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.brooklyn.core.sensor.DependentConfiguration$WaitInTaskForAttributeReady.call(DependentConfiguration.java:252)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

task=Task[formatting '%d' with 1 task]@lNzNRuj1; thread=Thread[brooklyn-execmanager-S0LtDxD8-11,5,main]
"brooklyn-execmanager-S0LtDxD8-11" daemon prio=5 tid=0x00007facf3a8b800 nid=0x6c07 waiting
on condition [0x0000700001ded000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007f64e81c0> (a java.util.concurrent.FutureTask)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
        at java.util.concurrent.FutureTask.get(FutureTask.java:187)
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
        at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:361)
        at org.apache.brooklyn.core.sensor.DependentConfiguration$1.call(DependentConfiguration.java:409)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

task=Task[formatting '%s' with 1 task]@ExQmzZBq; thread=Thread[brooklyn-execmanager-S0LtDxD8-10,5,main]
"brooklyn-execmanager-S0LtDxD8-10" daemon prio=5 tid=0x00007facf2898000 nid=0x6507 waiting
on condition [0x0000700001cea000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007f63a0cb8> (a java.util.concurrent.FutureTask)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
        at java.util.concurrent.FutureTask.get(FutureTask.java:187)
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
        at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:361)
        at org.apache.brooklyn.core.sensor.DependentConfiguration$1.call(DependentConfiguration.java:409)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

task=Task[Resolving dependent value]@wQxaXZ77; thread=Thread[brooklyn-execmanager-S0LtDxD8-7,5,main]
"brooklyn-execmanager-S0LtDxD8-7" daemon prio=5 tid=0x00007facf13d6000 nid=0x5a07 waiting
on condition [0x00007000018de000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007f60309b0> (a java.util.concurrent.FutureTask)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
        at java.util.concurrent.FutureTask.get(FutureTask.java:187)
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
        at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:361)
        at org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get(BrooklynDslDeferredSupplier.java:129)
        at org.apache.brooklyn.util.core.task.ValueResolver$2.call(ValueResolver.java:334)
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

task=Task[LSM.publishInitialValue(Application[9ftxjzc5], Sensor: port (java.lang.Integer));[subscription-delivery-entity-pf9ftxjzc5[Application[9ftxjzc5]]]]@C1UYl8il;
thread=Thread[brooklyn-execmanager-S0LtDxD8-1,5,main]
"brooklyn-execmanager-S0LtDxD8-1" daemon prio=5 tid=0x00007facf136f800 nid=0x6203 waiting
on condition [0x00007000019e0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007f4ee2190> (a java.util.concurrent.FutureTask)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:422)
        at java.util.concurrent.FutureTask.get(FutureTask.java:199)
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
        at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:467)
        at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:434)
        at org.apache.brooklyn.util.time.Durations.get(Durations.java:57)
        at org.apache.brooklyn.util.time.Durations.get(Durations.java:70)
        at org.apache.brooklyn.util.core.task.ValueResolver.getMaybeInternal(ValueResolver.java:353)
        at org.apache.brooklyn.util.core.task.ValueResolver.getMaybe(ValueResolver.java:257)
        at org.apache.brooklyn.enricher.stock.Transformer$4.apply(Transformer.java:95)
        at org.apache.brooklyn.enricher.stock.Transformer$4.apply(Transformer.java:1)
        at org.apache.brooklyn.enricher.stock.AbstractTransformer.compute(AbstractTransformer.java:153)
        at org.apache.brooklyn.enricher.stock.AbstractTransformer.onEvent(AbstractTransformer.java:147)
        at org.apache.brooklyn.core.mgmt.internal.LocalSubscriptionManager$1.run(LocalSubscriptionManager.java:149)
        at org.apache.brooklyn.util.concurrent.CallableFromRunnable.call(CallableFromRunnable.java:43)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
        at org.apache.brooklyn.util.core.task.SingleThreadedScheduler$1.call(SingleThreadedScheduler.java:116)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
{noformat}


> The sensor Transformer enricher fails to resolve its targetValue indeterministically.
> -------------------------------------------------------------------------------------
>
>                 Key: BROOKLYN-356
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-356
>             Project: Brooklyn
>          Issue Type: Bug
>            Reporter: Svetoslav Neykov
>
> The sensor {{Transformer}} enricher fails to resolve its {{targetValue}} indeterministically,
especially so on loaded systems. This is especially problematic if sourceSensor/triggerSensors
change infrequently or do not change ever (when using default values).
> Bueprints using the {{Transformer}} behave perfectly fine during development and on test
setups, but will fail on production machines (i.e. under load).
> The code [doing the value resolving|https://github.com/apache/brooklyn-server/blob/b59e7463a9b337c2d0e7931cd420d5bac68d8549/core/src/main/java/org/apache/brooklyn/enricher/stock/Transformer.java#L90-L94]
tries to do so in a non-blocking fashion by spinning a thread to try to resolve and cancelling
it after a short while without guarantees it ever got scheduled to run. It's more likely to
fail when nesting DSLs, for example nesting several levels of {{$brooklyn:formatString}} and
ending with a {{$brooklyn:attributeWhenRready}}. It's a common pattern in moderately complex
blueprints. It needs to schedule a thread for each nesting level thus maximizing the chance
that the value will not be resolved in the allotted time even if resolvabe. This is especially
bad for sensors which don't get updated, for example {{PortAttributeSensorAndConfigKey} set
only when initializing the entity or any config values which are always resolvable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message