ariatosca-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Orlov <ma...@gigaspaces.com>
Subject Re: [UPDATE] Work on ARIA-149: Support intrinsic functions in "dependencies" operation configuration
Date Thu, 25 May 2017 09:12:32 GMT
Not entirely sure why the "implementation" gets its own field. It basically
another input to the operation. Though it's a special input (since it
enables for the plugin do decide which "function" to run), it's still an
inputs.  feel we already have a lot of different possible inputs to an
operation: (1) interface inputs, (2) operation inputs (3) task inputs. I
feel that the "implementation" should have a special place under the
operation inputs, but not a field on its own...

On Thu, May 25, 2017 at 1:22 AM, Tal Liron <tal@gigaspaces.com> wrote:

> We had a long face-to-face today and reached an alignment. ARIA-149 is
> being updated to reflect it.
>
> So, we are going all the way with this:
>
> The "implementation" and "inputs" fields in the models refer *only* to what
> comes from the TOSCA parser. They are passed from template to instance as
> is. This means that "implementation" should always be considered an
> artifact name. (Indeed, when we implement artifact in the near future, this
> field may be refactored in some way, possibly to become a foreign key to an
> artifact-related model.)
>
> The OperationTemplate and Operation models both also have a "configuration"
> field, a dict of special parameters used as hints for orchestration. The
> parser can generate them using our magical "dependencies" grammar
> extensions. They are normal parameters (typed, and can contain intrinsic
> functions).
>
> And, we are introducing two *new* fields to Operation and Task models (and
> the related Task API): "function" and "arguments". "function" is the full
> path to a Python function, and "arguments" is a dict of parameters that are
> sent as arguments to the function. These are ARIA's implementation details
> for orchestration, and are *not* directly related to TOSCA's
> "implementation" and "inputs" fields, and indeed the Task model does not
> have nor need the "implementation" and "inputs" (to put it another way:
> we've renamed these two fields in Task, but they have the exact same usage
> as before).
>
> The logic for tasks has not changed, but let's go over it again for
> clarification with the new fields in mind:
>
>    1. The parser generates an OperationTemplate model, and sets the
>    "implementation" and "inputs" fields. Two ARIA grammar extensions
>    optionally apply:
>    1. If the ">" is used in the "implementation", then the parser actually
>       leaves "implementation" empty (there is no artifact) and instead sets
>       "function".
>       2. If the ">" is used in "dependencies", they are converted to
>       "configuration".
>       2. During instantiation, all these fields are copied as is from
>    OperationTemplate to Operation.
>    3. During the configuration phase of instantiation, there are two
>    possible routes:
>       1. For the default execution plugin, it generates "function" and
>       "arguments" according to whether the operation is local or remote,
> using
>       params from "configuration" if they are available (and valid). Also,
> any
>       extra "configuration" params set by the user, which are not
> recognized by
>       the plugin, just get appended to "arguments".
>       2. For other plugins, we already have "function" set for us by the
>       parser (according to what the user put in "implementation" after
> the ">").
>       So, we just append all "configuration" params if they exist to
> "arguments"
>    4. For all plugins, "inputs" are also appended to "arguments".
>
> Some implications of this logic worth considering:
>
>    1. "implementation" and "inputs" are never mangled and always adhere to
>    the strict TOSCA typed interface contract.
>    2. It's possible to use our special "dependencies" grammar to send
> *extra
>    arbitrary* arguments to the Python operation function, *beyond* what is
>    allowed by the strict TOSCA contract.
>    3. Python operation functions do not have a way to know which arguments
>    came from where: some arguments might be inputs and some might be extra
>    configuration. We have currently decided that this distinguishes the
> two is
>    cumbersome and not useful. If it does become important, we will need to
>    revisit this logic.
>
>
> On Tue, May 23, 2017 at 11:09 AM, Tal Liron <tal@gigaspaces.com> wrote:
>
> > Why do you think that on-the-fly updates won't require implementation and
> > inputs? It's true that the execution plugin squashes inputs into
> arguments
> > and keeps the implementation in arguments.process.script_path, but that's
> > an implementation detail. Some plugins may care more about what is an
> input
> > and what isn't, or do other kinds of mangling to the implementation. In
> any
> > case, the user should have to know how to massage these values: they can
> > set up inputs and possibly configuration (same way as in TOSCA) and then
> > call Operation.configure to do the right thing.
> >
> > The implementation and function are not the same, as you can see even in
> > my trivial example. The implementation is the path to a script. The
> > function is a path to a Python function. "function" is absolutely not an
> > evaluation of "implementation": for the execution plugin the function is
> > either run_script_local or run_script_with_ssh, which is entirely
> > determined by the topology (relationship to a host node) and has nothing
> to
> > do with the implementation. Who knows, some plugins in some cases might
> > totally throw away the implementation string in some cases if it becomes
> > irrelevant in some configurations.
> >
> > I just don't see why we need to introduce restrictions only to avoid
> > saving a single text column here. As for separating inputs -- seems very
> > obvious to me.
> >
> > We will follow up the discussion in person and post our final decision
> > here.
> >
> > On Tue, May 23, 2017 at 8:57 AM, Ran Ziv <ran@gigaspaces.com> wrote:
> >
> >> Actually I didn't mean ARIA-180 has already been merged, just that it
> >> might
> >> conflict with it and I think you two should sync about this.
> >>
> >> Regarding the new fields - I completely agree with your first paragraph
> -
> >> the original values should indeed be kept, and generally speaking the
> >> service instance models should be decoupled from the template models in
> >> order to allow on-the-fly updates and so on - but does this really apply
> >> here? What is the value of storing the original user values on the
> service
> >> instance models in this case? On-the-fly updates won't require the
> >> original
> >> value, and it does make sense to me that the only place the original
> >> values
> >> would be stored is on the template. At the end of the day the
> >> `implementation` and the `function` are one and the same - its simply
> that
> >> one of them is already evaluated and the other isn't. Also, Is it not
> the
> >> same case as properties that use the `get_input` intrinsic function?
> >>
> >> I guess the upside just isn't that clear to me here, and there is the
> >> downside of these being pretty confusing. The way I see it it's as if
> >> "implementation" is called "user_implementation" (vs "actual
> >> implementation") , or rather "template_implementation" - which again
> hints
> >> it should sit over at the template models..?
> >>
> >>
> >>
> >>
> >>
> >> On Mon, May 22, 2017 at 7:07 PM, Tal Liron <tal@gigaspaces.com> wrote:
> >>
> >> > I see Avia merged ARIA-180, so definitely will convert to one-to-many
> >> now.
> >> >
> >> > The reason the original values are preserved are the same reason all
> our
> >> > original values are preserved. The idea is that we don't want to make
> >> > service instance models dependent on service template models. They are
> >> > instantiated from them, but these foreign-key fields are all nullable.
> >> The
> >> > reason is to allow freedom for orchestration policies with on-the-fly
> >> > changes the topology. For example, a special kind of scaling policy
> >> might
> >> > involve adding an Nginx load balancer node for HTTP connections.
> >> However,
> >> > what if the user did not have a node template for that kind of the
> >> node? If
> >> > we require the creation of an OperationTemplate for every Operation,
> we
> >> > make this awkward, because it changes the original template. So,
> anyway,
> >> > this has been our principle until now and I suggest we continue this
> for
> >> > operations, too.
> >> >
> >> > Let's walk through a simple example.
> >> >
> >> >    1. A service template with a simple compute node, with a single
> >> "dazzle"
> >> >    operation on a custom interface, and this "dazzle" has a single
> >> >    implementation string, "dazzle_node.sh", with a single string
> input:
> >> >    "duration"="forever". To keep things simple, no special
> >> configuration is
> >> >    added via dependencies.
> >> >    2. The parser will create an OperationTemplate with these values:
> >> >       1. implementation = "dazzle_node.sh"
> >> >       2. inputs = {
> >> >          1. duration = "forever"}
> >> >       3. plugin_specification = None
> >> >       4. configuration = {}
> >> >       3. During initial instantiation phase, the above values will be
> >> >    copied to Operation model as is. At this early stage, "function"
> and
> >> >    "arguments" are still both empty.
> >> >    4. Now during instantiation *configuration* phase, we see that this
> >> >    operation uses the default execution plugin
> >> (plugin_specification=None).
> >> >    Because this node is a host, it will be configured to be a *remote*
> >> >    operation with very specific arguments. Also, we decided that for
> all
> >> >    plugins inputs get sent as *extra* arguments. So, here are all
> final
> >> >    values for Operation:
> >> >       1. implementation = "dazzle_node.sh"
> >> >       2. inputs = {
> >> >          1. duration = "forever"}
> >> >       3. configuration = {}
> >> >       4. plugin = None
> >> >       5. function = "aria.orchestrator.execution_
> >> >       plugin.operations.run_script_with_ssh"
> >> >       6. arguments = {
> >> >       1. process = {}
> >> >          2. script_path = "create_node.sh"
> >> >          3. use_sudo = False
> >> >          4. hide_output = []
> >> >          5. fabric_env = {user: DEFAULT_USER, password:
> >> DEFAULT_PASSWORD,
> >> >          key: None, key_filename: None}
> >> >          6. duration = forever}
> >> >
> >> > So, you can see here that if we change "implementation" to be
> >> > "aria.orchestrator.execution_plugin.operations.run_script_with_ssh"
> >> (like
> >> > we did before), the implementation is hidden in one of the arguments
> >> > (script_path), which would be very tricky for users to discover.
> That's
> >> an
> >> > implementation detail of the execution plugin, not something you want
> to
> >> > rely on. Also, if we merge everything into inputs, the original
> >> distinction
> >> > between what is an input and what is an implementation detail is lost.
> >> >
> >> > The fields in this suggested scheme have distinct and clear meanings:
> >> >
> >> >    1. "implementation" and "inputs" are directly from the TOSCA spec.
> >> They
> >> >    are preserved and never change.
> >> >    2. "configuration" is an extra optional ARIA extension dict
> allowing
> >> >    users to configure how the plugin will execute the operation.
> >> >    3. "function" and "arguments" are internal implementation details
> of
> >> the
> >> >    ARIA orchestrator. In a nicely straightforward sense they become a
> >> > Python
> >> >    function with kwargs.
> >> >
> >> > How this clears things up!
> >> >
> >> > On Sun, May 21, 2017 at 7:18 AM, Ran Ziv <ran@gigaspaces.com> wrote:
> >> >
> >> > > +1 for examples, or perhaps a link to specific code sections.
> >> > >
> >> > > From what I did understand though, before we talk about additional
> >> > changes,
> >> > > I'm not at all sure I'm in favor of the current ones.
> >> > >
> >> > > First, the many-to-many relationship sounds in contrast to what Avia
> >> is
> >> > > currently working on with (changing Parameters relationship type)
> >> > >
> >> > > Second, so what you're saying is that now Operation models will have
> >> both
> >> > > "implementation" and "function", and both "inputs" and "arguments"?
> >> This
> >> > > seems very confusing.
> >> > > I'm both not convinced the original values are needed at that stage
> >> (can
> >> > we
> >> > > have an actual use case?), but even if they are, do we really need
> to
> >> > keep
> >> > > all four on the same model?
> >> > > How about having the original values on the OperationTemplate, and
> the
> >> > > "coerced values" (for lack of a better term..) on the Operation
> model?
> >> > >
> >> > >
> >> > >
> >> > > On Sun, May 21, 2017 at 11:45 AM, Arthur Berezin <
> >> arthur@gigaspaces.com>
> >> > > wrote:
> >> > >
> >> > > > On Sat, May 20, 2017 at 3:47 AM Tal Liron <tal@gigaspaces.com>
> >> wrote:
> >> > > >
> >> > > > > Phew! Everything seems to work now: functions are parsed
and
> >> > evaluated
> >> > > > > properly. But there are some changes that require explaining.
> >> > > > >
> >> > > > > The "configuration" field in OperationTemplate and Operation
> >> models
> >> > is
> >> > > > now
> >> > > > > a many-to-many with Parameter. This allows the values to
fully
> >> > support
> >> > > > > intrinsic functions and types.
> >> > > > >
> >> > > > > And I made another change in modeling: Operation now also
has
> >> > > "function"
> >> > > > > and "arguments" fields, which are what actually gets used
by the
> >> Task
> >> > > API
> >> > > > > instead of "implementation" and "inputs". This means that
the
> >> > > > > "implementation" and "inputs" values are preserved as is.
During
> >> the
> >> > > > > configuration phase the "arguments" are created according
to the
> >> > usual
> >> > > > > logic: for the execution plugin, the "configuration" is
massaged
> >> > into a
> >> > > > few
> >> > > > > new "arguments", and otherwise other "inputs" are also merged
> into
> >> > > > > "arguments".
> >> > > > >
> >> > > > > Why make this change? Because I really think we should not
> mangle
> >> the
> >> > > > > "inputs": they are logically different from the arguments
that
> get
> >> > sent
> >> > > > to
> >> > > > > the @operation function, even if in many cases we might
treat
> them
> >> > > > > identically. Also, we cannot lose the "implementation" string:
> >> > plugins
> >> > > > will
> >> > > > > need to know what users put in there. The change is not
big in
> >> terms
> >> > of
> >> > > > > extra code, but I think it helps make the code easier to
> >> understand:
> >> > > you
> >> > > > > can see exactly what is being merged into the final "arguments"
> >> for
> >> > the
> >> > > > > task. The configure phase is now constructive rather than
> >> > destructive:
> >> > > > > existing fields are not changed.
> >> > > > >
> >> > > >
> >> > > >
> >> > > > Tal, could you provide several example models to showcase these
> >> > changes?
> >> > > >
> >> > > >
> >> > > >
> >> > > > >
> >> > > > > Before moving forward with a PR I want to see what other
> >> committers
> >> > > think
> >> > > > > about also renaming "implementation" and "inputs" in the
Task
> API
> >> to
> >> > > > > "function" and "arguments". For now I'm leaving them as
is, but
> I
> >> > think
> >> > > > it
> >> > > > > would be even cleaner if they followed the new naming convention
> >> in
> >> > > > > modeling. I continue being unhappy about how the Task API
treats
> >> both
> >> > > > > arguments and inputs as the same thing and squashing them
all
> >> > > together...
> >> > > > > :( However, for now I would be happy if we just called them
> >> > > "arguments",
> >> > > > > because that's what they really are for us.
> >> > > > >
> >> > > > > --
> >> > > > > Tal Liron
> >> > > > > Solution Architect
> >> > > > > tal@gigaspaces.com | +13123758299 <(312)%20375-8299>
> >> > > > > Cloudify | http://getcloudify.org
> >> > > > > <
> >> > > > > http://getcloudify.org?utm_source=signaturesatori&utm_
> >> > > > medium=email&utm_campaign=general_signature
> >> > > > > >
> >> > > > >
> >> > > > > <https://twitter.com/CloudifySource>
> >> > > > > <https://www.linkedin.com/groups/8467478>
> >> > > > > <https://github.com/cloudify-cosmo>   <
> >> https://github.com/cloudify-
> >> > > cosmo
> >> > > > >
> >> > > > > [image: Azure Webinar]
> >> > > > > <
> >> > > > > http://getcloudify.org/webinars/Azure-plugin-for-
> >> > > > cloudify-webinar.html?utm_source=signaturesatori&utm_
> >> > > > medium=email&utm_campaign=general_signature
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Tal Liron
> >> > Senior Engineer
> >> > tal@gigaspaces.com | +1 (773) 828-9339
> >> > Cloudify | http://getcloudify.org
> >> > <http://getcloudify.org?utm_source=signaturesatori&utm_
> >> > medium=email&utm_campaign=general_signature>
> >> >
> >> > <https://twitter.com/CloudifySource>
> >> > <https://www.linkedin.com/groups/8467478>
> >> > <https://github.com/cloudify-cosmo>   <https://github.com/cloudify-
> >> cosmo>
> >> > [image: Azure Webinar]
> >> > <http://getcloudify.org/webinars/Azure-plugin-for-
> >> > cloudify-webinar.html?utm_source=signaturesatori&utm_
> >> > medium=email&utm_campaign=general_signature>
> >> >
> >>
> >
> >
> >
> > --
> > Tal Liron
> > Senior Engineer
> > tal@gigaspaces.com | +1 (773) 828-9339
> > Cloudify | http://getcloudify.org
> > <http://getcloudify.org?utm_source=signaturesatori&utm_
> medium=email&utm_campaign=general_signature>
> >
> > <https://twitter.com/CloudifySource>
> > <https://www.linkedin.com/groups/8467478>
> > <https://github.com/cloudify-cosmo>   <https://github.com/cloudify-cosmo
> >
> > [image: Azure Webinar]
> > <http://getcloudify.org/webinars/Azure-plugin-for-
> cloudify-webinar.html?utm_source=signaturesatori&utm_
> medium=email&utm_campaign=general_signature>
> >
> >
>
>
>
> --
> Tal Liron
> Senior Engineer
> tal@gigaspaces.com | +1 (773) 828-9339
> Cloudify | http://getcloudify.org
> <http://getcloudify.org?utm_source=signaturesatori&utm_
> medium=email&utm_campaign=general_signature>
>
> <https://twitter.com/CloudifySource>
> <https://www.linkedin.com/groups/8467478>
> <https://github.com/cloudify-cosmo>   <https://github.com/cloudify-cosmo>
> [image: Azure Webinar]
> <http://getcloudify.org/webinars/Azure-plugin-for-
> cloudify-webinar.html?utm_source=signaturesatori&utm_
> medium=email&utm_campaign=general_signature>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message