ariatosca-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Ziv <...@cloudify.co>
Subject Re: Service Composition / Substitution Mapping
Date Wed, 16 Aug 2017 10:59:23 GMT
I'd say right now we're looking at "static service composition" which is
only about "substituting service templates", not "substituting service". If
a service is already running, it will not be used.

I think what Tal meant was that each service template - whether the top
level one or one of the substituting templates - needs to resolve its inner
reqs&caps internally first, and then resolve substitution reqs&caps across
service templates.


On Wed, Aug 16, 2017 at 12:00 PM, D Jayachandran <
d.jayachandran@ericsson.com> wrote:

> Hi Tal,
>
> Thanks for organizing the points.
> So if I understand correctly we are looking only at "Static service
> composition" which includes "substituting service template" and
> "substituting service".
>
> As you said with "substituting service template" approach ,we will have
> all the nodes aggregated from other service templates and a single workflow
> would be triggered to perform life-cycle operation on all the nodes.
> Am not sure why the workflows needs to be "boundary aware" for nodes being
> substituted ? I see nodes are already part of the composed service, Could
> you please help me understand this ?
>
>
> Regards,
> DJ
> -----Original Message-----
> From: Tal Liron [mailto:tal@cloudify.co]
> Sent: Saturday, August 12, 2017 4:52 AM
> To: dev@ariatosca.incubator.apache.org
> Subject: Re: Service Composition / Substitution Mapping
>
> You are correct -- to participate in this "multi-VIM" scenario, the
> Openstack plugin would have to know how to translate the TOSCA properties
> to a flavor ID. This could all be done in 100% TOSCA via policies (say, an
> aria.Openstack).
>
> Doing this automatically might not be a good idea, or even necessary.
> Worst case is you get a validation error if the ARIA plugin can't find a
> flavor in the table to match your requirements, in which you case you can
> go and manually find the right ID and add it to the table.
>
> And I agree about being fine with imperfection: the rule of thumb for our
> work is always to allow for sensible defaults even if no explicit policy is
> given.
>
> Anyway, we've gone way off the topic of this thread. We can talk about it
> more once it comes closer to an implementation.
>
> On Fri, Aug 11, 2017 at 3:52 PM, DeWayne Filppi <dewayne@cloudify.co>
> wrote:
>
> > Interesting.  Take Openstack (please <rim shot>).  If you model a
> > compute OS as explicitly as you like, there still has to be a "match"
> > to an Openstack image ID.  Or are you saying you must supply the image
> ID for
> > OSs.   Likewise, you can't supply RAM and CPUs without a flavor ID.
> > Openstack does allow for custom flavors, but I doubt the plugin is
> > doing that.  Much better to have a "caps-init" interface in some low
> > down base type that can be triggered at service creation to support
> > reqs/caps (IMHO).  Then the plugin can verify whether the service can
> > be construct based on fuzzy constraints.  Maybe this is a case that
> > the "general solution" is a nightmare of complexity, but having a
> > plugin scan the available flavors to make sure a requirement can be
> > met doesn't seem that tough.  If TOSCA provided a formal lifecycle
> > interface for it, then orchestrators or just plugins could determine
> > how tricky they wanted to be.  IOW, let not the perfect be the enemy of
> the good.
> >
> > DeWayne
> >
> >
> > On Fri, Aug 11, 2017 at 1:26 PM, Tal Liron <tal@cloudify.co> wrote:
> >
> > > OK, that's a whole different can of worms. :)
> > >
> > > TOSCA's Compute capabilities (Container and OperatingSystem) are
> > explicit.
> > > You specify which OS you want, how much RAM you want, how many CPUs,
> etc.
> > > ARIA's explicit node types (for example, the AWS Compute node) are
> > likewise
> > > explicit. So there is not querying here: the plugin will attempt to
> > > spin
> > up
> > > exactly the virtual machine you asked for. If it fails, the workflow
> > > will fail.
> > >
> > > This is not good enough, I think, for real world scenarios. There
> > > are two possible solutions:
> > >
> > > 1) Support ranges or fallbacks. So instead of saying "I need 4 GB of
> RAM"
> > > you can say "I would like 4 GB of RAM, but 2 GB would also be OK."
> > There's
> > > no easy way to do this now without totally changing how the
> > > capability types are designed. But, it may be possible to override
> > > this via
> > policies.
> > > So, the capabilities would perhaps specify the minimal requirements,
> > while
> > > policies specify preferences. Some aspects of this were discussed in
> > > the OPEN-O project. DeWayne, has any of this survived in ONAP, or
> > > have we not reached that point in the discussion yet?
> > >
> > > 2) Incorporate this into the bigger topic of resource orchestration.
> > This a
> > > huge challenge for the industry. The problem field contains not just
> > > "I need X amount of RAM" but also "I want all my virtual machines
> > > and containers running on the same high-performance network backend
> > > and have these two nodes on the same bare-metal machine or at least
> > > in the same
> > data
> > > center rack with a NUMA interconnect, and I also don't want all this
> > > to cost more that $100 per hour." That's not crazy: these are real
> > > world requirements for high-performance VNFs and service chaining.
> > > Resource orchestration requires a full map of what is available in
> > > the data
> > centers,
> > > a negotiation-based algorithm for properly allocating and placing
> > > resources, connection to billing services, etc. Of course resource
> > > orchestration is not within the scope of ARIA, but it would be great
> > > for ARIA to have plugins for them (and TOSCA be able to model
> > > resource requirement policies) when these become available.
> > >
> > >
> > >
> > > On Fri, Aug 11, 2017 at 3:02 PM, DeWayne Filppi
> > > <dewayne@cloudify.co>
> > > wrote:
> > >
> > > > For the "resource realization" part, I was not even considering
> > > > multicloud/vim.   I was considering single cloud even outside of
> > > > composition.  Just reqs and caps.  If my node "requires" a compute
> > > > node with Suse Linux version X with a minimum of 4GB RAM, how does
> the
> > > > orchestrator match that without querying the cloud via the plugin?
>  If
> > > it
> > > > does query it, which it seems it must in addition to the implicit
> > > > quota query, how is this done?  TOSCA seems to not really care,
> > > > which is fine
> > > and
> > > > perhaps a good idea.  But ARIA has to care.
> > > >
> > > > DeWayne
> > > >
> > > > On Fri, Aug 11, 2017 at 12:34 PM, Tal Liron <tal@cloudify.co> wrote:
> > > >
> > > > > In my opinion, the new composite service should indeed be a
> > > > > single
> > > > service,
> > > > > so the ARIA CLI will show it as one (if a substituting service
> > already
> > > > > existed, it would be "dissolved" into the new top-level
> > > > > service). The composition will show its traces if you look more
> > > > > deeply, because
> > > you'll
> > > > > see that some node templates come from a different service
> template.
> > > > > Perhaps our CLI can detect this and tell the user which service
> > > templates
> > > > > were used to create the nodes in current service.
> > > > >
> > > > > The rest of what you described I think is not related directly
> > > > > to
> > > service
> > > > > composition, but resource realization. The way we currently
> > > > > handle
> > this
> > > > is
> > > > > by being very explicit: you have to use the derived Compute
> > > > > types
> > > > included
> > > > > in our AWS or Openstack plugins, for example.
> > > > >
> > > > > I think we should still keep this, because sometimes you want to
> > > > > be
> > > very
> > > > > explicit (and use specific features of the platform), but I
> > > > > actually
> > > have
> > > > > some ideas for multi-VIM support that would work differently:
> > > > > the
> > idea
> > > is
> > > > > to use the basic Compute type, with detailed properties in its
> > > Container
> > > > > and OperatingSystem capabilities. There would then be a policy,
> > perhaps
> > > > > aria.Realization, that would hint at what kind of Compute nodes
> > > > > you
> > are
> > > > > looking for (you can apply the policy to specific nodes or
> > > > > groups, or generally). It might be possible to still have strong
> > platform-specific
> > > > > feature support here, perhaps by implementing a general string
> > > key-value
> > > > > dict that could include hints for the specific plugin to use.
> > > > >
> > > > > Anyway, just ideas at this point...
> > > > >
> > > > >
> > > > > On Fri, Aug 11, 2017 at 2:25 PM, DeWayne Filppi
> > > > > <dewayne@cloudify.co
> > >
> > > > > wrote:
> > > > >
> > > > > > Good stuff.  Obviously a "fancy include" was the wrong metaphor.
> > The
> > > > > > lifecycles are linked though.  When a create the referencing
> > service
> > > > > (aria
> > > > > > service create), ARIA will match the other service template,
> > > > > > and
> > run
> > > > > create
> > > > > > service on it, and then continue on.  Uninstall of the
> > > > > > referencer
> > > will
> > > > > > destroy the referenced service.  After such an instantiation,
> > > > > > would
> > > > > listing
> > > > > > all services produce two (perhaps with some parent indicator)
> > > > > > or
> > one?
> > > > > From
> > > > > > what I've seen, reqs and caps don't have the equivalent of a
> > > lifecycle
> > > > > > interface associated.  Maybe I missed it.  It is implicit in
> > > > > > the orchestrator to match stuff up at service creation time,
> > > > > > I'm
> > > assuming,
> > > > > and
> > > > > > using plugins somehow?  The sort of flagship scenario is
> > > > > > matching
> > > cloud
> > > > > > compute resources without specifying an image.  Since
> > > > > > discovering
> > > > > whether a
> > > > > > particular cloud has a given capability, I guessing a plugin
> > > > > > for
> > that
> > > > > cloud
> > > > > > has to search cloud inventory and perhaps quota to provide
> > > > > > info for
> > > the
> > > > > > match.  Since plugins (at least the Cloudify variety) tend to
> > > > > > be
> > > > > triggered
> > > > > > due to operations that they connect to lifecycle methods, and
> > > > > > there
> > > is
> > > > no
> > > > > > "materialize_capabilities" interface that I noticed, some
> > > > > > other
> > kind
> > > of
> > > > > > magic must be performed.  Maybe this is an
> > > > > > orchestrator-specific
> > > detail
> > > > > > outside of TOSCA.
> > > > > >
> > > > > > DeWayne
> > > > > >
> > > > > > On Fri, Aug 11, 2017 at 10:57 AM, Tal Liron <tal@cloudify.co>
> > wrote:
> > > > > >
> > > > > > > OK, let me try to organize this differently. Three potential
> > > > > conceptions
> > > > > > > here:
> > > > > > >
> > > > > > > 1) A "fancy include," as you say. All that would happen
here
> > > > > > > is
> > > that
> > > > > the
> > > > > > > TOSCA parser would automatically find the service template
> > > > > > > to
> > > include
> > > > > > from
> > > > > > > some kind of repository of recognized service templates,
and
> > > > > > > just
> > > > > include
> > > > > > > that. The language in the TOSCA spec suggests that this
is
> > > > > > > not
> > the
> > > > > > > intention: it is something that happens at the "orchestrator,"
> > not
> > > > the
> > > > > > > "parser."
> > > > > > >
> > > > > > > 2) Static service composition. This happens not at the
> > > > > > > parsing
> > > stage,
> > > > > but
> > > > > > > rather the instantiation stage, where reqs-and-caps happens.
> > > > > > > I
> > > think
> > > > > this
> > > > > > > is what is intended: substitution mapping is specifically
> > > > > > > about
> > > > mapping
> > > > > > > reqs-and-caps. And this is also where things like
> > > > > > > scalability and
> > > > > > placement
> > > > > > > happen: think of a requirement matching a capability, but
> > > > > > > that
> > > > > capability
> > > > > > > not having enough capacity. So, the node might be able
to
> > > > > > > scale
> > > out:
> > > > in
> > > > > > the
> > > > > > > case of substitution, this would mean duplicating an entire
> > service
> > > > > > > instance. My understanding is that this is what is intended
> > > > > > > by
> > > TOSCA,
> > > > > and
> > > > > > > it's entirely within the scope of what we can do. We've
> > > > > > > recently
> > > just
> > > > > > > refactored the instantiation phase into a new "topology"
> > > > > > > module
> > > where
> > > > > > > exactly all this logic is concentrated. So think of it
this
> > > > > > > way
> > --
> > > > ARIA
> > > > > > has
> > > > > > > three parts: "parser," "topology manager," and "workflow
> engine"
> > > > > > > ("orchestrator").
> > > > > > >
> > > > > > > (I think I might have confused some thing here a bit when
> > > mentioning
> > > > a
> > > > > > > "logical proxy node." I do not mean an actual piece of
proxy
> > > software
> > > > > > > running on a machine somewhere. I mean just a data point
in
> > > > > > > the ARIA-generated topology that can be used as a barrier
of
> > > > > > > sorts
> > when
> > > > > > > constructing the task graph -- because the task graph
> > > > > > > follows relationships, the edges of the topology. It could
> > > > > > > be that we
> > > > discover
> > > > > in
> > > > > > > our POC that this is not needed, because actually the
> > substitution
> > > > node
> > > > > > is
> > > > > > > already part of our topology data model and we might be
able
> > > > > > > to
> > > > easily
> > > > > > take
> > > > > > > that into account when generating the task graph.)
> > > > > > >
> > > > > > > 3) Live service composition. I think this is a fantasy
of
> > > > > > > some
> > > > people:
> > > > > > that
> > > > > > > ARIA would be able to take existing, "running" service
> > > > > > > chains and
> > > run
> > > > > > > workflows with them. I do think this is a solvable problem,
> > > > > > > but
> > not
> > > > via
> > > > > > > substitution mapping per se. A solution could involve
> > > > > > > deployment
> > > of a
> > > > > > proxy
> > > > > > > service (which could actually be encapsulated in a
> > > > > > > substitution
> > > > > mapping,
> > > > > > > but doesn't have to be), or configuring specialized virtual
> > > > > > > ports
> > > via
> > > > > an
> > > > > > > SDN controller, or via logical proxy nodes created via
an
> > > inspection
> > > > > > tool,
> > > > > > > etc. I cannot believe that there is a one-size-fits-all
> > > > > > > solution
> > to
> > > > > this
> > > > > > > problem. The dream of NFV might be to have everything
> > > > > > > connect to
> > > each
> > > > > > other
> > > > > > > like LEGO blocks, but there are far too many protocols,
> > > > configuration,
> > > > > > and
> > > > > > > security standards. I think point #2, what I called "static
> > service
> > > > > > > composition," is a realistic compromise and within the
scope
> > > > > > > of
> > > what
> > > > > > TOSCA
> > > > > > > promises.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 11, 2017 at 12:08 PM, DeWayne Filppi <
> > > > dewayne@cloudify.co>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > To my eyes, the spec doesn't speak of runtime service
> > > substituion,
> > > > > but
> > > > > > > > parsetime template composition.  IOW, substitution
mapping
> > > > > > > > is a
> > > > fancy
> > > > > > > > "include", or the equivalent of an interface definition.
> > > > > > > > Is it
> > > > > > > understood
> > > > > > > > by the ARIA team that this includes proxying of running
> > services?
> > > > > IOW,
> > > > > > > if
> > > > > > > > my template requires a database service that my template
> > > > > > > > does
> > > *not*
> > > > > > want
> > > > > > > to
> > > > > > > > control the lifecycle of, I can "substitution map"
an
> > > > > > > > instance
> > > of a
> > > > > > > > template (i.e. a running service)?  This would be
a lovely
> > > feature,
> > > > > but
> > > > > > > > it's not really a "substitution map", rather more
of a
> > > > > > > > "service
> > > > > proxy"
> > > > > > > (as
> > > > > > > > implemented as a plugin in Cloudify).   Just trying
to
> clarify.
> > > > > Maybe
> > > > > > > the
> > > > > > > > community thinks that "substitution map" as something
that
> > occurs
> > > > > > beyond
> > > > > > > > parsetime, or should.
> > > > > > > >
> > > > > > > > On Fri, Aug 11, 2017 at 9:52 AM, Tal Liron
> > > > > > > > <tal@cloudify.co>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Well, DJ, it's all just opinion at this stage
and
> > > > > > > > > feedback is
> > > > > > welcome!
> > > > > > > > >
> > > > > > > > > Option 1:
> > > > > > > > > >         To look at satisfying nodes present
in a
> > substituting
> > > > > > > service,
> > > > > > > > > > Have these nodes part of the newly created
service and
> > remove
> > > > the
> > > > > > > > > > substituting service(nodes with different
ID's. Also
> > > > > > > > > > we are
> > > > very
> > > > > > much
> > > > > > > > in
> > > > > > > > > > favor of      UUID )
> > > > > > > > > >         With this approach I guess the substituting
> > > > > > > > > > service
> > > > > should
> > > > > > > not
> > > > > > > > > > have any associated workflows running. If
at all an
> > workflow
> > > > > > > execution
> > > > > > > > is
> > > > > > > > > > already triggered I hope this service will
not be
> > considered
> > > > for
> > > > > > > > > > substitution.
> > > > > > > > > >         I hope this is the correct approach
when we
> > > > > > > > > > are
> > > looking
> > > > > at
> > > > > > a
> > > > > > > > > > service for the substitution
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, this is a good idea. It would be easy to
discover
> > > > > > > > > this
> > > > > according
> > > > > > > to
> > > > > > > > > the stored node state -- just make sure that
all nodes
> > > > > > > > > are
> > in a
> > > > > > stable
> > > > > > > > > state before composition.
> > > > > > > > >
> > > > > > > > > This leads to a general issue: before composition,
the
> > > > substituting
> > > > > > > > > services must be validated in some way before
> > > > > > > > > composition
> > > begins.
> > > > > > > > >
> > > > > > > > > Also, let's all start using TOSCA terminology
here: the
> > > > containing
> > > > > > > > service
> > > > > > > > > is called the "top-level service," and the contained
> > > > > > > > > services
> > > are
> > > > > > > called
> > > > > > > > > "substituting services."
> > > > > > > > >
> > > > > > > > > Also, we keep using trivial examples, but it's
> > > > > > > > > definitely
> > > > possible
> > > > > > for
> > > > > > > a
> > > > > > > > > top-level service to require several substituting
> > > > > > > > > services at
> > > the
> > > > > > same
> > > > > > > > > time. I can definitely see such things happening
in NFV
> > > > > > > > > with
> > > even
> > > > > > > simple
> > > > > > > > > service chains. Basically every VNF could be
a
> > > > > > > > > substituting
> > > > > service.
> > > > > > > > >
> > > > > > > > > So, actually one of the validations would be
to make
> > > > > > > > > sure you
> > > do
> > > > > not
> > > > > > > > create
> > > > > > > > > circular composition: if the top-level also has
> > > > > > > > > substitution
> > > > > > mappings,
> > > > > > > > you
> > > > > > > > > need to make sure that one of the substituting
ones
> > > > > > > > > doesn't
> > > > require
> > > > > > it.
> > > > > > > > :)
> > > > > > > > > Not a very likely situation, but it would lead
to failure.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Option 2:
> > > > > > > > > >         While creating a service look at
the req-caps
> > > > > > > > > > at
> > the
> > > > > > > > > > service-template level and create a service
including
> > > > > > > > > > the
> > > nodes
> > > > > > > > provided
> > > > > > > > > by
> > > > > > > > > > the substituting service-template. With
this approach
> > > > > > > > > > there
> > > > would
> > > > > > not
> > > > > > > > be
> > > > > > > > > > any   service created from the service-template
which is
> > > > > providing
> > > > > > > the
> > > > > > > > > > substitution functionality. The service-template
would
> > remain
> > > > the
> > > > > > > same
> > > > > > > > > but
> > > > > > > > > > only the service would be added with extra
nodes.
> > > > > > > > > >
> > > > > > > > > > Are you considering both option 1 &
2 for the
> > implementation
> > > ?
> > > > If
> > > > > > not
> > > > > > > > > > which one do you feel which take priority.
I see
> > > > > > > > > > option 2
> > at
> > > > this
> > > > > > > stage
> > > > > > > > > > could be the best possible approach Also
could you
> > > > > > > > > > please let me know a tentative time for
this
> > > > > feature
> > > > > > > to
> > > > > > > > be
> > > > > > > > > > available?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think both options 1 and 2 make sense and are
useful,
> > > > > > > > > and
> > > > > actually
> > > > > > > one
> > > > > > > > is
> > > > > > > > > a subset of the other.
> > > > > > > > >
> > > > > > > > > With option #2 (substituting a service template),
it
> > > > > > > > > means
> > new
> > > > > nodes
> > > > > > > are
> > > > > > > > > instantiated and the composed service would include
all
> > nodes.
> > > > So,
> > > > > an
> > > > > > > > > "install" workflow would install everything at
once. In
> > > > > > > > > this
> > > case
> > > > > we
> > > > > > do
> > > > > > > > > need to fix the lifecycle workflows to be "boundary
aware,"
> > so
> > > > that
> > > > > > > > > workflows of substituting service nodes are part
of
> > > > > > > > > their own
> > > > task
> > > > > > > > graph. I
> > > > > > > > > think that possibly using a logical proxy node
in
> > > > > > > > > between
> > might
> > > > > solve
> > > > > > > > this
> > > > > > > > > situation automatically.
> > > > > > > > >
> > > > > > > > > With option #1 (substituting a service) the substituting
> > nodes
> > > > > might
> > > > > > > > > already be installed. Or not (they might have
been
> > instantiated
> > > > but
> > > > > > > still
> > > > > > > > > not installed). So, the lifecycle workflows should
only
> > > > > > > > > work
> > on
> > > > the
> > > > > > > nodes
> > > > > > > > > that have not yet been installed.
> > > > > > > > >
> > > > > > > > > The point is that we just need to beef up the
lifecycle
> > > workflows
> > > > > to
> > > > > > > > > properly work with boundaries and with a mix
of nodes in
> > > > different
> > > > > > > > states.
> > > > > > > > > If they can do that, then they can handle any
kind of
> > > > > > > > > service
> > > > > > > > composition,
> > > > > > > > > whether it's option #1 or option #2.
> > > > > > > > >
> > > > > > > > > I don't think we can provide a timeline yet,
but I will
> > > > > > > > > say
> > > that
> > > > we
> > > > > > are
> > > > > > > > in
> > > > > > > > > the research phase and may have a POC soon. Avia
is in
> > > > > > > > > charge
> > > of
> > > > > this
> > > > > > > > JIRA,
> > > > > > > > > so I will let him chime in with the current state
of
> things.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message