ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Speidel <jspei...@hortonworks.com>
Subject Re: About extensibility of Ambari stacks (not inheritance)
Date Fri, 05 Dec 2014 01:08:29 GMT
Cos,

Yes, if you are able to push a new stack to the ambari server hosts
filesystem, you can use a REST api call to update the cluster to be
associated with the new stack.

So, the steps would would look like this:
- install cluster with existing stack
- create new stack with additional services/components
- write the new stack to the filesystem of the ambari server host
- bounce ambari server
  -- after the server restart, the hash of the stacks directory is
recalculated which will result in the new updated stack to be pushed to the
agents (not immediate, the check of the stack dir hash occurs every n(not
sure of exact value) heartbeats from the agents)
- do an update (PUT)  on the cluster resource changing the value of the
Clusters/version property to point to the new stack

When these steps are completed the new service/components should be
recognized by ambari.

-John


On Thu, Dec 4, 2014 at 6:07 PM, Konstantin Boudnik <cos@apache.org> wrote:

> Hi John.
>
> Thanks for more info. I'd like to pry more for a little bit of information
> (in-lined):
>
> On Thu, Dec 04, 2014 at 04:19PM, John Speidel wrote:
> > Hi Cos,
> > As mentioned above, it isn't currently possible to augment an existing
> > stack at runtime with new services/components.  You could possibly change
> > the stack associated with a running cluster to a new stack which contains
> > the required services/components, but this new stack definition would
> need
> > to be available to ambari so it really isn't very dynamic.  We understand
>
> Let me describe this scenario as I understood this. Let's say I have two
> stacks:
>   - StackA (with usual set of components), currently installed on a cluster
>   - StackB == StackA + ComponentX
> So, if I add StackB definition to the Ambari server and restart it, will I
> be
> able to switch from StackA to StackB _without_ resintalling the cluster,
> and
> then add/configure ComponentX's service?
>
> If my depiction is correct then it might be an acceptable workaround in the
> absence of dynamic extensibility.
>
> > that going forward that the stack framework will need to be more
> extensible
> > and dynamic as additional stack definitions are created.  Some of the
> > underlying work to make stack processing more flexible has already been
> > done in anticipation of new requirements in this area.
>
> Shivaji Dutta, in an offline email exchange, pointed me to AMBARI-7175 and
> AMBARI-7201, which is seemingly going into the right direction, but still
> would require a stack re-definition, if I read it right.
>
> > This is actually the second time that this scenario has came up for me
> > today so it would be nice to get a Jira filed for an enhancement where
> the
> > community can start to discuss the associated use cases.
>
> I will start a JIRA and put together the set of requirements.
>
> Regards,
>   Cos
>
> > -John
> >
> > On Tue, Dec 2, 2014 at 2:00 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
> >
> > > Thanks for chiming in, Erin! Do you know if there's any plan to have
> > > dynamic
> > > extensibility for services? For instance, Cloudera Manager has this
> > > concept of
> > > Custom Services. And while it was done as an added late fix for
> parcels, it
> > > provides a way of adding new software components into an existing
> Hadoop
> > > cluster. Which seems to be a pretty handy concept, considering the
> liquid
> > > state of the whole Hadoop ecosystem.
> > >
> > > It seems that this functionality, if desired, is going to be brand new
> and
> > > as
> > > such require not just development but also re-consideration at the
> design
> > > level. Am I right? Appreciate the thoughts!
> > >
> > > The best part of active-active masters paradigm is that a client can
> work
> > > with
> > > either of them without a concern of which one is ahead or behind of the
> > > rest
> > > as they are equal. Essentially this solves some major issues that
> pester
> > > active-standby solutions.
> > >
> > > And yes, we do install the services to the cluster's nodes as you
> > > suggested in
> > > the later part of your email - after all they are just Linux daemons
> > > (although
> > > Ambari tries to hide the fact for whatever reason that is). However
> this
> > > presents the core of the very issue: Ambari isn't aware about services
> > > installed outside of its realm, and it doesn't provide any monitoring
> or
> > > life-cycle management for those. And because of that I am trying to
> figure
> > > out
> > > what can be done in this regards to have this dynamism.
> > >
> > > With regards,
> > >   Cos
> > >
> > > On Tue, Dec 02, 2014 at 08:41AM, Erin Boyd wrote:
> > > > Hi Cos,
> > > > Currently, adding services dynamically is not supported in Ambari.
> It is
> > > > generally done, as you mentioned through the creation of a stack
> with the
> > > > services definition defined within in.  I also don't believe we
> currently
> > > > support multiple masters on the same cluster.  How would one
> > > deferientiate
> > > > which master it should be using in such an environment?
> > > >
> > > > Of course services can be installed independent of Ambari on the
> hosts
> > > > inside the cluster...but I don't think that is what you are getting
> at.
> > > Are
> > > > you expecting the services to have a presence on the UI though
> installed
> > > > outside of Ambari?
> > > >
> > > > Erin
> > > >
> > > > ----- Original Message -----
> > > > From: "Konstantin Boudnik" <cos@apache.org>
> > > > To: dev@ambari.apache.org
> > > > Sent: Monday, December 1, 2014 6:42:30 PM
> > > > Subject: About extensibility of Ambari stacks (not inheritance)
> > > >
> > > > Guys,
> > > >
> > > > I am looking into possible stack extensibility properties of Ambari
> (not
> > > to be
> > > > confused with inheritance), but haven't been able to derive any final
> > > > conclusions just yet. Hence, I'd appreciate the input from the people
> > > behind
> > > > the system.
> > > >
> > > > I have a few questions about current state of the Ambari (version
> 1.6.1
> > > and
> > > > coming 1.7, and possibly later?) with regards to ability to expand an
> > > existing
> > > > stack definitions with a 3rd party services and do it in the runtime,
> > > rather
> > > > than only during the installation. We need to be able to run a
> multiple
> > > > instances of our master service in the cluster, which isn't typical
> for
> > > > "normal" Hadoop concept where only one master can exist for any
> giving
> > > > service.
> > > >
> > > > Our use case is to be able to amend an exiting cluster setup (HDP,
> > > Bigtop,
> > > > etc.) with a new service running on top of HDFS; but not to
> reinstall the
> > > > whole stack. The reason we have the use case is that oftentimes our
> > > software
> > > > is being added to an existing 3rd party environment, as an added
> bonus,
> > > not
> > > > available during the initial planning and setup of the cluster.
> > > >
> > > > The way I understand Ambari's stack inheritance is that a scion stack
> > > will
> > > > be a brand new entity, e.g. I won't be able to cherry-pick services
> from
> > > it
> > > > and add them to the already installed parent stack's cluster, right?
> > > >
> > > > So far from I see it doesn't seem possible without introducing a
> > > brand-new
> > > > version of a stack e.g. 'stack inheritance'. Which, unfortunately,
> won't
> > > work
> > > > as per my explanation of the use above case. I guess another way to
> look
> > > at it
> > > > is this: would it be possible to add a component (or a service) that
> will
> > > > override an existing component or a service, but without a need to
> > > reinstall
> > > > the rest of the stack?
> > > >
> > > > Thanks in advance for any info/ideas.
> > > >   Cos
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message