ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: About extensibility of Ambari stacks (not inheritance)
Date Sat, 06 Dec 2014 01:38:40 GMT
release Thanks for the info John. I think I am getting there, but not completely....

So, here's what I've done:
- added service, bounced Ambari server

- curl to http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/ says that I have
new stack
    {
      "href" : "http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/versions/2.1.WANdisco",
      "Versions" : {
        "stack_name" : "HDP",
        "stack_version" : "2.1.WANdisco"
      }
    }

- ran update
  curl -u admin:admin -i -X PUT -H  'X-Requested-By: ambari' -d '{"Clusters":{"version":"HDP-2.1.WANdisco"}}'
http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2

The result was:
HTTP/1.1 200 OK
Set-Cookie: AMBARISESSIONID=vueucwdh8p6mkmcygj803vi9;Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Content-Length: 0
Server: Jetty(7.6.7.v20120910)

However, running 
        curl -u admin:admin http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/

Still gives me this....
{
  "href" : "http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/",
  "items" : [
    {
      "href" : "http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2",
      "Clusters" : {
        "cluster_name" : "DC-2",
        "version" : "HDP-2.1"
      }
    }
  ]

Any ideas/hints? Thank you very much for your help!
  Cos

On Thu, Dec 04, 2014 at 08:08PM, John Speidel wrote:
> Cos,
> 
> Yes, if you are able to push a new stack to the ambari server hosts
> filesystem, you can use a REST api call to update the cluster to be
> associated with the new stack.
> 
> So, the steps would would look like this:
> - install cluster with existing stack
> - create new stack with additional services/components
> - write the new stack to the filesystem of the ambari server host
> - bounce ambari server
>   -- after the server restart, the hash of the stacks directory is
> recalculated which will result in the new updated stack to be pushed to the
> agents (not immediate, the check of the stack dir hash occurs every n(not
> sure of exact value) heartbeats from the agents)
> - do an update (PUT)  on the cluster resource changing the value of the
> Clusters/version property to point to the new stack
> 
> When these steps are completed the new service/components should be
> recognized by ambari.
> 
> -John
> 
> 
> On Thu, Dec 4, 2014 at 6:07 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > Hi John.
> >
> > Thanks for more info. I'd like to pry more for a little bit of information
> > (in-lined):
> >
> > On Thu, Dec 04, 2014 at 04:19PM, John Speidel wrote:
> > > Hi Cos,
> > > As mentioned above, it isn't currently possible to augment an existing
> > > stack at runtime with new services/components.  You could possibly change
> > > the stack associated with a running cluster to a new stack which contains
> > > the required services/components, but this new stack definition would
> > need
> > > to be available to ambari so it really isn't very dynamic.  We understand
> >
> > Let me describe this scenario as I understood this. Let's say I have two
> > stacks:
> >   - StackA (with usual set of components), currently installed on a cluster
> >   - StackB == StackA + ComponentX
> > So, if I add StackB definition to the Ambari server and restart it, will I
> > be
> > able to switch from StackA to StackB _without_ resintalling the cluster,
> > and
> > then add/configure ComponentX's service?
> >
> > If my depiction is correct then it might be an acceptable workaround in the
> > absence of dynamic extensibility.
> >
> > > that going forward that the stack framework will need to be more
> > extensible
> > > and dynamic as additional stack definitions are created.  Some of the
> > > underlying work to make stack processing more flexible has already been
> > > done in anticipation of new requirements in this area.
> >
> > Shivaji Dutta, in an offline email exchange, pointed me to AMBARI-7175 and
> > AMBARI-7201, which is seemingly going into the right direction, but still
> > would require a stack re-definition, if I read it right.
> >
> > > This is actually the second time that this scenario has came up for me
> > > today so it would be nice to get a Jira filed for an enhancement where
> > the
> > > community can start to discuss the associated use cases.
> >
> > I will start a JIRA and put together the set of requirements.
> >
> > Regards,
> >   Cos
> >
> > > -John
> > >
> > > On Tue, Dec 2, 2014 at 2:00 PM, Konstantin Boudnik <cos@apache.org>
> > wrote:
> > >
> > > > Thanks for chiming in, Erin! Do you know if there's any plan to have
> > > > dynamic
> > > > extensibility for services? For instance, Cloudera Manager has this
> > > > concept of
> > > > Custom Services. And while it was done as an added late fix for
> > parcels, it
> > > > provides a way of adding new software components into an existing
> > Hadoop
> > > > cluster. Which seems to be a pretty handy concept, considering the
> > liquid
> > > > state of the whole Hadoop ecosystem.
> > > >
> > > > It seems that this functionality, if desired, is going to be brand new
> > and
> > > > as
> > > > such require not just development but also re-consideration at the
> > design
> > > > level. Am I right? Appreciate the thoughts!
> > > >
> > > > The best part of active-active masters paradigm is that a client can
> > work
> > > > with
> > > > either of them without a concern of which one is ahead or behind of the
> > > > rest
> > > > as they are equal. Essentially this solves some major issues that
> > pester
> > > > active-standby solutions.
> > > >
> > > > And yes, we do install the services to the cluster's nodes as you
> > > > suggested in
> > > > the later part of your email - after all they are just Linux daemons
> > > > (although
> > > > Ambari tries to hide the fact for whatever reason that is). However
> > this
> > > > presents the core of the very issue: Ambari isn't aware about services
> > > > installed outside of its realm, and it doesn't provide any monitoring
> > or
> > > > life-cycle management for those. And because of that I am trying to
> > figure
> > > > out
> > > > what can be done in this regards to have this dynamism.
> > > >
> > > > With regards,
> > > >   Cos
> > > >
> > > > On Tue, Dec 02, 2014 at 08:41AM, Erin Boyd wrote:
> > > > > Hi Cos,
> > > > > Currently, adding services dynamically is not supported in Ambari.
> > It is
> > > > > generally done, as you mentioned through the creation of a stack
> > with the
> > > > > services definition defined within in.  I also don't believe we
> > currently
> > > > > support multiple masters on the same cluster.  How would one
> > > > deferientiate
> > > > > which master it should be using in such an environment?
> > > > >
> > > > > Of course services can be installed independent of Ambari on the
> > hosts
> > > > > inside the cluster...but I don't think that is what you are getting
> > at.
> > > > Are
> > > > > you expecting the services to have a presence on the UI though
> > installed
> > > > > outside of Ambari?
> > > > >
> > > > > Erin
> > > > >
> > > > > ----- Original Message -----
> > > > > From: "Konstantin Boudnik" <cos@apache.org>
> > > > > To: dev@ambari.apache.org
> > > > > Sent: Monday, December 1, 2014 6:42:30 PM
> > > > > Subject: About extensibility of Ambari stacks (not inheritance)
> > > > >
> > > > > Guys,
> > > > >
> > > > > I am looking into possible stack extensibility properties of Ambari
> > (not
> > > > to be
> > > > > confused with inheritance), but haven't been able to derive any final
> > > > > conclusions just yet. Hence, I'd appreciate the input from the people
> > > > behind
> > > > > the system.
> > > > >
> > > > > I have a few questions about current state of the Ambari (version
> > 1.6.1
> > > > and
> > > > > coming 1.7, and possibly later?) with regards to ability to expand
an
> > > > existing
> > > > > stack definitions with a 3rd party services and do it in the runtime,
> > > > rather
> > > > > than only during the installation. We need to be able to run a
> > multiple
> > > > > instances of our master service in the cluster, which isn't typical
> > for
> > > > > "normal" Hadoop concept where only one master can exist for any
> > giving
> > > > > service.
> > > > >
> > > > > Our use case is to be able to amend an exiting cluster setup (HDP,
> > > > Bigtop,
> > > > > etc.) with a new service running on top of HDFS; but not to
> > reinstall the
> > > > > whole stack. The reason we have the use case is that oftentimes our
> > > > software
> > > > > is being added to an existing 3rd party environment, as an added
> > bonus,
> > > > not
> > > > > available during the initial planning and setup of the cluster.
> > > > >
> > > > > The way I understand Ambari's stack inheritance is that a scion stack
> > > > will
> > > > > be a brand new entity, e.g. I won't be able to cherry-pick services
> > from
> > > > it
> > > > > and add them to the already installed parent stack's cluster, right?
> > > > >
> > > > > So far from I see it doesn't seem possible without introducing a
> > > > brand-new
> > > > > version of a stack e.g. 'stack inheritance'. Which, unfortunately,
> > won't
> > > > work
> > > > > as per my explanation of the use above case. I guess another way
to
> > look
> > > > at it
> > > > > is this: would it be possible to add a component (or a service) that
> > will
> > > > > override an existing component or a service, but without a need to
> > > > reinstall
> > > > > the rest of the stack?
> > > > >
> > > > > Thanks in advance for any info/ideas.
> > > > >   Cos
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
> >
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.

Mime
View raw message