ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: About extensibility of Ambari stacks (not inheritance)
Date Wed, 10 Dec 2014 02:42:34 GMT
Thanks Nate - that does the trick. With the command-line upgrade + a curl call
to add my new service now I can see it among
  http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2/services

The issues I still see are:
 - the service doesn't appear in "Add Service" wizard
 - even after getting added by POST'ing
   {"ServiceInfo":{"service_name":"NSNN"}} info, I don't see the service in
   the dashboard. It might be caused by some inconsistencies in my service
   definition as I can use the same procedure to add Tez service, and being
   able to see it in the dashboard.

Will dig a bit more into this tomorrow and report back if I hit anything new.

Thank you all guys for your help - quite appreciated!
  Cos

On Mon, Dec 08, 2014 at 05:11PM, Nate Cole wrote:
> Cos,
> 
> There is an ambari server command that should allow you to switch the version.  Here
are the steps I took to change the cluster version:
> 
> 1. Install a cluster as normal (I used cluster HDP-2.1 as the stack).
> 2. In /var/lib/ambari-server/resources/stacks/HDP, copy 2.1 to 2.1.TestStack
> 3. Shutdown ambari.
> 4. ran:  ambari-server upgradestack HDP-2.1.TestStack
> 5. Start ambari.
> 6. On the cluster admin page, verified at the top of the page that the cluster says "HDP-2.1.TestStack"
> 
> Hope that helps,
> Nate
> 
> On Dec 8, 2014, at 4:24 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > On Mon, Dec 08, 2014 at 04:14PM, John Speidel wrote:
> >> Cos,
> >> 
> >> I don't know why this is occurring.  I have not actually ever changed the
> >> stack of a running cluster, my response was a result of my understanding
> >> and talking to others that have done this.
> >> 
> >> Hopefully somebody that has actually done this can comment.
> >> In the meantime, I will see if I can find somebody that may may know why
> >> this isn't working for you.
> > 
> > Thank you very much - really appreciate your help! In the interest of full
> > disclosure: I am using 1.6.1 server, not sure if this is of any significance.
> > 
> > Cos
> > 
> >> On Fri, Dec 5, 2014 at 8:38 PM, Konstantin Boudnik <cos@apache.org> wrote:
> >> 
> >>> release Thanks for the info John. I think I am getting there, but not
> >>> completely....
> >>> 
> >>> So, here's what I've done:
> >>> - added service, bounced Ambari server
> >>> 
> >>> - curl to http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/
> >>> says that I have new stack
> >>>    {
> >>>      "href" : "
> >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/versions/2.1.WANdisco
> >>> ",
> >>>      "Versions" : {
> >>>        "stack_name" : "HDP",
> >>>        "stack_version" : "2.1.WANdisco"
> >>>      }
> >>>    }
> >>> 
> >>> - ran update
> >>>  curl -u admin:admin -i -X PUT -H  'X-Requested-By: ambari' -d
> >>> '{"Clusters":{"version":"HDP-2.1.WANdisco"}}'
> >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2
> >>> 
> >>> The result was:
> >>> HTTP/1.1 200 OK
> >>> Set-Cookie: AMBARISESSIONID=vueucwdh8p6mkmcygj803vi9;Path=/
> >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> >>> Content-Type: text/plain
> >>> Content-Length: 0
> >>> Server: Jetty(7.6.7.v20120910)
> >>> 
> >>> However, running
> >>>        curl -u admin:admin
> >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/
> >>> 
> >>> Still gives me this....
> >>> {
> >>>  "href" : "http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/
> >>> ",
> >>>  "items" : [
> >>>    {
> >>>      "href" : "
> >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2",
> >>>      "Clusters" : {
> >>>        "cluster_name" : "DC-2",
> >>>        "version" : "HDP-2.1"
> >>>      }
> >>>    }
> >>>  ]
> >>> 
> >>> Any ideas/hints? Thank you very much for your help!
> >>>  Cos
> >>> 
> >>> On Thu, Dec 04, 2014 at 08:08PM, John Speidel wrote:
> >>>> Cos,
> >>>> 
> >>>> Yes, if you are able to push a new stack to the ambari server hosts
> >>>> filesystem, you can use a REST api call to update the cluster to be
> >>>> associated with the new stack.
> >>>> 
> >>>> So, the steps would would look like this:
> >>>> - install cluster with existing stack
> >>>> - create new stack with additional services/components
> >>>> - write the new stack to the filesystem of the ambari server host
> >>>> - bounce ambari server
> >>>>  -- after the server restart, the hash of the stacks directory is
> >>>> recalculated which will result in the new updated stack to be pushed
to
> >>> the
> >>>> agents (not immediate, the check of the stack dir hash occurs every
n(not
> >>>> sure of exact value) heartbeats from the agents)
> >>>> - do an update (PUT)  on the cluster resource changing the value of
the
> >>>> Clusters/version property to point to the new stack
> >>>> 
> >>>> When these steps are completed the new service/components should be
> >>>> recognized by ambari.
> >>>> 
> >>>> -John
> >>>> 
> >>>> 
> >>>> On Thu, Dec 4, 2014 at 6:07 PM, Konstantin Boudnik <cos@apache.org>
> >>> wrote:
> >>>> 
> >>>>> Hi John.
> >>>>> 
> >>>>> Thanks for more info. I'd like to pry more for a little bit of
> >>> information
> >>>>> (in-lined):
> >>>>> 
> >>>>> On Thu, Dec 04, 2014 at 04:19PM, John Speidel wrote:
> >>>>>> Hi Cos,
> >>>>>> As mentioned above, it isn't currently possible to augment an
> >>> existing
> >>>>>> stack at runtime with new services/components.  You could possibly
> >>> change
> >>>>>> the stack associated with a running cluster to a new stack which
> >>> contains
> >>>>>> the required services/components, but this new stack definition
would
> >>>>> need
> >>>>>> to be available to ambari so it really isn't very dynamic. 
We
> >>> understand
> >>>>> 
> >>>>> Let me describe this scenario as I understood this. Let's say I
have
> >>> two
> >>>>> stacks:
> >>>>>  - StackA (with usual set of components), currently installed on
a
> >>> cluster
> >>>>>  - StackB == StackA + ComponentX
> >>>>> So, if I add StackB definition to the Ambari server and restart
it,
> >>> will I
> >>>>> be
> >>>>> able to switch from StackA to StackB _without_ resintalling the
> >>> cluster,
> >>>>> and
> >>>>> then add/configure ComponentX's service?
> >>>>> 
> >>>>> If my depiction is correct then it might be an acceptable workaround
> >>> in the
> >>>>> absence of dynamic extensibility.
> >>>>> 
> >>>>>> that going forward that the stack framework will need to be
more
> >>>>> extensible
> >>>>>> and dynamic as additional stack definitions are created.  Some
of the
> >>>>>> underlying work to make stack processing more flexible has already
> >>> been
> >>>>>> done in anticipation of new requirements in this area.
> >>>>> 
> >>>>> Shivaji Dutta, in an offline email exchange, pointed me to AMBARI-7175
> >>> and
> >>>>> AMBARI-7201, which is seemingly going into the right direction,
but
> >>> still
> >>>>> would require a stack re-definition, if I read it right.
> >>>>> 
> >>>>>> This is actually the second time that this scenario has came
up for
> >>> me
> >>>>>> today so it would be nice to get a Jira filed for an enhancement
> >>> where
> >>>>> the
> >>>>>> community can start to discuss the associated use cases.
> >>>>> 
> >>>>> I will start a JIRA and put together the set of requirements.
> >>>>> 
> >>>>> Regards,
> >>>>>  Cos
> >>>>> 
> >>>>>> -John
> >>>>>> 
> >>>>>> On Tue, Dec 2, 2014 at 2:00 PM, Konstantin Boudnik <cos@apache.org>
> >>>>> wrote:
> >>>>>> 
> >>>>>>> Thanks for chiming in, Erin! Do you know if there's any
plan to
> >>> have
> >>>>>>> dynamic
> >>>>>>> extensibility for services? For instance, Cloudera Manager
has this
> >>>>>>> concept of
> >>>>>>> Custom Services. And while it was done as an added late
fix for
> >>>>> parcels, it
> >>>>>>> provides a way of adding new software components into an
existing
> >>>>> Hadoop
> >>>>>>> cluster. Which seems to be a pretty handy concept, considering
the
> >>>>> liquid
> >>>>>>> state of the whole Hadoop ecosystem.
> >>>>>>> 
> >>>>>>> It seems that this functionality, if desired, is going to
be brand
> >>> new
> >>>>> and
> >>>>>>> as
> >>>>>>> such require not just development but also re-consideration
at the
> >>>>> design
> >>>>>>> level. Am I right? Appreciate the thoughts!
> >>>>>>> 
> >>>>>>> The best part of active-active masters paradigm is that
a client
> >>> can
> >>>>> work
> >>>>>>> with
> >>>>>>> either of them without a concern of which one is ahead or
behind
> >>> of the
> >>>>>>> rest
> >>>>>>> as they are equal. Essentially this solves some major issues
that
> >>>>> pester
> >>>>>>> active-standby solutions.
> >>>>>>> 
> >>>>>>> And yes, we do install the services to the cluster's nodes
as you
> >>>>>>> suggested in
> >>>>>>> the later part of your email - after all they are just Linux
> >>> daemons
> >>>>>>> (although
> >>>>>>> Ambari tries to hide the fact for whatever reason that is).
However
> >>>>> this
> >>>>>>> presents the core of the very issue: Ambari isn't aware
about
> >>> services
> >>>>>>> installed outside of its realm, and it doesn't provide any
> >>> monitoring
> >>>>> or
> >>>>>>> life-cycle management for those. And because of that I am
trying to
> >>>>> figure
> >>>>>>> out
> >>>>>>> what can be done in this regards to have this dynamism.
> >>>>>>> 
> >>>>>>> With regards,
> >>>>>>>  Cos
> >>>>>>> 
> >>>>>>> On Tue, Dec 02, 2014 at 08:41AM, Erin Boyd wrote:
> >>>>>>>> Hi Cos,
> >>>>>>>> Currently, adding services dynamically is not supported
in
> >>> Ambari.
> >>>>> It is
> >>>>>>>> generally done, as you mentioned through the creation
of a stack
> >>>>> with the
> >>>>>>>> services definition defined within in.  I also don't
believe we
> >>>>> currently
> >>>>>>>> support multiple masters on the same cluster.  How would
one
> >>>>>>> deferientiate
> >>>>>>>> which master it should be using in such an environment?
> >>>>>>>> 
> >>>>>>>> Of course services can be installed independent of Ambari
on the
> >>>>> hosts
> >>>>>>>> inside the cluster...but I don't think that is what
you are
> >>> getting
> >>>>> at.
> >>>>>>> Are
> >>>>>>>> you expecting the services to have a presence on the
UI though
> >>>>> installed
> >>>>>>>> outside of Ambari?
> >>>>>>>> 
> >>>>>>>> Erin
> >>>>>>>> 
> >>>>>>>> ----- Original Message -----
> >>>>>>>> From: "Konstantin Boudnik" <cos@apache.org>
> >>>>>>>> To: dev@ambari.apache.org
> >>>>>>>> Sent: Monday, December 1, 2014 6:42:30 PM
> >>>>>>>> Subject: About extensibility of Ambari stacks (not inheritance)
> >>>>>>>> 
> >>>>>>>> Guys,
> >>>>>>>> 
> >>>>>>>> I am looking into possible stack extensibility properties
of
> >>> Ambari
> >>>>> (not
> >>>>>>> to be
> >>>>>>>> confused with inheritance), but haven't been able to
derive any
> >>> final
> >>>>>>>> conclusions just yet. Hence, I'd appreciate the input
from the
> >>> people
> >>>>>>> behind
> >>>>>>>> the system.
> >>>>>>>> 
> >>>>>>>> I have a few questions about current state of the Ambari
(version
> >>>>> 1.6.1
> >>>>>>> and
> >>>>>>>> coming 1.7, and possibly later?) with regards to ability
to
> >>> expand an
> >>>>>>> existing
> >>>>>>>> stack definitions with a 3rd party services and do it
in the
> >>> runtime,
> >>>>>>> rather
> >>>>>>>> than only during the installation. We need to be able
to run a
> >>>>> multiple
> >>>>>>>> instances of our master service in the cluster, which
isn't
> >>> typical
> >>>>> for
> >>>>>>>> "normal" Hadoop concept where only one master can exist
for any
> >>>>> giving
> >>>>>>>> service.
> >>>>>>>> 
> >>>>>>>> Our use case is to be able to amend an exiting cluster
setup
> >>> (HDP,
> >>>>>>> Bigtop,
> >>>>>>>> etc.) with a new service running on top of HDFS; but
not to
> >>>>> reinstall the
> >>>>>>>> whole stack. The reason we have the use case is that
oftentimes
> >>> our
> >>>>>>> software
> >>>>>>>> is being added to an existing 3rd party environment,
as an added
> >>>>> bonus,
> >>>>>>> not
> >>>>>>>> available during the initial planning and setup of the
cluster.
> >>>>>>>> 
> >>>>>>>> The way I understand Ambari's stack inheritance is that
a scion
> >>> stack
> >>>>>>> will
> >>>>>>>> be a brand new entity, e.g. I won't be able to cherry-pick
> >>> services
> >>>>> from
> >>>>>>> it
> >>>>>>>> and add them to the already installed parent stack's
cluster,
> >>> right?
> >>>>>>>> 
> >>>>>>>> So far from I see it doesn't seem possible without introducing
a
> >>>>>>> brand-new
> >>>>>>>> version of a stack e.g. 'stack inheritance'. Which,
> >>> unfortunately,
> >>>>> won't
> >>>>>>> work
> >>>>>>>> as per my explanation of the use above case. I guess
another way
> >>> to
> >>>>> look
> >>>>>>> at it
> >>>>>>>> is this: would it be possible to add a component (or
a service)
> >>> that
> >>>>> will
> >>>>>>>> override an existing component or a service, but without
a need
> >>> to
> >>>>>>> reinstall
> >>>>>>>> the rest of the stack?
> >>>>>>>> 
> >>>>>>>> Thanks in advance for any info/ideas.
> >>>>>>>>  Cos
> >>>>>>> 
> >>>>>> 
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual
or
> >>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >>> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law.
If the
> >>> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> >>> notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure
or
> >>>>>> forwarding of this communication is strictly prohibited. If
you have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>>> 
> >>>>> 
> >>>> 
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or entity
> >>> to
> >>>> which it is addressed and may contain information that is confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the reader
> >>>> of this message is not the intended recipient, you are hereby notified
> >>> that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> >>> immediately
> >>>> and delete it from your system. Thank You.
> >>> 
> >> 
> >> -- 
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity to

> >> which it is addressed and may contain information that is confidential, 
> >> privileged and exempt from disclosure under applicable law. If the reader 
> >> of this message is not the intended recipient, you are hereby notified that

> >> any printing, copying, dissemination, distribution, disclosure or 
> >> forwarding of this communication is strictly prohibited. If you have 
> >> received this communication in error, please contact the sender immediately

> >> and delete it from your system. Thank You.
> 
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.

Mime
View raw message