ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: About extensibility of Ambari stacks (not inheritance)
Date Fri, 12 Dec 2014 00:22:26 GMT
And it seems to be working in 1.7. 

Thanks everyone for your help - now into more detailed service definition.

Cos

On Tue, Dec 09, 2014 at 08:45PM, Konstantin Boudnik wrote:
> Ah! Very timely remark indeed! Will spin a new one in the morning and have it
> a go. Thanks!
> 
> On Tue, Dec 09, 2014 at 07:25PM, Yusaku Sako wrote:
> > Hi Konstantin,
> > 
> > Custom services added via the API won't automatically show up in the UI if
> > you are using Ambari 1.6.1 or below.
> > Please try with Ambari 1.7.0.
> > 
> > Thanks,
> > Yusaku
> > 
> > 
> > On Tue, Dec 9, 2014 at 6:42 PM, Konstantin Boudnik <cos@apache.org> wrote:
> > 
> > > Thanks Nate - that does the trick. With the command-line upgrade + a curl
> > > call
> > > to add my new service now I can see it among
> > >
> > > http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2/services
> > >
> > > The issues I still see are:
> > >  - the service doesn't appear in "Add Service" wizard
> > >  - even after getting added by POST'ing
> > >    {"ServiceInfo":{"service_name":"NSNN"}} info, I don't see the service in
> > >    the dashboard. It might be caused by some inconsistencies in my service
> > >    definition as I can use the same procedure to add Tez service, and being
> > >    able to see it in the dashboard.
> > >
> > > Will dig a bit more into this tomorrow and report back if I hit anything
> > > new.
> > >
> > > Thank you all guys for your help - quite appreciated!
> > >   Cos
> > >
> > > On Mon, Dec 08, 2014 at 05:11PM, Nate Cole wrote:
> > > > Cos,
> > > >
> > > > There is an ambari server command that should allow you to switch the
> > > version.  Here are the steps I took to change the cluster version:
> > > >
> > > > 1. Install a cluster as normal (I used cluster HDP-2.1 as the stack).
> > > > 2. In /var/lib/ambari-server/resources/stacks/HDP, copy 2.1 to
> > > 2.1.TestStack
> > > > 3. Shutdown ambari.
> > > > 4. ran:  ambari-server upgradestack HDP-2.1.TestStack
> > > > 5. Start ambari.
> > > > 6. On the cluster admin page, verified at the top of the page that the
> > > cluster says "HDP-2.1.TestStack"
> > > >
> > > > Hope that helps,
> > > > Nate
> > > >
> > > > On Dec 8, 2014, at 4:24 PM, Konstantin Boudnik <cos@apache.org>
wrote:
> > > >
> > > > > On Mon, Dec 08, 2014 at 04:14PM, John Speidel wrote:
> > > > >> Cos,
> > > > >>
> > > > >> I don't know why this is occurring.  I have not actually ever
changed
> > > the
> > > > >> stack of a running cluster, my response was a result of my
> > > understanding
> > > > >> and talking to others that have done this.
> > > > >>
> > > > >> Hopefully somebody that has actually done this can comment.
> > > > >> In the meantime, I will see if I can find somebody that may may
know
> > > why
> > > > >> this isn't working for you.
> > > > >
> > > > > Thank you very much - really appreciate your help! In the interest
of
> > > full
> > > > > disclosure: I am using 1.6.1 server, not sure if this is of any
> > > significance.
> > > > >
> > > > > Cos
> > > > >
> > > > >> On Fri, Dec 5, 2014 at 8:38 PM, Konstantin Boudnik <cos@apache.org>
> > > wrote:
> > > > >>
> > > > >>> release Thanks for the info John. I think I am getting there,
but not
> > > > >>> completely....
> > > > >>>
> > > > >>> So, here's what I've done:
> > > > >>> - added service, bounced Ambari server
> > > > >>>
> > > > >>> - curl to
> > > http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/
> > > > >>> says that I have new stack
> > > > >>>    {
> > > > >>>      "href" : "
> > > > >>>
> > > http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/versions/2.1.WANdisco
> > > > >>> ",
> > > > >>>      "Versions" : {
> > > > >>>        "stack_name" : "HDP",
> > > > >>>        "stack_version" : "2.1.WANdisco"
> > > > >>>      }
> > > > >>>    }
> > > > >>>
> > > > >>> - ran update
> > > > >>>  curl -u admin:admin -i -X PUT -H  'X-Requested-By: ambari'
-d
> > > > >>> '{"Clusters":{"version":"HDP-2.1.WANdisco"}}'
> > > > >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2
> > > > >>>
> > > > >>> The result was:
> > > > >>> HTTP/1.1 200 OK
> > > > >>> Set-Cookie: AMBARISESSIONID=vueucwdh8p6mkmcygj803vi9;Path=/
> > > > >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> > > > >>> Content-Type: text/plain
> > > > >>> Content-Length: 0
> > > > >>> Server: Jetty(7.6.7.v20120910)
> > > > >>>
> > > > >>> However, running
> > > > >>>        curl -u admin:admin
> > > > >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/
> > > > >>>
> > > > >>> Still gives me this....
> > > > >>> {
> > > > >>>  "href" : "
> > > http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/
> > > > >>> ",
> > > > >>>  "items" : [
> > > > >>>    {
> > > > >>>      "href" : "
> > > > >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2",
> > > > >>>      "Clusters" : {
> > > > >>>        "cluster_name" : "DC-2",
> > > > >>>        "version" : "HDP-2.1"
> > > > >>>      }
> > > > >>>    }
> > > > >>>  ]
> > > > >>>
> > > > >>> Any ideas/hints? Thank you very much for your help!
> > > > >>>  Cos
> > > > >>>
> > > > >>> On Thu, Dec 04, 2014 at 08:08PM, John Speidel wrote:
> > > > >>>> Cos,
> > > > >>>>
> > > > >>>> Yes, if you are able to push a new stack to the ambari
server hosts
> > > > >>>> filesystem, you can use a REST api call to update the
cluster to be
> > > > >>>> associated with the new stack.
> > > > >>>>
> > > > >>>> So, the steps would would look like this:
> > > > >>>> - install cluster with existing stack
> > > > >>>> - create new stack with additional services/components
> > > > >>>> - write the new stack to the filesystem of the ambari
server host
> > > > >>>> - bounce ambari server
> > > > >>>>  -- after the server restart, the hash of the stacks
directory is
> > > > >>>> recalculated which will result in the new updated stack
to be
> > > pushed to
> > > > >>> the
> > > > >>>> agents (not immediate, the check of the stack dir hash
occurs every
> > > n(not
> > > > >>>> sure of exact value) heartbeats from the agents)
> > > > >>>> - do an update (PUT)  on the cluster resource changing
the value of
> > > the
> > > > >>>> Clusters/version property to point to the new stack
> > > > >>>>
> > > > >>>> When these steps are completed the new service/components
should be
> > > > >>>> recognized by ambari.
> > > > >>>>
> > > > >>>> -John
> > > > >>>>
> > > > >>>>
> > > > >>>> On Thu, Dec 4, 2014 at 6:07 PM, Konstantin Boudnik <cos@apache.org>
> > > > >>> wrote:
> > > > >>>>
> > > > >>>>> Hi John.
> > > > >>>>>
> > > > >>>>> Thanks for more info. I'd like to pry more for a
little bit of
> > > > >>> information
> > > > >>>>> (in-lined):
> > > > >>>>>
> > > > >>>>> On Thu, Dec 04, 2014 at 04:19PM, John Speidel wrote:
> > > > >>>>>> Hi Cos,
> > > > >>>>>> As mentioned above, it isn't currently possible
to augment an
> > > > >>> existing
> > > > >>>>>> stack at runtime with new services/components.
 You could possibly
> > > > >>> change
> > > > >>>>>> the stack associated with a running cluster to
a new stack which
> > > > >>> contains
> > > > >>>>>> the required services/components, but this new
stack definition
> > > would
> > > > >>>>> need
> > > > >>>>>> to be available to ambari so it really isn't
very dynamic.  We
> > > > >>> understand
> > > > >>>>>
> > > > >>>>> Let me describe this scenario as I understood this.
Let's say I
> > > have
> > > > >>> two
> > > > >>>>> stacks:
> > > > >>>>>  - StackA (with usual set of components), currently
installed on a
> > > > >>> cluster
> > > > >>>>>  - StackB == StackA + ComponentX
> > > > >>>>> So, if I add StackB definition to the Ambari server
and restart it,
> > > > >>> will I
> > > > >>>>> be
> > > > >>>>> able to switch from StackA to StackB _without_ resintalling
the
> > > > >>> cluster,
> > > > >>>>> and
> > > > >>>>> then add/configure ComponentX's service?
> > > > >>>>>
> > > > >>>>> If my depiction is correct then it might be an acceptable
> > > workaround
> > > > >>> in the
> > > > >>>>> absence of dynamic extensibility.
> > > > >>>>>
> > > > >>>>>> that going forward that the stack framework will
need to be more
> > > > >>>>> extensible
> > > > >>>>>> and dynamic as additional stack definitions are
created.  Some of
> > > the
> > > > >>>>>> underlying work to make stack processing more
flexible has already
> > > > >>> been
> > > > >>>>>> done in anticipation of new requirements in this
area.
> > > > >>>>>
> > > > >>>>> Shivaji Dutta, in an offline email exchange, pointed
me to
> > > AMBARI-7175
> > > > >>> and
> > > > >>>>> AMBARI-7201, which is seemingly going into the right
direction, but
> > > > >>> still
> > > > >>>>> would require a stack re-definition, if I read it
right.
> > > > >>>>>
> > > > >>>>>> This is actually the second time that this scenario
has came up
> > > for
> > > > >>> me
> > > > >>>>>> today so it would be nice to get a Jira filed
for an enhancement
> > > > >>> where
> > > > >>>>> the
> > > > >>>>>> community can start to discuss the associated
use cases.
> > > > >>>>>
> > > > >>>>> I will start a JIRA and put together the set of requirements.
> > > > >>>>>
> > > > >>>>> Regards,
> > > > >>>>>  Cos
> > > > >>>>>
> > > > >>>>>> -John
> > > > >>>>>>
> > > > >>>>>> On Tue, Dec 2, 2014 at 2:00 PM, Konstantin Boudnik
<
> > > cos@apache.org>
> > > > >>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Thanks for chiming in, Erin! Do you know
if there's any plan to
> > > > >>> have
> > > > >>>>>>> dynamic
> > > > >>>>>>> extensibility for services? For instance,
Cloudera Manager has
> > > this
> > > > >>>>>>> concept of
> > > > >>>>>>> Custom Services. And while it was done as
an added late fix for
> > > > >>>>> parcels, it
> > > > >>>>>>> provides a way of adding new software components
into an existing
> > > > >>>>> Hadoop
> > > > >>>>>>> cluster. Which seems to be a pretty handy
concept, considering
> > > the
> > > > >>>>> liquid
> > > > >>>>>>> state of the whole Hadoop ecosystem.
> > > > >>>>>>>
> > > > >>>>>>> It seems that this functionality, if desired,
is going to be
> > > brand
> > > > >>> new
> > > > >>>>> and
> > > > >>>>>>> as
> > > > >>>>>>> such require not just development but also
re-consideration at
> > > the
> > > > >>>>> design
> > > > >>>>>>> level. Am I right? Appreciate the thoughts!
> > > > >>>>>>>
> > > > >>>>>>> The best part of active-active masters paradigm
is that a client
> > > > >>> can
> > > > >>>>> work
> > > > >>>>>>> with
> > > > >>>>>>> either of them without a concern of which
one is ahead or behind
> > > > >>> of the
> > > > >>>>>>> rest
> > > > >>>>>>> as they are equal. Essentially this solves
some major issues that
> > > > >>>>> pester
> > > > >>>>>>> active-standby solutions.
> > > > >>>>>>>
> > > > >>>>>>> And yes, we do install the services to the
cluster's nodes as you
> > > > >>>>>>> suggested in
> > > > >>>>>>> the later part of your email - after all
they are just Linux
> > > > >>> daemons
> > > > >>>>>>> (although
> > > > >>>>>>> Ambari tries to hide the fact for whatever
reason that is).
> > > However
> > > > >>>>> this
> > > > >>>>>>> presents the core of the very issue: Ambari
isn't aware about
> > > > >>> services
> > > > >>>>>>> installed outside of its realm, and it doesn't
provide any
> > > > >>> monitoring
> > > > >>>>> or
> > > > >>>>>>> life-cycle management for those. And because
of that I am trying
> > > to
> > > > >>>>> figure
> > > > >>>>>>> out
> > > > >>>>>>> what can be done in this regards to have
this dynamism.
> > > > >>>>>>>
> > > > >>>>>>> With regards,
> > > > >>>>>>>  Cos
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Dec 02, 2014 at 08:41AM, Erin Boyd
wrote:
> > > > >>>>>>>> Hi Cos,
> > > > >>>>>>>> Currently, adding services dynamically
is not supported in
> > > > >>> Ambari.
> > > > >>>>> It is
> > > > >>>>>>>> generally done, as you mentioned through
the creation of a stack
> > > > >>>>> with the
> > > > >>>>>>>> services definition defined within in.
 I also don't believe we
> > > > >>>>> currently
> > > > >>>>>>>> support multiple masters on the same
cluster.  How would one
> > > > >>>>>>> deferientiate
> > > > >>>>>>>> which master it should be using in such
an environment?
> > > > >>>>>>>>
> > > > >>>>>>>> Of course services can be installed independent
of Ambari on the
> > > > >>>>> hosts
> > > > >>>>>>>> inside the cluster...but I don't think
that is what you are
> > > > >>> getting
> > > > >>>>> at.
> > > > >>>>>>> Are
> > > > >>>>>>>> you expecting the services to have a
presence on the UI though
> > > > >>>>> installed
> > > > >>>>>>>> outside of Ambari?
> > > > >>>>>>>>
> > > > >>>>>>>> Erin
> > > > >>>>>>>>
> > > > >>>>>>>> ----- Original Message -----
> > > > >>>>>>>> From: "Konstantin Boudnik" <cos@apache.org>
> > > > >>>>>>>> To: dev@ambari.apache.org
> > > > >>>>>>>> Sent: Monday, December 1, 2014 6:42:30
PM
> > > > >>>>>>>> Subject: About extensibility of Ambari
stacks (not inheritance)
> > > > >>>>>>>>
> > > > >>>>>>>> Guys,
> > > > >>>>>>>>
> > > > >>>>>>>> I am looking into possible stack extensibility
properties of
> > > > >>> Ambari
> > > > >>>>> (not
> > > > >>>>>>> to be
> > > > >>>>>>>> confused with inheritance), but haven't
been able to derive any
> > > > >>> final
> > > > >>>>>>>> conclusions just yet. Hence, I'd appreciate
the input from the
> > > > >>> people
> > > > >>>>>>> behind
> > > > >>>>>>>> the system.
> > > > >>>>>>>>
> > > > >>>>>>>> I have a few questions about current
state of the Ambari
> > > (version
> > > > >>>>> 1.6.1
> > > > >>>>>>> and
> > > > >>>>>>>> coming 1.7, and possibly later?) with
regards to ability to
> > > > >>> expand an
> > > > >>>>>>> existing
> > > > >>>>>>>> stack definitions with a 3rd party services
and do it in the
> > > > >>> runtime,
> > > > >>>>>>> rather
> > > > >>>>>>>> than only during the installation. We
need to be able to run a
> > > > >>>>> multiple
> > > > >>>>>>>> instances of our master service in the
cluster, which isn't
> > > > >>> typical
> > > > >>>>> for
> > > > >>>>>>>> "normal" Hadoop concept where only one
master can exist for any
> > > > >>>>> giving
> > > > >>>>>>>> service.
> > > > >>>>>>>>
> > > > >>>>>>>> Our use case is to be able to amend an
exiting cluster setup
> > > > >>> (HDP,
> > > > >>>>>>> Bigtop,
> > > > >>>>>>>> etc.) with a new service running on top
of HDFS; but not to
> > > > >>>>> reinstall the
> > > > >>>>>>>> whole stack. The reason we have the use
case is that oftentimes
> > > > >>> our
> > > > >>>>>>> software
> > > > >>>>>>>> is being added to an existing 3rd party
environment, as an added
> > > > >>>>> bonus,
> > > > >>>>>>> not
> > > > >>>>>>>> available during the initial planning
and setup of the cluster.
> > > > >>>>>>>>
> > > > >>>>>>>> The way I understand Ambari's stack inheritance
is that a scion
> > > > >>> stack
> > > > >>>>>>> will
> > > > >>>>>>>> be a brand new entity, e.g. I won't be
able to cherry-pick
> > > > >>> services
> > > > >>>>> from
> > > > >>>>>>> it
> > > > >>>>>>>> and add them to the already installed
parent stack's cluster,
> > > > >>> right?
> > > > >>>>>>>>
> > > > >>>>>>>> So far from I see it doesn't seem possible
without introducing a
> > > > >>>>>>> brand-new
> > > > >>>>>>>> version of a stack e.g. 'stack inheritance'.
Which,
> > > > >>> unfortunately,
> > > > >>>>> won't
> > > > >>>>>>> work
> > > > >>>>>>>> as per my explanation of the use above
case. I guess another way
> > > > >>> to
> > > > >>>>> look
> > > > >>>>>>> at it
> > > > >>>>>>>> is this: would it be possible to add
a component (or a service)
> > > > >>> that
> > > > >>>>> will
> > > > >>>>>>>> override an existing component or a service,
but without a need
> > > > >>> to
> > > > >>>>>>> reinstall
> > > > >>>>>>>> the rest of the stack?
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks in advance for any info/ideas.
> > > > >>>>>>>>  Cos
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> CONFIDENTIALITY NOTICE
> > > > >>>>>> NOTICE: This message is intended for the use
of the individual or
> > > > >>> entity
> > > > >>>>> to
> > > > >>>>>> which it is addressed and may contain information
that is
> > > > >>> confidential,
> > > > >>>>>> privileged and exempt from disclosure under applicable
law. If the
> > > > >>> reader
> > > > >>>>>> of this message is not the intended recipient,
you are hereby
> > > > >>> notified
> > > > >>>>> that
> > > > >>>>>> any printing, copying, dissemination, distribution,
disclosure or
> > > > >>>>>> forwarding of this communication is strictly
prohibited. If you
> > > have
> > > > >>>>>> received this communication in error, please
contact the sender
> > > > >>>>> immediately
> > > > >>>>>> and delete it from your system. Thank You.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> CONFIDENTIALITY NOTICE
> > > > >>>> NOTICE: This message is intended for the use of the individual
or
> > > entity
> > > > >>> to
> > > > >>>> which it is addressed and may contain information that
is
> > > confidential,
> > > > >>>> privileged and exempt from disclosure under applicable
law. If the
> > > reader
> > > > >>>> of this message is not the intended recipient, you are
hereby
> > > notified
> > > > >>> that
> > > > >>>> any printing, copying, dissemination, distribution, disclosure
or
> > > > >>>> forwarding of this communication is strictly prohibited.
If you have
> > > > >>>> received this communication in error, please contact
the sender
> > > > >>> immediately
> > > > >>>> and delete it from your system. Thank You.
> > > > >>>
> > > > >>
> > > > >> --
> > > > >> CONFIDENTIALITY NOTICE
> > > > >> NOTICE: This message is intended for the use of the individual
or
> > > entity to
> > > > >> which it is addressed and may contain information that is
> > > confidential,
> > > > >> privileged and exempt from disclosure under applicable law. If
the
> > > reader
> > > > >> of this message is not the intended recipient, you are hereby
> > > notified that
> > > > >> any printing, copying, dissemination, distribution, disclosure
or
> > > > >> forwarding of this communication is strictly prohibited. If you
have
> > > > >> received this communication in error, please contact the sender
> > > immediately
> > > > >> and delete it from your system. Thank You.
> > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or entity
> > > to
> > > > which it is addressed and may contain information that is confidential,
> > > > privileged and exempt from disclosure under applicable law. If the reader
> > > > of this message is not the intended recipient, you are hereby notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > >
> > 
> > -- 
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity to 
> > which it is addressed and may contain information that is confidential, 
> > privileged and exempt from disclosure under applicable law. If the reader 
> > of this message is not the intended recipient, you are hereby notified that 
> > any printing, copying, dissemination, distribution, disclosure or 
> > forwarding of this communication is strictly prohibited. If you have 
> > received this communication in error, please contact the sender immediately 
> > and delete it from your system. Thank You.



Mime
View raw message