Mailing-List: contact dev-help@hawq.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hawq.incubator.apache.org
MIME-Version: 1.0
References: <CAH_aXGNaqvgCgLV3C9AiCzTLe+35GP3f78Z9-LmU4nL9BO=n+w@mail.gmail.com>
 <CA+-yw=J8D2TRg7dThE=-QxmUbbzbVhTRWFZqBZdJzmj34F4ECQ@mail.gmail.com>
 <CAH_aXGMjv87xGtzcaP6RM4nbAC3WSa3adEcB0NYfb7HPf5-XOQ@mail.gmail.com>
 <CA+-yw=+i-SOXwNNuKkk03fOgGQ+SBvTh_9pMxi3_1C7DSeH9Jw@mail.gmail.com>
 <CAH_aXGMuK=LyKMPdO-xtPn7Fx55CAfSp8PVULhT1XngUx-1H1A@mail.gmail.com>
 <CA+-yw=+DV=57BweLrwSNJ9THeBJGY6Op=PH-Rpy-xOCYXA4X6A@mail.gmail.com>
 <CAE44UQdejn4y7fQ0w+nsvOpYvPtCmR5KH2eSp1HFL4Vj-+1iAw@mail.gmail.com>
 <CA+-yw=JSCi0h8=uTDe-UoT_t1de4VJ6wyZZtCEXgyys8u=8usA@mail.gmail.com> <CAKomjQaW+3opH5EEV6x1qpn8gjpP=o9AdeG6S1fRWEW_ztA9Tg@mail.gmail.com>
In-Reply-To: <CAKomjQaW+3opH5EEV6x1qpn8gjpP=o9AdeG6S1fRWEW_ztA9Tg@mail.gmail.com>
From: Vineet Goel <vvineet@apache.org>
Date: Fri, 03 Mar 2017 01:27:22 +0000
Message-ID: <CAMLEP5eQMAMqGs05RXC-AU2qsxucx+KkLef7DN7vVsOhv8-XiQ@mail.gmail.com>
Subject: Re: hawq Ambari integration
To: dev@hawq.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1146f47e2ed4680549c97331
archived-at: Fri, 03 Mar 2017 01:27:41 -0000

--001a1146f47e2ed4680549c97331
Content-Type: text/plain; charset=UTF-8

Having worked with Ambari and knowing the complexities of this desired
integration, I agree with Alex on his comments below. Users need some time
to get used to exclusive HAWQ CLI or Ambari usage model, and avoid mix and
match despite the fact that HAWQ CLI and Ambari have their
advantages/disadvantages.

-Vineet


On Thu, Mar 2, 2017 at 5:17 PM Alexander Denissov <adenissov@pivotal.io>
wrote:

> Ambari is designed as a tool to sit on top of individual service CLI tools
> and provide consistent user experience. Ambari can also have wizards and
> custom actions that go beyond service lifecycle management. It is assumed
> by Ambari design that once Ambari is used to manage configurations, CLI
> tools should not be used, otherwise changes will be out-of-sync or will be
> overwritten by Ambari upon service restart.
>
> Having HAWQ CLI tools communicate back to Ambari is not desirable because:
> - knowledge of Ambari REST APIs needs to be built into the tools
> - Ambari APIs change outside of service release lifecycle
> - complexity in error handling (what do you do if Ambari call failed ?)
> - introduction of feedback loop and potential infinite cycle (Ambari calls
> the tool, the tool calls Ambari back, etc)
>
> If customers want to do some extra automation they can decide how to tie
> tools and Ambari, if they like, but I will say we should not have HAWQ CLI
> tools call Ambari.
>
> --
> Thanks,
> Alex Denissov
>
> On Thu, Mar 2, 2017 at 4:44 PM, Alex (Oleksandr) Diachenko <
> odiachenko@pivotal.io> wrote:
>
> > Sure,
> >
> > Users can automate configuration changes via Ambari API, which provides
> > unified access
> > to all services in a cluster. It's very likely they need to configure
> HAWQ,
> > HDFS and YARN
> > when working with configuration changes, so with Ambari API it would be
> > easier to call one API
> > rather than using all different CLI's.
> >
> > Regards, Alex.
> >
> > On Thu, Mar 2, 2017 at 4:39 PM, Lei Chang <lei_chang@apache.org> wrote:
> >
> > > I think there are some use cases we need What Jon proposed.
> > >
> > > For example, users installed hawq via Ambari, but they want to automate
> > the
> > > configuration changes, it would be convenient to have hawq CLI change
> the
> > > configurations.
> > >
> > > Cheers
> > > Lei
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Mar 3, 2017 at 8:36 AM, Alex (Oleksandr) Diachenko <
> > > odiachenko@pivotal.io> wrote:
> > >
> > > > I see, that makes sense.
> > > > But is there any action users cannot do via Ambari?
> > > >
> > > > Ranger is also a good example, there we are making assumption,
> > > > users either use Ranger or HAWQ's authorization engine.
> > > >
> > > > The same logic might be extrapolated to HAWQ/Ambari - users might use
> > > > either Ambari or HAWQ CLI, but not both at the same time.
> > > > In that way, we can keep things simple.
> > > >
> > > > Regards, Alex.
> > > >
> > > >
> > > >
> > > > On Thu, Mar 2, 2017 at 4:28 PM, Jon Roberts <jroberts@pivotal.io>
> > wrote:
> > > >
> > > > > Right.  Just like HAWQ will be operational without Ranger.
> > > > >
> > > > > We have the hawq CLI and will obviously continue to have it.  Some
> > > people
> > > > > use Ambari while others don't.  So just like with Ranger support,
> > > > integrate
> > > > > when possible but don't require it.
> > > > >
> > > > >
> > > > > Jon
> > > > >
> > > > > On Thu, Mar 2, 2017 at 6:26 PM, Alex (Oleksandr) Diachenko <
> > > > > odiachenko@pivotal.io> wrote:
> > > > >
> > > > > > Not really, because HAWQ should be operational even without
> > Ambari(if
> > > > > > that's the case).
> > > > > >
> > > > > > On Thu, Mar 2, 2017 at 4:21 PM, Jon Roberts <jroberts@pivotal.io
> >
> > > > wrote:
> > > > > >
> > > > > > > If that is the case, should we remove the "hawq" CLI?
> > > > > > >
> > > > > > > Jon
> > > > > > >
> > > > > > > On Thu, Mar 2, 2017 at 6:12 PM, Alex (Oleksandr) Diachenko <
> > > > > > > odiachenko@pivotal.io> wrote:
> > > > > > >
> > > > > > > > Hi Jon,
> > > > > > > >
> > > > > > > > I think it was designed that Ambari is supposed to be only
> one
> > > > source
> > > > > > of
> > > > > > > > true.
> > > > > > > > The whole purpose of integration id to provide a
> user-friendly
> > > > > > interface
> > > > > > > > and avoid manually editing/distributing config files
> > > > > > > > or running CLI commands.
> > > > > > > > The idea of coupling HAWQ master with Ambari doesn't seem to
> be
> > > > > clean.
> > > > > > > >
> > > > > > > > Regards, Alex.
> > > > > > > >
> > > > > > > > On Thu, Mar 2, 2017 at 4:05 PM, Jon Roberts <
> > jroberts@pivotal.io
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > It would be handy if the "hawq config" also updated
> Ambari's
> > > > > database
> > > > > > > so
> > > > > > > > > that changes could be made in either place are retained
> when
> > > > > changes
> > > > > > > are
> > > > > > > > > made in either place.
> > > > > > > > >
> > > > > > > > > Register Ambari:
> > > > > > > > > hawq ambari -u admin -w admin -h myhost -p 8080
> > > > > > > > >
> > > > > > > > > "hawq config" could then raise INFO/WARN messages about
> > > updating
> > > > > > > Ambari.
> > > > > > > > >
> > > > > > > > > Example:
> > > > > > > > > hawq config -c hawq_rm_stmt_vseg_memory -v 16gb
> > > > > > > > > INFO: Updated Ambari with hawq_rm_stmt_vseg_memory=16gb
> > > > > > > > > or
> > > > > > > > > hawq config -c hawq_rm_stmt_vseg_memory -v 16gb
> > > > > > > > > WARN: Failed to update Ambari with
> > > hawq_rm_stmt_vseg_memory=16gb.
> > > > > > > Please
> > > > > > > > > update Ambari credentials manually to retain this
> > configuration
> > > > > > change
> > > > > > > > > after a restart.
> > > > > > > > >
> > > > > > > > > The implementation would require interacting with the
> Ambari
> > > APIs
> > > > > and
> > > > > > > > also
> > > > > > > > > storing the credentials in an encrypted file on the HAWQ
> > > Master.
> > > > > > > > >
> > > > > > > > > Thoughts?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Jon Roberts
> > > > > > > > > Principal Engineer | jroberts@pivotal.io | 615-426-8661
> <(615)%20426-8661>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001a1146f47e2ed4680549c97331--