hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Roberts <jrobe...@pivotal.io>
Subject Re: hawq Ambari integration
Date Fri, 03 Mar 2017 02:20:30 GMT
Have I misunderstood the Ranger integration work?  When a GRANT/REVOKE is
executed in HAWQ it will be replicated to Ranger and when a GRANT/REVOKE is
executed in Ranger, it will be replicated to HAWQ.  Right?  If yes, then
this is the model that I'm suggesting for Ambari.

Jon Roberts
Principal Engineer | jroberts@pivotal.io | 615-426-8661

On Thu, Mar 2, 2017 at 7:27 PM, Vineet Goel <vvineet@apache.org> wrote:

> Having worked with Ambari and knowing the complexities of this desired
> integration, I agree with Alex on his comments below. Users need some time
> to get used to exclusive HAWQ CLI or Ambari usage model, and avoid mix and
> match despite the fact that HAWQ CLI and Ambari have their
> advantages/disadvantages.
>
> -Vineet
>
>
> On Thu, Mar 2, 2017 at 5:17 PM Alexander Denissov <adenissov@pivotal.io>
> wrote:
>
> > Ambari is designed as a tool to sit on top of individual service CLI
> tools
> > and provide consistent user experience. Ambari can also have wizards and
> > custom actions that go beyond service lifecycle management. It is assumed
> > by Ambari design that once Ambari is used to manage configurations, CLI
> > tools should not be used, otherwise changes will be out-of-sync or will
> be
> > overwritten by Ambari upon service restart.
> >
> > Having HAWQ CLI tools communicate back to Ambari is not desirable
> because:
> > - knowledge of Ambari REST APIs needs to be built into the tools
> > - Ambari APIs change outside of service release lifecycle
> > - complexity in error handling (what do you do if Ambari call failed ?)
> > - introduction of feedback loop and potential infinite cycle (Ambari
> calls
> > the tool, the tool calls Ambari back, etc)
> >
> > If customers want to do some extra automation they can decide how to tie
> > tools and Ambari, if they like, but I will say we should not have HAWQ
> CLI
> > tools call Ambari.
> >
> > --
> > Thanks,
> > Alex Denissov
> >
> > On Thu, Mar 2, 2017 at 4:44 PM, Alex (Oleksandr) Diachenko <
> > odiachenko@pivotal.io> wrote:
> >
> > > Sure,
> > >
> > > Users can automate configuration changes via Ambari API, which provides
> > > unified access
> > > to all services in a cluster. It's very likely they need to configure
> > HAWQ,
> > > HDFS and YARN
> > > when working with configuration changes, so with Ambari API it would be
> > > easier to call one API
> > > rather than using all different CLI's.
> > >
> > > Regards, Alex.
> > >
> > > On Thu, Mar 2, 2017 at 4:39 PM, Lei Chang <lei_chang@apache.org>
> wrote:
> > >
> > > > I think there are some use cases we need What Jon proposed.
> > > >
> > > > For example, users installed hawq via Ambari, but they want to
> automate
> > > the
> > > > configuration changes, it would be convenient to have hawq CLI change
> > the
> > > > configurations.
> > > >
> > > > Cheers
> > > > Lei
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Mar 3, 2017 at 8:36 AM, Alex (Oleksandr) Diachenko <
> > > > odiachenko@pivotal.io> wrote:
> > > >
> > > > > I see, that makes sense.
> > > > > But is there any action users cannot do via Ambari?
> > > > >
> > > > > Ranger is also a good example, there we are making assumption,
> > > > > users either use Ranger or HAWQ's authorization engine.
> > > > >
> > > > > The same logic might be extrapolated to HAWQ/Ambari - users might
> use
> > > > > either Ambari or HAWQ CLI, but not both at the same time.
> > > > > In that way, we can keep things simple.
> > > > >
> > > > > Regards, Alex.
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Mar 2, 2017 at 4:28 PM, Jon Roberts <jroberts@pivotal.io>
> > > wrote:
> > > > >
> > > > > > Right.  Just like HAWQ will be operational without Ranger.
> > > > > >
> > > > > > We have the hawq CLI and will obviously continue to have it.
> Some
> > > > people
> > > > > > use Ambari while others don't.  So just like with Ranger support,
> > > > > integrate
> > > > > > when possible but don't require it.
> > > > > >
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Thu, Mar 2, 2017 at 6:26 PM, Alex (Oleksandr) Diachenko <
> > > > > > odiachenko@pivotal.io> wrote:
> > > > > >
> > > > > > > Not really, because HAWQ should be operational even without
> > > Ambari(if
> > > > > > > that's the case).
> > > > > > >
> > > > > > > On Thu, Mar 2, 2017 at 4:21 PM, Jon Roberts <
> jroberts@pivotal.io
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > If that is the case, should we remove the "hawq" CLI?
> > > > > > > >
> > > > > > > > Jon
> > > > > > > >
> > > > > > > > On Thu, Mar 2, 2017 at 6:12 PM, Alex (Oleksandr) Diachenko
<
> > > > > > > > odiachenko@pivotal.io> wrote:
> > > > > > > >
> > > > > > > > > Hi Jon,
> > > > > > > > >
> > > > > > > > > I think it was designed that Ambari is supposed
to be only
> > one
> > > > > source
> > > > > > > of
> > > > > > > > > true.
> > > > > > > > > The whole purpose of integration id to provide
a
> > user-friendly
> > > > > > > interface
> > > > > > > > > and avoid manually editing/distributing config
files
> > > > > > > > > or running CLI commands.
> > > > > > > > > The idea of coupling HAWQ master with Ambari
doesn't seem
> to
> > be
> > > > > > clean.
> > > > > > > > >
> > > > > > > > > Regards, Alex.
> > > > > > > > >
> > > > > > > > > On Thu, Mar 2, 2017 at 4:05 PM, Jon Roberts <
> > > jroberts@pivotal.io
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > It would be handy if the "hawq config" also
updated
> > Ambari's
> > > > > > database
> > > > > > > > so
> > > > > > > > > > that changes could be made in either place
are retained
> > when
> > > > > > changes
> > > > > > > > are
> > > > > > > > > > made in either place.
> > > > > > > > > >
> > > > > > > > > > Register Ambari:
> > > > > > > > > > hawq ambari -u admin -w admin -h myhost
-p 8080
> > > > > > > > > >
> > > > > > > > > > "hawq config" could then raise INFO/WARN
messages about
> > > > updating
> > > > > > > > Ambari.
> > > > > > > > > >
> > > > > > > > > > Example:
> > > > > > > > > > hawq config -c hawq_rm_stmt_vseg_memory
-v 16gb
> > > > > > > > > > INFO: Updated Ambari with hawq_rm_stmt_vseg_memory=16gb
> > > > > > > > > > or
> > > > > > > > > > hawq config -c hawq_rm_stmt_vseg_memory
-v 16gb
> > > > > > > > > > WARN: Failed to update Ambari with
> > > > hawq_rm_stmt_vseg_memory=16gb.
> > > > > > > > Please
> > > > > > > > > > update Ambari credentials manually to retain
this
> > > configuration
> > > > > > > change
> > > > > > > > > > after a restart.
> > > > > > > > > >
> > > > > > > > > > The implementation would require interacting
with the
> > Ambari
> > > > APIs
> > > > > > and
> > > > > > > > > also
> > > > > > > > > > storing the credentials in an encrypted
file on the HAWQ
> > > > Master.
> > > > > > > > > >
> > > > > > > > > > Thoughts?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jon Roberts
> > > > > > > > > > Principal Engineer | jroberts@pivotal.io
| 615-426-8661
> > <(615)%20426-8661>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message