hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet Goel <vvin...@apache.org>
Subject Re: hawq Ambari integration
Date Fri, 03 Mar 2017 03:04:39 GMT
Ranger community advises against sync between Ranger policies and native
authorization metastores. It complicates design and users haven't shown a
real need to switch between auth models back and forth.

-Vineet


On Thu, Mar 2, 2017 at 6:33 PM Alexander Denissov <adenissov@pivotal.io>
wrote:

> Not quite.
>
> For the first phase, there will be no integration between GRANT/REVOKE and
> Ranger policies -- once Ranger is turned on, GRANT/REVOKE from psql will no
> longer work.
>
> In a later stage, we will likely provide integration HAWQ --> Ranger, such
> that when GRANT/REVOKE is issued from psql, an appropriate policy is
> created / deleted from Ranger.
>
> I don't think Ranger --> HAWQ integration is planned or is even possible,
> as admin action in Ranger UI would need to be somehow detected and
> translated to HAWQ grants which are not going to be used anyways as
> authorization will be performed in Ranger only.
>
> --
> Thanks,
> Alex
>
> On Thu, Mar 2, 2017 at 6:20 PM, Jon Roberts <jroberts@pivotal.io> wrote:
>
> > Have I misunderstood the Ranger integration work?  When a GRANT/REVOKE is
> > executed in HAWQ it will be replicated to Ranger and when a GRANT/REVOKE
> is
> > executed in Ranger, it will be replicated to HAWQ.  Right?  If yes, then
> > this is the model that I'm suggesting for Ambari.
> >
> > Jon Roberts
> > Principal Engineer | jroberts@pivotal.io | 615-426-8661
> <(615)%20426-8661>
> >
> > On Thu, Mar 2, 2017 at 7:27 PM, Vineet Goel <vvineet@apache.org> wrote:
> >
> > > Having worked with Ambari and knowing the complexities of this desired
> > > integration, I agree with Alex on his comments below. Users need some
> > time
> > > to get used to exclusive HAWQ CLI or Ambari usage model, and avoid mix
> > and
> > > match despite the fact that HAWQ CLI and Ambari have their
> > > advantages/disadvantages.
> > >
> > > -Vineet
> > >
> > >
> > > On Thu, Mar 2, 2017 at 5:17 PM Alexander Denissov <
> adenissov@pivotal.io>
> > > wrote:
> > >
> > > > Ambari is designed as a tool to sit on top of individual service CLI
> > > tools
> > > > and provide consistent user experience. Ambari can also have wizards
> > and
> > > > custom actions that go beyond service lifecycle management. It is
> > assumed
> > > > by Ambari design that once Ambari is used to manage configurations,
> CLI
> > > > tools should not be used, otherwise changes will be out-of-sync or
> will
> > > be
> > > > overwritten by Ambari upon service restart.
> > > >
> > > > Having HAWQ CLI tools communicate back to Ambari is not desirable
> > > because:
> > > > - knowledge of Ambari REST APIs needs to be built into the tools
> > > > - Ambari APIs change outside of service release lifecycle
> > > > - complexity in error handling (what do you do if Ambari call failed
> ?)
> > > > - introduction of feedback loop and potential infinite cycle (Ambari
> > > calls
> > > > the tool, the tool calls Ambari back, etc)
> > > >
> > > > If customers want to do some extra automation they can decide how to
> > tie
> > > > tools and Ambari, if they like, but I will say we should not have
> HAWQ
> > > CLI
> > > > tools call Ambari.
> > > >
> > > > --
> > > > Thanks,
> > > > Alex Denissov
> > > >
> > > > On Thu, Mar 2, 2017 at 4:44 PM, Alex (Oleksandr) Diachenko <
> > > > odiachenko@pivotal.io> wrote:
> > > >
> > > > > Sure,
> > > > >
> > > > > Users can automate configuration changes via Ambari API, which
> > provides
> > > > > unified access
> > > > > to all services in a cluster. It's very likely they need to
> configure
> > > > HAWQ,
> > > > > HDFS and YARN
> > > > > when working with configuration changes, so with Ambari API it
> would
> > be
> > > > > easier to call one API
> > > > > rather than using all different CLI's.
> > > > >
> > > > > Regards, Alex.
> > > > >
> > > > > On Thu, Mar 2, 2017 at 4:39 PM, Lei Chang <lei_chang@apache.org>
> > > wrote:
> > > > >
> > > > > > I think there are some use cases we need What Jon proposed.
> > > > > >
> > > > > > For example, users installed hawq via Ambari, but they want
to
> > > automate
> > > > > the
> > > > > > configuration changes, it would be convenient to have hawq CLI
> > change
> > > > the
> > > > > > configurations.
> > > > > >
> > > > > > Cheers
> > > > > > Lei
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 8:36 AM, Alex (Oleksandr) Diachenko <
> > > > > > odiachenko@pivotal.io> wrote:
> > > > > >
> > > > > > > I see, that makes sense.
> > > > > > > But is there any action users cannot do via Ambari?
> > > > > > >
> > > > > > > Ranger is also a good example, there we are making assumption,
> > > > > > > users either use Ranger or HAWQ's authorization engine.
> > > > > > >
> > > > > > > The same logic might be extrapolated to HAWQ/Ambari - users
> might
> > > use
> > > > > > > either Ambari or HAWQ CLI, but not both at the same time.
> > > > > > > In that way, we can keep things simple.
> > > > > > >
> > > > > > > Regards, Alex.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 2, 2017 at 4:28 PM, Jon Roberts <
> jroberts@pivotal.io
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Right.  Just like HAWQ will be operational without
Ranger.
> > > > > > > >
> > > > > > > > We have the hawq CLI and will obviously continue to
have it.
> > > Some
> > > > > > people
> > > > > > > > use Ambari while others don't.  So just like with
Ranger
> > support,
> > > > > > > integrate
> > > > > > > > when possible but don't require it.
> > > > > > > >
> > > > > > > >
> > > > > > > > Jon
> > > > > > > >
> > > > > > > > On Thu, Mar 2, 2017 at 6:26 PM, Alex (Oleksandr) Diachenko
<
> > > > > > > > odiachenko@pivotal.io> wrote:
> > > > > > > >
> > > > > > > > > Not really, because HAWQ should be operational
even without
> > > > > Ambari(if
> > > > > > > > > that's the case).
> > > > > > > > >
> > > > > > > > > On Thu, Mar 2, 2017 at 4:21 PM, Jon Roberts <
> > > jroberts@pivotal.io
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > If that is the case, should we remove the
"hawq" CLI?
> > > > > > > > > >
> > > > > > > > > > Jon
> > > > > > > > > >
> > > > > > > > > > On Thu, Mar 2, 2017 at 6:12 PM, Alex (Oleksandr)
> Diachenko
> > <
> > > > > > > > > > odiachenko@pivotal.io> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jon,
> > > > > > > > > > >
> > > > > > > > > > > I think it was designed that Ambari
is supposed to be
> > only
> > > > one
> > > > > > > source
> > > > > > > > > of
> > > > > > > > > > > true.
> > > > > > > > > > > The whole purpose of integration id
to provide a
> > > > user-friendly
> > > > > > > > > interface
> > > > > > > > > > > and avoid manually editing/distributing
config files
> > > > > > > > > > > or running CLI commands.
> > > > > > > > > > > The idea of coupling HAWQ master with
Ambari doesn't
> seem
> > > to
> > > > be
> > > > > > > > clean.
> > > > > > > > > > >
> > > > > > > > > > > Regards, Alex.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 2, 2017 at 4:05 PM, Jon
Roberts <
> > > > > jroberts@pivotal.io
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > It would be handy if the "hawq
config" also updated
> > > > Ambari's
> > > > > > > > database
> > > > > > > > > > so
> > > > > > > > > > > > that changes could be made in
either place are
> retained
> > > > when
> > > > > > > > changes
> > > > > > > > > > are
> > > > > > > > > > > > made in either place.
> > > > > > > > > > > >
> > > > > > > > > > > > Register Ambari:
> > > > > > > > > > > > hawq ambari -u admin -w admin
-h myhost -p 8080
> > > > > > > > > > > >
> > > > > > > > > > > > "hawq config" could then raise
INFO/WARN messages
> about
> > > > > > updating
> > > > > > > > > > Ambari.
> > > > > > > > > > > >
> > > > > > > > > > > > Example:
> > > > > > > > > > > > hawq config -c hawq_rm_stmt_vseg_memory
-v 16gb
> > > > > > > > > > > > INFO: Updated Ambari with
> hawq_rm_stmt_vseg_memory=16gb
> > > > > > > > > > > > or
> > > > > > > > > > > > hawq config -c hawq_rm_stmt_vseg_memory
-v 16gb
> > > > > > > > > > > > WARN: Failed to update Ambari
with
> > > > > > hawq_rm_stmt_vseg_memory=16gb.
> > > > > > > > > > Please
> > > > > > > > > > > > update Ambari credentials manually
to retain this
> > > > > configuration
> > > > > > > > > change
> > > > > > > > > > > > after a restart.
> > > > > > > > > > > >
> > > > > > > > > > > > The implementation would require
interacting with the
> > > > Ambari
> > > > > > APIs
> > > > > > > > and
> > > > > > > > > > > also
> > > > > > > > > > > > storing the credentials in an
encrypted file on the
> > HAWQ
> > > > > > Master.
> > > > > > > > > > > >
> > > > > > > > > > > > Thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Jon Roberts
> > > > > > > > > > > > Principal Engineer | jroberts@pivotal.io
|
> > 615-426-8661 <(615)%20426-8661>
> > > > <(615)%20426-8661>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message