airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Meickle <jmeic...@quantopian.com>
Subject Re: RBAC Update
Date Mon, 02 Apr 2018 16:14:27 GMT
To my mind, I would expect the MVP of per-DAG RBAC to include three
settings: viewing DAG state, executing or modifying DAGs, and viewing tasks
within the DAG (logs/code/details). For instance we would love to expose a
view of the production dataload state to our engineers, without exposing
production logs or allowing them to run/modify anything.

Creating these mappings would be somewhat painful to manage, but that seems
fine for an MVP. I would expect some future version to add grouping/tagging
to DAGs and to users, to say e.g. "users in group X can see DAGs in group Y
and modify DAGs in group Z."

On Fri, Mar 30, 2018 at 1:26 PM, Brian Greene <
brian@heisenbergwoodworking.com> wrote:

> I’d think we’d have privilege ‘can_view’ etc, and then a join table (priv)
> <-> (dagid) <-> (user/group).  Then it’s a simple query to get the perms
> for a given dag (as you list In option 2 below).
>
> It also makes a “secure by default” easy - a lack of entries in that table
> for a dag can mean only “admin” access or some such.
>
> Then any dag can have any combo of permissions for any combo of users.
> Adding the groups option raises complexity around nesting, so maybe skip it
> for r1?
>
> My $.02
>
> Brian
>
> Sent from a device with less than stellar autocorrect
>
> > On Mar 29, 2018, at 10:27 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
> >
> > Hijacking the thread further here, any thoughts on how to breakdown per
> DAG
> > access?
> >
> > Tao & I are talking about introducing per-DAG permissions and one big
> > question is whether we'll need to support different operation-types at a
> > per-DAG level, which changes the way we need to model the perms.
> >
> > First [simpler] option is to introduce one perm per DAG. If you have
> access
> > to 5 DAGs, and you have `can_clear` and `can_run`, you'll have homogenous
> > rights on the DAGs you have access to.
> >
> > Second option is to have a breakdown per DAG. Meaning for each DAG we
> > create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...). So
> one
> > user could have modify on some DAGs, view on others, and other DAGs would
> > be invisible. This could be broken down further ({dag_id}_can_clear, ...)
> > but it gets hard to manage.
> >
> > Thoughts?
> >
> > Max
> >
> >> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fengtao04@gmail.com> wrote:
> >>
> >> Great work Joy. This is awesome! I am interested in helping out the per
> dag
> >> level access.  Just created a ticket to check(AIRFLOW-2267). Let me
> know if
> >> you have any suggestions. I will share my proposal once I am ready.
> >>
> >>> On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <joyg@wepay.com> wrote:
> >>>
> >>> Hey guys!
> >>>
> >>> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015>
> has
> >>> been merged to master. I'm looking forward to early adopters' feedback
> >> and
> >>> bug reports. I also hope to have more folks helping out with the RBAC
> UI,
> >>> especially with introducing DAG-Level access control, which is a
> feature
> >>> that a lot of people have been asking. If you are interested in helping
> >> out
> >>> with this effort, let's talk more!
> >>>
> >>> This commit will be in the 1.10.0 release, and we are going to maintain
> >>> both UIs simultaneously for a short period of time. Once RBAC UI is
> >> stable
> >>> and battle-tested, we will deprecate the old UI and eventually remove
> it
> >>> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to
> prevent
> >>> two UIs from forking into separate paths, as that would become very
> >>> difficult to maintain.
> >>>
> >>> Going forward while both UIs are up, if you are making a change to any
> >>> files in airflow/www/ (old UI), where applicable, please also make the
> >>> change to the airflow/www_rbac/ (new UI). If you rather not make
> changes
> >> in
> >>> both UIs, it is recommended that you only make the changes to the RBAC
> >> UI,
> >>> since that is the one we are maintaining in the long term.
> >>>
> >>> I'm excited that the RBAC UI will be able to bring additional security
> to
> >>> Airflow, and with FAB framework in place we can look into leveraging it
> >> for
> >>> a unified set of APIs used by both UI and CLI.
> >>>
> >>> Joy
> >>>
> >>>
> >>>
> >>>> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <joyg@wepay.com> wrote:
> >>>>
> >>>> Hi folks,
> >>>>
> >>>> I have a PR <https://github.com/apache/incubator-airflow/pull/3015>
> >> out
> >>>> for the new UI. I've included instructions on how to test it out in
> the
> >>> PR
> >>>> description. Looking forward to your feedbacks.
> >>>>
> >>>> Cheers,
> >>>> Joy
> >>>>
> >>>>> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <joyg@wepay.com> wrote:
> >>>>>
> >>>>> Thanks for the background info. Would be really awesome for you
to
> >> have
> >>>>> PyPi access :D I'll make the change to have Airflow Webserver's
FAB
> >>>>> dependency pointing to my fork for the mean time.
> >>>>>
> >>>>> For folks who are interested in RBAC, I will be giving a talk/demo
at
> >>> the Airflow
> >>>>> Meet-Up
> >>>>> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> >>> Meetup/events/244525050/>
> >>>>> next Monday. Happy to chat afterwards about it as well :)
> >>>>>
> >>>>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>
> >>>>>> A bit of related history here:
> >>>>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>>>>>
> >>>>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>>
> >>>>>>> Given I have merge rights on FAB I could probably do another
round
> >> of
> >>>>>>> review and get your PRs through. I would really like to
get the
> >> main
> >>>>>>> maintainer's input on things that touch the core (composite-key
> >>>>>> support) as
> >>>>>>> he might have concerns/intuitions that we can't know about.
> >>>>>>>
> >>>>>>> I do not have Pypi access though so I cannot push new releases
> >> out. I
> >>>>>>> could ask for that.
> >>>>>>>
> >>>>>>> I've threatened to fork the project before, that's always
an
> >> option.
> >>>>>> I've
> >>>>>>> noticed his involvement is sporadic and comes in bursts.
> >>>>>>>
> >>>>>>> In the meantime, you can have the dependency in Airflow
Webserver
> >>>>>> pointing
> >>>>>>> straight to your fork.
> >>>>>>>
> >>>>>>> Max
> >>>>>>>
> >>>>>>>> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <joyg@wepay.com>
wrote:
> >>>>>>>>
> >>>>>>>> I just created a new webserver instance if you haven't
gotten a
> >>>>>> chance to
> >>>>>>>> fiddle around with the new web UI and the RBAC configurations
> >>> (thanks
> >>>>>>>> Maxime for getting started with this earlier!):
> >>>>>>>>
> >>>>>>>> http://104.209.38.171:8080/
> >>>>>>>>
> >>>>>>>> Admin Account
> >>>>>>>> username: admin
> >>>>>>>> password: admin
> >>>>>>>>
> >>>>>>>> Read-Only Account
> >>>>>>>> username: viewer
> >>>>>>>> password: password
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <joyg@wepay.com>
wrote:
> >>>>>>>>>
> >>>>>>>>> Hi folks,
> >>>>>>>>>
> >>>>>>>>> Thanks for all the feedback regarding to the new
Airflow
> >> Webserver
> >>>>>> UI
> >>>>>>>>> <https://github.com/wepay/airflow-webserver/>!
I've been
> >> actively
> >>>>>>>>> addressing all the bugs that were raised on Github.
So I want to
> >>>>>> take
> >>>>>>>> this
> >>>>>>>>> opportunity to discuss two issues coming up:
> >>>>>>>>>
> >>>>>>>>> The first issue is unaddressed PRs in FAB. If these
PRs continue
> >>> to
> >>>>>> stay
> >>>>>>>>> unaddressed, RBAC is blocked from making further
progress. If
> >> this
> >>>>>>>> continue
> >>>>>>>>> to be an issue, I'm inclined to fork FAB, even though
it's not
> >>>>>>>> idealistic.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>   - PR/631 <https://github.com/dpgaspar/F
> >>> lask-AppBuilder/pull/631>
> >>>>>>>> Binary
> >>>>>>>>>   column support (merged, unreleased)
> >>>>>>>>>   <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >>>>>>>>>   - PR/639 <https://github.com/dpgaspar/F
> >>> lask-AppBuilder/pull/639>
> >>>>>>>> Composite
> >>>>>>>>>   primary key support (unmerged)
> >>>>>>>>>   - PR/655 <https://github.com/dpgaspar/F
> >>> lask-AppBuilder/pull/655>
> >>>>>>>> Form
> >>>>>>>>>   prefill support (unmerged)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The second issue is an open question about the next
step of
> >>> Airflow
> >>>>>>>>> Webserver itself. Here are the 3 potential directions
we could
> >>>>>> take, and
> >>>>>>>>> I've added my thought on each.
> >>>>>>>>>
> >>>>>>>>> 1. Permanently keep Airflow Webserver as a separated
package
> >> from
> >>>>>>>> Airflow,
> >>>>>>>>> and treat it as another UI option. Keep `www` in
Airflow. Allow
> >>>>>>>> development
> >>>>>>>>> on both UIs.
> >>>>>>>>> *I'm not a fan of this. When there is an existing
UI in Airflow,
> >>>>>> most
> >>>>>>>>> contributors would prefer to maintain the official
version that
> >> is
> >>>>>>>>> installed out-of-the-box. **Having a second UI outside
of
> >> Airflow
> >>>>>> will
> >>>>>>>>> make maintaining it very difficult, leading to an
eventual death
> >>> of
> >>>>>> the
> >>>>>>>> new
> >>>>>>>>> UI :(*
> >>>>>>>>>
> >>>>>>>>> 2. Permanently keep Airflow Webserver as a separated
package
> >> from
> >>>>>>>> Airflow,
> >>>>>>>>> but freeze all development on `www`  and direct
all future UI
> >>>>>>>> development
> >>>>>>>>> to Airflow Webserver, eventually removing `www`
completely when
> >>>>>> Airflow
> >>>>>>>>> Webserver is stable.
> >>>>>>>>> *I'm not a fan of this either. First of all, the
views and
> >> models
> >>>>>> are
> >>>>>>>>> tightly coupled in both old and new UI; until we
have a
> >>> full-fledged
> >>>>>>>> REST
> >>>>>>>>> API to build the UI (and cli) on top of it, separating
them to a
> >>>>>>>> separate
> >>>>>>>>> package now will potentially cause dependency issues
and add
> >>>>>>>> complication
> >>>>>>>>> to our release cycle. **Secondly, **majority of
Airflow users
> >> run
> >>>>>>>> Airflow
> >>>>>>>>> with the UI; it's one of Airflow's best features.
Separating UI
> >>> out
> >>>>>> of
> >>>>>>>>> Airflow core will complicate setup and configuration,
while
> >> making
> >>>>>>>> Airflow
> >>>>>>>>> core less complete.*
> >>>>>>>>>
> >>>>>>>>> 3. Merge Airflow Webserver back into Airflow as
`www2`, freeze
> >> all
> >>>>>>>>> development on `www`, eventually removing `www`
completely when
> >>>>>> `www2`
> >>>>>>>> is
> >>>>>>>>> stable.
> >>>>>>>>> *This makes the most sense to me. Airflow Webserver
is developed
> >>>>>> with
> >>>>>>>> the
> >>>>>>>>> goal of feature parity to the current UI, plus additional
RBAC
> >>>>>>>> capability,
> >>>>>>>>> in hope to replace the old UI completely. Yes, this
means there
> >>>>>> will be
> >>>>>>>> a
> >>>>>>>>> short period of having to maintain two UIs, but
once we freeze
> >>>>>>>> development
> >>>>>>>>> on www, it shouldn't be a concern for long.*
> >>>>>>>>>
> >>>>>>>>> I'd love to hear everyone's thoughts on this! I'm
excited about
> >>>>>> bringing
> >>>>>>>>> RBAC to airflow and I hope it's something others
will find
> >> useful
> >>> as
> >>>>>>>> well!
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Joy
> >>>>>>>>>
> >>>>>>>>> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <joyg@wepay.com>
> >> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thank you everyone for the active feedback so
far, and thanks
> >> for
> >>>>>>>> setting
> >>>>>>>>>> up the demo Maxime!
> >>>>>>>>>>
> >>>>>>>>>> Going to work on pruning through the issues
in the upcoming
> >> days.
> >>>>>>>>>>
> >>>>>>>>>> Fokko/Maxime, do you recall the SQLAlchemy Exception
message
> >> so I
> >>>>>> can
> >>>>>>>>>> look into it? Otherwise I'll wait until it's
down again =P
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>>
> >>>>>>>>>> Joy
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin
<
> >>>>>>>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I just restarted it, not sure how long it
will take to get in
> >> a
> >>>>>> bad
> >>>>>>>> state
> >>>>>>>>>>> again...
> >>>>>>>>>>>
> >>>>>>>>>>> Max
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong,
Fokko
> >>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Good morning,
> >>>>>>>>>>>>
> >>>>>>>>>>>> The demo provided by Max is down, it
throws a
> >>>>>> SQLAlchemyexception
> >>>>>>>> :'(
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2017-11-18 19:14 GMT+01:00 Chris Riccomini
<
> >>>>>> criccomini@apache.org>:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> @bolke, open issues on the Github
repo, please.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Nov 18, 2017 at 10:13 AM,
Bolke de Bruin <
> >>>>>>>> bdbruin@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Chris,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Do you want us to report bugs
somewhere (I have
> >>> encountered
> >>>>>> a
> >>>>>>>>>>> few)? Or
> >>>>>>>>>>>>>> just generic user experiences
posted here?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 18 Nov 2017, at 00:47,
Chris Riccomini <
> >>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I know the weekend is coming
up, and for those of us
> >> in
> >>>>>> the
> >>>>>>>> US,
> >>>>>>>>>>> next
> >>>>>>>>>>>>> week
> >>>>>>>>>>>>>>> is a bit of a slow holiday
week. Would love to get
> >> some
> >>>>>>>> feedback
> >>>>>>>>>>> from
> >>>>>>>>>>>>>>> everyone on this. The goal
would ideally to be to
> >>>>>> converge on
> >>>>>>>>>>> this
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> eventually replace the existing
Airflow UI with this
> >>> one.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Nov 17, 2017 at
1:44 PM, Joy Gao <
> >>> joyg@wepay.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi guys.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I've been working on
moving airflow from Flask-Admin
> >> to
> >>>>>>>>>>>>> Flask-AppBuilder
> >>>>>>>>>>>>>>>> for RBAC
> >>>>>>>>>>>>>>>> <https://cwiki.apache.org/
> >> confluence/display/AIRFLOW/
> >>>>>>>>>>>>>> Airflow+RBAC+proposal
> >>>>>>>>>>>>>>>>> ,
> >>>>>>>>>>>>>>>> check it out at https://github.com/wepay/airfl
> >>>>>> ow-webserver.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> It's still a work-in-progress,
but most features you
> >>> see
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>>>> webserver
> >>>>>>>>>>>>>>>> UI today is available
there. For those who are
> >>>>>> interested in
> >>>>>>>>>>> RBAC,
> >>>>>>>>>>>> I'd
> >>>>>>>>>>>>>> love
> >>>>>>>>>>>>>>>> to get some early feedback
in terms of the following:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - New Flask-AppBuilder
UI (any bugs/regressions)
> >>>>>>>>>>>>>>>> - Setup issues
> >>>>>>>>>>>>>>>> - Ease of integration
with third party auth (i.e.
> >> LDAP,
> >>>>>> AD,
> >>>>>>>>>>> OAuth,
> >>>>>>>>>>>>> etc.)
> >>>>>>>>>>>>>>>> - Any other thoughts/concerns
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks a lot!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>> Joy
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message