airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Greene <br...@heisenbergwoodworking.com>
Subject Re: RBAC Update
Date Fri, 30 Mar 2018 17:26:28 GMT
I’d think we’d have privilege ‘can_view’ etc, and then a join table (priv) <->
(dagid) <-> (user/group).  Then it’s a simple query to get the perms for a given dag
(as you list In option 2 below).

It also makes a “secure by default” easy - a lack of entries in that table for a dag can
mean only “admin” access or some such.

Then any dag can have any combo of permissions for any combo of users.  Adding the groups
option raises complexity around nesting, so maybe skip it for r1?

My $.02 

Brian

Sent from a device with less than stellar autocorrect

> On Mar 29, 2018, at 10:27 AM, Maxime Beauchemin <maximebeauchemin@gmail.com> wrote:
> 
> Hijacking the thread further here, any thoughts on how to breakdown per DAG
> access?
> 
> Tao & I are talking about introducing per-DAG permissions and one big
> question is whether we'll need to support different operation-types at a
> per-DAG level, which changes the way we need to model the perms.
> 
> First [simpler] option is to introduce one perm per DAG. If you have access
> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have homogenous
> rights on the DAGs you have access to.
> 
> Second option is to have a breakdown per DAG. Meaning for each DAG we
> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...). So one
> user could have modify on some DAGs, view on others, and other DAGs would
> be invisible. This could be broken down further ({dag_id}_can_clear, ...)
> but it gets hard to manage.
> 
> Thoughts?
> 
> Max
> 
>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fengtao04@gmail.com> wrote:
>> 
>> Great work Joy. This is awesome! I am interested in helping out the per dag
>> level access.  Just created a ticket to check(AIRFLOW-2267). Let me know if
>> you have any suggestions. I will share my proposal once I am ready.
>> 
>>> On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <joyg@wepay.com> wrote:
>>> 
>>> Hey guys!
>>> 
>>> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
>>> been merged to master. I'm looking forward to early adopters' feedback
>> and
>>> bug reports. I also hope to have more folks helping out with the RBAC UI,
>>> especially with introducing DAG-Level access control, which is a feature
>>> that a lot of people have been asking. If you are interested in helping
>> out
>>> with this effort, let's talk more!
>>> 
>>> This commit will be in the 1.10.0 release, and we are going to maintain
>>> both UIs simultaneously for a short period of time. Once RBAC UI is
>> stable
>>> and battle-tested, we will deprecate the old UI and eventually remove it
>>> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
>>> two UIs from forking into separate paths, as that would become very
>>> difficult to maintain.
>>> 
>>> Going forward while both UIs are up, if you are making a change to any
>>> files in airflow/www/ (old UI), where applicable, please also make the
>>> change to the airflow/www_rbac/ (new UI). If you rather not make changes
>> in
>>> both UIs, it is recommended that you only make the changes to the RBAC
>> UI,
>>> since that is the one we are maintaining in the long term.
>>> 
>>> I'm excited that the RBAC UI will be able to bring additional security to
>>> Airflow, and with FAB framework in place we can look into leveraging it
>> for
>>> a unified set of APIs used by both UI and CLI.
>>> 
>>> Joy
>>> 
>>> 
>>> 
>>>> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <joyg@wepay.com> wrote:
>>>> 
>>>> Hi folks,
>>>> 
>>>> I have a PR <https://github.com/apache/incubator-airflow/pull/3015>
>> out
>>>> for the new UI. I've included instructions on how to test it out in the
>>> PR
>>>> description. Looking forward to your feedbacks.
>>>> 
>>>> Cheers,
>>>> Joy
>>>> 
>>>>> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <joyg@wepay.com> wrote:
>>>>> 
>>>>> Thanks for the background info. Would be really awesome for you to
>> have
>>>>> PyPi access :D I'll make the change to have Airflow Webserver's FAB
>>>>> dependency pointing to my fork for the mean time.
>>>>> 
>>>>> For folks who are interested in RBAC, I will be giving a talk/demo at
>>> the Airflow
>>>>> Meet-Up
>>>>> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
>>> Meetup/events/244525050/>
>>>>> next Monday. Happy to chat afterwards about it as well :)
>>>>> 
>>>>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
>>>>> maximebeauchemin@gmail.com> wrote:
>>>>> 
>>>>>> A bit of related history here:
>>>>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>>>>>> 
>>>>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>> 
>>>>>>> Given I have merge rights on FAB I could probably do another
round
>> of
>>>>>>> review and get your PRs through. I would really like to get the
>> main
>>>>>>> maintainer's input on things that touch the core (composite-key
>>>>>> support) as
>>>>>>> he might have concerns/intuitions that we can't know about.
>>>>>>> 
>>>>>>> I do not have Pypi access though so I cannot push new releases
>> out. I
>>>>>>> could ask for that.
>>>>>>> 
>>>>>>> I've threatened to fork the project before, that's always an
>> option.
>>>>>> I've
>>>>>>> noticed his involvement is sporadic and comes in bursts.
>>>>>>> 
>>>>>>> In the meantime, you can have the dependency in Airflow Webserver
>>>>>> pointing
>>>>>>> straight to your fork.
>>>>>>> 
>>>>>>> Max
>>>>>>> 
>>>>>>>> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <joyg@wepay.com>
wrote:
>>>>>>>> 
>>>>>>>> I just created a new webserver instance if you haven't gotten
a
>>>>>> chance to
>>>>>>>> fiddle around with the new web UI and the RBAC configurations
>>> (thanks
>>>>>>>> Maxime for getting started with this earlier!):
>>>>>>>> 
>>>>>>>> http://104.209.38.171:8080/
>>>>>>>> 
>>>>>>>> Admin Account
>>>>>>>> username: admin
>>>>>>>> password: admin
>>>>>>>> 
>>>>>>>> Read-Only Account
>>>>>>>> username: viewer
>>>>>>>> password: password
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <joyg@wepay.com>
wrote:
>>>>>>>>> 
>>>>>>>>> Hi folks,
>>>>>>>>> 
>>>>>>>>> Thanks for all the feedback regarding to the new Airflow
>> Webserver
>>>>>> UI
>>>>>>>>> <https://github.com/wepay/airflow-webserver/>!
I've been
>> actively
>>>>>>>>> addressing all the bugs that were raised on Github. So
I want to
>>>>>> take
>>>>>>>> this
>>>>>>>>> opportunity to discuss two issues coming up:
>>>>>>>>> 
>>>>>>>>> The first issue is unaddressed PRs in FAB. If these PRs
continue
>>> to
>>>>>> stay
>>>>>>>>> unaddressed, RBAC is blocked from making further progress.
If
>> this
>>>>>>>> continue
>>>>>>>>> to be an issue, I'm inclined to fork FAB, even though
it's not
>>>>>>>> idealistic.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>   - PR/631 <https://github.com/dpgaspar/F
>>> lask-AppBuilder/pull/631>
>>>>>>>> Binary
>>>>>>>>>   column support (merged, unreleased)
>>>>>>>>>   <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>>>>>>>>>   - PR/639 <https://github.com/dpgaspar/F
>>> lask-AppBuilder/pull/639>
>>>>>>>> Composite
>>>>>>>>>   primary key support (unmerged)
>>>>>>>>>   - PR/655 <https://github.com/dpgaspar/F
>>> lask-AppBuilder/pull/655>
>>>>>>>> Form
>>>>>>>>>   prefill support (unmerged)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> The second issue is an open question about the next step
of
>>> Airflow
>>>>>>>>> Webserver itself. Here are the 3 potential directions
we could
>>>>>> take, and
>>>>>>>>> I've added my thought on each.
>>>>>>>>> 
>>>>>>>>> 1. Permanently keep Airflow Webserver as a separated
package
>> from
>>>>>>>> Airflow,
>>>>>>>>> and treat it as another UI option. Keep `www` in Airflow.
Allow
>>>>>>>> development
>>>>>>>>> on both UIs.
>>>>>>>>> *I'm not a fan of this. When there is an existing UI
in Airflow,
>>>>>> most
>>>>>>>>> contributors would prefer to maintain the official version
that
>> is
>>>>>>>>> installed out-of-the-box. **Having a second UI outside
of
>> Airflow
>>>>>> will
>>>>>>>>> make maintaining it very difficult, leading to an eventual
death
>>> of
>>>>>> the
>>>>>>>> new
>>>>>>>>> UI :(*
>>>>>>>>> 
>>>>>>>>> 2. Permanently keep Airflow Webserver as a separated
package
>> from
>>>>>>>> Airflow,
>>>>>>>>> but freeze all development on `www`  and direct all future
UI
>>>>>>>> development
>>>>>>>>> to Airflow Webserver, eventually removing `www` completely
when
>>>>>> Airflow
>>>>>>>>> Webserver is stable.
>>>>>>>>> *I'm not a fan of this either. First of all, the views
and
>> models
>>>>>> are
>>>>>>>>> tightly coupled in both old and new UI; until we have
a
>>> full-fledged
>>>>>>>> REST
>>>>>>>>> API to build the UI (and cli) on top of it, separating
them to a
>>>>>>>> separate
>>>>>>>>> package now will potentially cause dependency issues
and add
>>>>>>>> complication
>>>>>>>>> to our release cycle. **Secondly, **majority of Airflow
users
>> run
>>>>>>>> Airflow
>>>>>>>>> with the UI; it's one of Airflow's best features. Separating
UI
>>> out
>>>>>> of
>>>>>>>>> Airflow core will complicate setup and configuration,
while
>> making
>>>>>>>> Airflow
>>>>>>>>> core less complete.*
>>>>>>>>> 
>>>>>>>>> 3. Merge Airflow Webserver back into Airflow as `www2`,
freeze
>> all
>>>>>>>>> development on `www`, eventually removing `www` completely
when
>>>>>> `www2`
>>>>>>>> is
>>>>>>>>> stable.
>>>>>>>>> *This makes the most sense to me. Airflow Webserver is
developed
>>>>>> with
>>>>>>>> the
>>>>>>>>> goal of feature parity to the current UI, plus additional
RBAC
>>>>>>>> capability,
>>>>>>>>> in hope to replace the old UI completely. Yes, this means
there
>>>>>> will be
>>>>>>>> a
>>>>>>>>> short period of having to maintain two UIs, but once
we freeze
>>>>>>>> development
>>>>>>>>> on www, it shouldn't be a concern for long.*
>>>>>>>>> 
>>>>>>>>> I'd love to hear everyone's thoughts on this! I'm excited
about
>>>>>> bringing
>>>>>>>>> RBAC to airflow and I hope it's something others will
find
>> useful
>>> as
>>>>>>>> well!
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> Joy
>>>>>>>>> 
>>>>>>>>> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <joyg@wepay.com>
>> wrote:
>>>>>>>>> 
>>>>>>>>>> Thank you everyone for the active feedback so far,
and thanks
>> for
>>>>>>>> setting
>>>>>>>>>> up the demo Maxime!
>>>>>>>>>> 
>>>>>>>>>> Going to work on pruning through the issues in the
upcoming
>> days.
>>>>>>>>>> 
>>>>>>>>>> Fokko/Maxime, do you recall the SQLAlchemy Exception
message
>> so I
>>>>>> can
>>>>>>>>>> look into it? Otherwise I'll wait until it's down
again =P
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> 
>>>>>>>>>> Joy
>>>>>>>>>> 
>>>>>>>>>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin
<
>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> I just restarted it, not sure how long it will
take to get in
>> a
>>>>>> bad
>>>>>>>> state
>>>>>>>>>>> again...
>>>>>>>>>>> 
>>>>>>>>>>> Max
>>>>>>>>>>> 
>>>>>>>>>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong,
Fokko
>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Good morning,
>>>>>>>>>>>> 
>>>>>>>>>>>> The demo provided by Max is down, it throws
a
>>>>>> SQLAlchemyexception
>>>>>>>> :'(
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>> 
>>>>>>>>>>>> 2017-11-18 19:14 GMT+01:00 Chris Riccomini
<
>>>>>> criccomini@apache.org>:
>>>>>>>>>>>> 
>>>>>>>>>>>>> @bolke, open issues on the Github repo,
please.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Nov 18, 2017 at 10:13 AM, Bolke
de Bruin <
>>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Chris,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Do you want us to report bugs somewhere
(I have
>>> encountered
>>>>>> a
>>>>>>>>>>> few)? Or
>>>>>>>>>>>>>> just generic user experiences posted
here?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 18 Nov 2017, at 00:47, Chris
Riccomini <
>>>>>>>> criccomini@apache.org
>>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I know the weekend is coming
up, and for those of us
>> in
>>>>>> the
>>>>>>>> US,
>>>>>>>>>>> next
>>>>>>>>>>>>> week
>>>>>>>>>>>>>>> is a bit of a slow holiday week.
Would love to get
>> some
>>>>>>>> feedback
>>>>>>>>>>> from
>>>>>>>>>>>>>>> everyone on this. The goal would
ideally to be to
>>>>>> converge on
>>>>>>>>>>> this
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> eventually replace the existing
Airflow UI with this
>>> one.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Nov 17, 2017 at 1:44
PM, Joy Gao <
>>> joyg@wepay.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi guys.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I've been working on moving
airflow from Flask-Admin
>> to
>>>>>>>>>>>>> Flask-AppBuilder
>>>>>>>>>>>>>>>> for RBAC
>>>>>>>>>>>>>>>> <https://cwiki.apache.org/
>> confluence/display/AIRFLOW/
>>>>>>>>>>>>>> Airflow+RBAC+proposal
>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>> check it out at https://github.com/wepay/airfl
>>>>>> ow-webserver.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> It's still a work-in-progress,
but most features you
>>> see
>>>>>> in
>>>>>>>> the
>>>>>>>>>>>>>> webserver
>>>>>>>>>>>>>>>> UI today is available there.
For those who are
>>>>>> interested in
>>>>>>>>>>> RBAC,
>>>>>>>>>>>> I'd
>>>>>>>>>>>>>> love
>>>>>>>>>>>>>>>> to get some early feedback
in terms of the following:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - New Flask-AppBuilder UI
(any bugs/regressions)
>>>>>>>>>>>>>>>> - Setup issues
>>>>>>>>>>>>>>>> - Ease of integration with
third party auth (i.e.
>> LDAP,
>>>>>> AD,
>>>>>>>>>>> OAuth,
>>>>>>>>>>>>> etc.)
>>>>>>>>>>>>>>>> - Any other thoughts/concerns
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks a lot!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Joy
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Mime
View raw message