airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <>
Subject Re: 1.10.0beta1 now available for download
Date Tue, 01 May 2018 21:31:28 GMT
Hi Jakob,

To be honest I’m confused now. In software land (and I assume you know)
Alpha -> Beta -> RC -> Release is well known and so well established that I would

be surprised if anyone got confused by that. Even the oldest project from Apache
have alpha-s and beta-s ( and something
called GA which is equal to a release I guess.

If you would expect people to pick up from a git tag and build from there and then report
to us, that doesn’t really happen. We are always having a challenge to have enough test
that would diminish that surface.

Other projects also “publish” other than voted upon artefacts. E.g. Spark has nightly
builds and SNAPSHOTS.
A snapshot clearly has a different state than a nightly. Apache Flink state that 1.4.2 is
their latest stable release.
So there seems to be a “non-stable” release as well. I did see that their git repositories
only mention “RC-X” tags
or branches.

Reading through it does not mention
that we need to have RCs. It just states that if you want to do a release you need to call
a vote and for distribution
it must be at a certain location. As mentioned this is a “beta” which is not a “release”.
We haven’t released it either as 
it wasn’t voted upon and no vote was called. It was just made available for convenience
of the community.

So I am not sure what is expected from us here. How do wo go though dev -> test -> acc
-> prod release process 
together with the community? The release process you seem to be referring is only part of
the last state imho. Or
do we need to call a vote on every state change?


> On 1 May 2018, at 22:47, Jakob Homan <> wrote:
> Hey Bolke-
>   To be clear, I'm not suggesting anyone is trying to do anything
> wrong.  Release wasn't mentioned, but a new tar ball with a new
> version number with a 'beta' tag is published in some way for people
> to come and test.  How is that different than the expected release/RC
> process (specify a git point, offer a tar ball, add an RCx tag and
> invite people to test that)?  Seems like a parallel process with lots
> of similarities that could confuse both our end users and the IPMC.
> Thanks,
> Jakob
> On 1 May 2018 at 13:08, Bolke de Bruin <> wrote:
>> Hi Jakob,
>> Understood. But isn’t that in this case not just wording? Ie. this is a tar-ball
that we think is beyond just developer testing (alpha) but more towards the enthusiasts (beta)
but not a version of the tarball that is for the general public to test (RC) and not a Release
(release)? Ie. is the issue in calling it a ‘release’ which in this case is just meta
for a tarball? In the original email in never mentioned the word release in conjunction with
the beta I think.
>> Cheers
>> Bolke
>>> On 1 May 2018, at 22:01, Jakob Homan <> wrote:
>>> Hey all-
>>> With my Mentor hat on, I need to point out that ASF doesn't really
>>> have beta releases.  This work is awesome, but really needs to go
>>> through the proper steps.  The Release Candidate process is pretty
>>> well described:
>>>  This is
>>> particularly important since, as was mentioned, graduation should be
>>> imminent and this process will be heavily scrutinized.
>>> -Jakob
>>> On 1 May 2018 at 12:41, James Meickle <> wrote:
>>>> Thanks for the pointer! I went through and set this up today, using Google
>>>> OAuth as the RBAC provider. Overall I'm quite enthusiastic about this move,
>>>> but I thought that it might be helpful to collect feedback as someone who
>>>> hasn't been following the overall process and is therefore coming at it
>>>> with fresh eyes.
>>>> - The Flask appbuilder security documentation is poor quality (e.g.,
>>>> there's some broken sentences); if Airflow is to send people there, it
>>>> might be worth PRing some of the docs to at least look more professional.
>>>> - There's not much documentation out there on how to properly set up an
>>>> OAuth app in Google (in my case, using the G+ API). From an adoption POV,
>>>> it would be good to screenshot the (current) steps in the process, and
>>>> point out which values should be used in which fields on Google. For
>>>> example, I had to grep the code base to find the callback URL.
>>>> - The initial login UI seems over-complex: you have to click the provider
>>>> icon, and then click either login or register. The standard for this
>>>> workflow is that you login by clicking the desired provider's icon, and
>>>> doing so will register you automatically if you aren't already. In my case
>>>> I only have one provider, so this menu was even more confusing.
>>>> - It was not clear to me that the "Public" role has absolutely no
>>>> permissions. When I set this as the default role and registered, I could
>>>> longer access the site until I cleared cookies. I thought it was an OAuth
>>>> error at first, but it turns out the Public role has fewer effective
>>>> permissions than an anonymous user; this resulted in a redirect loop
>>>> because I could not even view the homepage. I had to correct this in the
>>>> database to be able to log in.
>>>> - The roles list (at roles/list/ ) is intimidatingly large and hard to
>>>> parse. For instance, I couldn't tell at a glance what "user" allows
>>>> relative to "viewer". It would be good to have a narrative description of
>>>> what each of these roles is intended for, and to present the list of
>>>> permissions in a more clustered or diffable way. Permissions lists tend to
>>>> only grow, after all.
>>>> - A "Viewer" currently lacks enough access to see their own profile.
>>>> - "User Statistics" (userstatschartview/chart/) uses the internal name,
>>>> rather than firstname/lastname - which in my case is a `google_idnumber`
>>>> name. Should probably show both names.
>>>> Unrelatedly to RBAC (I think), on this branch on my sandbox instance, tasks
>>>> appear to be failing with the only logs present in the UI as:
>>>> [{'end_of_log': True}, {'end_of_log': True}, {'end_of_log': True},
>>>> {'end_of_log': True}, {'end_of_log': True}, {'end_of_log': True}]
>>>> Finally, in case anyone else wanted to test run a similar setup, here is
>>>> the that I ended up using (note that it has Jinja
>>>> templating via Ansible):
>>>> import os
>>>> from airflow import configuration as conf
>>>> from import AUTH_OAUTH
>>>> basedir = os.path.abspath(os.path.dirname(__file__))
>>>> # The SQLAlchemy connection string.
>>>> # Flask-WTF flag for CSRF
>>>> CSRF_ENABLED = True
>>>> # The name to display, e.g. "Airflow Staging Sandbox"
>>>> APP_NAME = "Airflow {{ env }} {{ app_config | capitalize }}"
>>>> # Use OAuth
>>>> # Will allow user self registration
>>>> # The default user self registration role
>>>> AUTH_USER_REGISTRATION_ROLE = "{{ airflow_rbac_registration_role |
>>>> default('Viewer') }}"
>>>> # Google OAuth:
>>>> # The name of the provider
>>>> 'name': 'google',
>>>> # The icon to use
>>>> 'icon': 'fa-google',
>>>> # The name of the key that the provider sends
>>>> 'token_key': 'access_token',
>>>> # Just in case, whitelist to only emails
>>>> 'whitelist': [''],
>>>> # Define the remote app:
>>>> 'remote_app': {
>>>> 'base_url': '',
>>>> 'access_token_url': '',
>>>> 'authorize_url': '',
>>>> 'request_token_url': None,
>>>> 'request_token_params': {
>>>> # Uses the Google+ API, requestingf the 'email' and 'profile' scope
>>>> 'scope': 'email profile'
>>>> },
>>>> 'consumer_key': '{{ vault_airflow_google_oauth_key }}',
>>>> 'consumer_secret': '{{ vault_airflow_google_oauth_secret }}'
>>>> }
>>>> }]
>>>> On Mon, Apr 30, 2018 at 12:54 PM, Jørn A Hansen <>
>>>> wrote:
>>>>> On Mon, 30 Apr 2018 at 15.56, James Meickle <>
>>>>> wrote:
>>>>>> Installed this off of the branch, and I do get the Kubernetes executor
>>>>>> (incl. demo DAG) and some bug fixes - but I don't see any RBAC feature
>>>>>> anywhere I'd think to look. Do I need to set up some config to get
>>>>> to
>>>>>> show up?
>>>>> See
>>>>> test/
>>>>> It had me left wondering as well - so I decided to go hunt for it in
>>>>> RBAC PR. And there it was :-)
>>>>> Cheers,
>>>>> JornH
>>>>>> On Mon, Apr 23, 2018 at 2:06 PM, Bolke de Bruin <>
>>>>> wrote:
>>>>>>> Hi All,
>>>>>>> I am really happy that Fokko and I have created the v1-10-test
>>>>> and
>>>>>>> subsequently build the first beta of Apache Airflow 1.10!
>>>>>>> It is available for testing here:
>>>>>>> Highlights include:
>>>>>>> * New RBAC web interface in beta
>>>>>>> * Timezone support
>>>>>>> * First class kubernetes operator
>>>>>>> * Experimental kubernetes executor
>>>>>>> * Documentation improvements
>>>>>>> * Performance optimizations for large DAGs
>>>>>>> * many GCP and S3 integration improvements
>>>>>>> * many new operators
>>>>>>> * many many many bug fixes
>>>>>>> We are aiming for a fully compliant Apache release so we should
be able
>>>>>> to
>>>>>>> kick off the graduation process after this release. I hope you
help us
>>>>>> out
>>>>>>> getting there!
>>>>>>> Kind regards,
>>>>>>> Bolke & Fokko

View raw message