airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2
Date Thu, 06 Jul 2017 20:10:11 GMT
Hi Folks,

We probably need to adjust our release process as can be observed in the IPMC thread. As we
are packaging a “sdist” it does not pass license checks and one cannot verify the validity
of what we are doing. It was suggested by one of the maintainers of another python project
to create 3 different packages:

1. A source tarball which is essentially a snapshot of the repository
2. A sdist
3. A bdist

1 should then be the canonical Apache release. It should be accompanied by build instructions
and it should pass RAT checks. This is the package we will vote on.
2 is what we have voted upon until now. It should contain (it does) LICENSE, NOTICE, and DISCLAIMER
3 bdist, wheel package. Same as 2. Not really required, but more a convenience package as
is 2

2 and 3 can be published to PyPi.

Max: can you take care of this? We need to vote on 1.  Build instruction could be added to
an INSTALL file or just to the README.md file? See for inspiration the GitHub page of ariatosca:
https://github.com/apache/incubator-ariatosca <https://github.com/apache/incubator-ariatosca>

In the meantime I am figuring out an issue with one of the dependencies of nvd3 which might
be or have been GPL3 which is incompatible with the APL, which we are distributing together
with our source. Ideally, we should move to a “yarn/webpack” build which will resolve
those issues automatically as these become runtime dependencies then in case of 1,2,3.

Cheers
Bolke



> On 6 Jul 2017, at 05:20, Sumit Maheshwari <sumeet.manit@gmail.com> wrote:
> 
> Awesome.. thanks a lot Max for taking the RM responsibility..
> 
> On Jul 5, 2017 11:10 PM, "Chris Riccomini" <criccomini@apache.org> wrote:
> 
>> Nice, thanks Max!
>> 
>> On Mon, Jul 3, 2017 at 2:36 PM, Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>> 
>>> Hello,
>>> 
>>> Apache Airflow (incubating) 1.8.2 (RC2) has been accepted.
>>> 
>>> 5 “+1” votes received, 4 out of which are "binding":
>>> 
>>> - Maxime Beauchemin (binding)
>>> - Bolke de Bruin (binding)
>>> - Chris Riccomini (binding)
>>> - Sumit Maheshwari  (binding)
>>> 
>>> - Kengo Seki  (non-binding)
>>> 
>>> Vote thread (start):
>>> *http://mail-archives.apache.org/mod_mbox/incubator-
>>> airflow-dev/201706.mbox/%3CCAHEEp7XRRk__hGUMjOCw%2BTW0Gc--
>>> 78qEewcsAsQBLrv1x6j71w%40mail.gmail.com%3E
>>> <http://mail-archives.apache.org/mod_mbox/incubator-
>>> airflow-dev/201706.mbox/%3CCAHEEp7XRRk__hGUMjOCw%2BTW0Gc--
>>> 78qEewcsAsQBLrv1x6j71w%40mail.gmail.com%3E>*
>>> 
>>> Next steps:
>>> 1) will start the voting process at the IPMC mailinglist. I don’t expect
>>> changes.
>>> 2) Only after the positive voting on the IPMC and finalisation I will
>>> rebrand the RC to Release.
>>> 3) I will upload it to the incubator release page, then the tar ball
>> needs
>>> to propagate to the mirrors.
>>> 4) Update the website
>>> 5) Finally, will upload to Pypi.
>>> 
>>> Cheers,
>>> 
>>> Max
>>> 
>>> On Mon, Jul 3, 2017 at 12:11 PM, Bolke de Bruin <bdbruin@gmail.com>
>> wrote:
>>> 
>>>> Ping?
>>>> 
>>>> 
>>>>> On 30 Jun 2017, at 14:21, Bolke de Bruin <bdbruin@gmail.com> wrote:
>>>>> 
>>>>> Max,
>>>>> 
>>>>> I think you can close the vote?
>>>>> 
>>>>> Bolke
>>>>> 
>>>>>> On 27 Jun 2017, at 02:45, Kengo Seki <sekikn@apache.org> wrote:
>>>>>> 
>>>>>> +1 (non-binding)
>>>>>> 
>>>>>> - verified signatures and checksums
>>>>>> - ran scheduler and webserver, confirmed they worked fine
>>>>>> - confirmed the latest fix on v1.8 branch (AIRFLOW-809) is included
>>>>>> 
>>>>>> Kengo Seki <sekikn@apache.org>
>>>>>> 
>>>>>> 
>>>>>> 2017-06-27 8:53 GMT+09:00 Chris Riccomini <criccomini@apache.org>:
>>>>>>> +1 (binding)
>>>>>>> 
>>>>>>> Been running in our dev env, and everything looks good.
>>>>>>> 
>>>>>>> On Mon, Jun 26, 2017 at 3:00 PM, Alex Guziel <
>> alex.guziel@airbnb.com
>>> .
>>>> invalid
>>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Yeah that makes sense. It pages by default at 500 so it explains
>> why
>>>> we saw
>>>>>>>> it.
>>>>>>>> 
>>>>>>>> On Mon, Jun 26, 2017 at 2:53 PM, Chris Riccomini <
>>>> criccomini@apache.org>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> In 1.8.1, the "DAGs" page has "Show <blank> entries".
In 1.8.2,
>> it
>>>> has
>>>>>>>>> "Show <25> entries". So it looks like prior to
1.8.2, the
>>> pagination
>>>> was
>>>>>>>>> broken in the sense that it defaulted to the whole list.
We have
>>> 479
>>>> DAGs
>>>>>>>>> in one env, and it shows them all. It looks like someone
fixed
>> the
>>>> entry
>>>>>>>> to
>>>>>>>>> default to 25 now, which exposed the problem for our
>> environments.
>>>>>>>>> 
>>>>>>>>> On Mon, Jun 26, 2017 at 2:47 PM, Alex Guziel <
>>> alex.guziel@airbnb.com
>>>> .
>>>>>>>>> invalid
>>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> We're running 1.8.0 + some extras, and none of us
added
>> pagination
>>>>>>>>>> recently, and our homepage is paginated. Are you
sure it's not
>> the
>>>>>>>> number
>>>>>>>>>> of dags crossing the threshold? Maybe it's some Flask
version
>>> thing?
>>>>>>>>>> 
>>>>>>>>>> On Mon, Jun 26, 2017 at 2:45 PM, Chris Riccomini
<
>>>>>>>> criccomini@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Yes, I did the 1.8.1 release.
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:44 PM, Alex Guziel
<
>>>> alex.guziel@airbnb.com
>>>>>>>> .
>>>>>>>>>>> invalid
>>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> There's no pagination in 1.8.1? Are you sure?
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:37 PM, Chris Riccomini
<
>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> It's not happening on 1.8.1 (since there's
no pagination in
>>> that
>>>>>>>>>>>> version),
>>>>>>>>>>>>> so I'd count this as a regression. I
wouldn't say it's
>>> blocking,
>>>>>>>>> but
>>>>>>>>>>> it's
>>>>>>>>>>>>> pretty ugly.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:34 PM, Alex
Guziel <
>>>>>>>>> alex.guziel@airbnb.com
>>>>>>>>>> .
>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm not so sure this is a new issue.
I think we've seen it
>> on
>>>>>>>> our
>>>>>>>>>>>>>> production for quite a while.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:31 PM,
Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I am seeing a strange UI behavior
on 1.8.2.RC2. I've
>> opened a
>>>>>>>>>> JIRA
>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1348
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Has anyone else seen this?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 3:27
AM, Sumit Maheshwari <
>>>>>>>>>>>>>> sumeet.manit@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> +1, binding.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 3:49
PM, Bolke de Bruin <
>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> We have been running
it for the last couple of days. No
>>>>>>>>>> issues
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> more responsive.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> +1, binding
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On 25 Jun 2017, at
01:10, Maxime Beauchemin <
>>>>>>>>>>>>>>>> maximebeauchemin@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 1.8.2 RC2 is baked
and available at:
>>>>>>>>>>>>>>>>>> https://dist.apache.org/repos/
>>>>>>>> dist/dev/incubator/airflow
>>>>>>>>> ,
>>>>>>>>>>>> public
>>>>>>>>>>>>>>> keys
>>>>>>>>>>>>>>>>>> are available
>>>>>>>>>>>>>>>>>> at https://dist.apache.org/repos/
>>>>>>>>>>>> dist/release/incubator/airflow.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Note that RC1 was
the first RC (skipped RC0) and was
>>>>>>>>> never
>>>>>>>>>>>>>> announced
>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>> it had issues coming
out of the oven, so RC2 is the
>>>>>>>> first
>>>>>>>>>>>> public
>>>>>>>>>>>>>> RC.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 1.8.2 RC2 is build
on to of 1.8.1 with these listed
>>>>>>>>>>> "cherries"
>>>>>>>>>>>> on
>>>>>>>>>>>>>>> top.
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>> added the JIRAs that
were identified blockers and
>>>>>>>>> targeted
>>>>>>>>>>>>> 1.8.2. I
>>>>>>>>>>>>>>>>>> attempted to bring
in all of the JIRAs that targeted
>>>>>>>>> 1.8.2
>>>>>>>>>>> but
>>>>>>>>>>>>>> bailed
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> the ones that were
generating merge conflicts. I also
>>>>>>>>> added
>>>>>>>>>>> all
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> JIRAs that we've
been running in production at Airbnb.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Issues fixed:
>>>>>>>>>>>>>>>>>> 9a53e66 [AIRFLOW-809][AIRFLOW-1]
Use __eq__
>>>>>>>>> ColumnOperator
>>>>>>>>>>> When
>>>>>>>>>>>>>>> Testing
>>>>>>>>>>>>>>>>>> Booleans
>>>>>>>>>>>>>>>>>> 333e0b3 [AIRFLOW-1296]
Propagate SKIPPED to all
>>>>>>>>> downstream
>>>>>>>>>>>> tasks
>>>>>>>>>>>>>>>>>> 93825d5 [AIRFLOW-XXX]
Re-enable caching for hadoop
>>>>>>>>>> components
>>>>>>>>>>>>>>>>>> 33a9dcb [AIRFLOW-XXX]
Pin Hive and Hadoop to a specific
>>>>>>>>>>> version
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>> writable warehouse
dir
>>>>>>>>>>>>>>>>>> 7cff6cd [AIRFLOW-1308]
Disable nanny usage for Dask
>>>>>>>>>>>>>>>>>> 570b2ed [AIRFLOW-1294]
Backfills can loose tasks to
>>>>>>>>> execute
>>>>>>>>>>>>>>>>>> 3f48d48 [AIRFLOW-1291]
Update NOTICE and LICENSE files
>>>>>>>> to
>>>>>>>>>>> match
>>>>>>>>>>>>> ASF
>>>>>>>>>>>>>>>>>> requirements
>>>>>>>>>>>>>>>>>> 69bd269 [AIRFLOW-1160]
Update Spark parameters for
>>>>>>>> Mesos
>>>>>>>>>>>>>>>>>> 9692510 [AIRFLOW
1149][AIRFLOW-1149] Allow for custom
>>>>>>>>>> filters
>>>>>>>>>>>> in
>>>>>>>>>>>>>>> Jinja2
>>>>>>>>>>>>>>>>>> templates
>>>>>>>>>>>>>>>>>> 6de5330 [AIRFLOW-1119]
Fix unload query so headers are
>>>>>>>> on
>>>>>>>>>>> first
>>>>>>>>>>>>>> row[]
>>>>>>>>>>>>>>>>>> b4e9eb8 [AIRFLOW-1089]
Add Spark application arguments
>>>>>>>>>>>>>>>>>> a4083f3 [AIRFLOW-1078]
Fix latest_runs endpoint for old
>>>>>>>>>> flask
>>>>>>>>>>>>>>> versions
>>>>>>>>>>>>>>>>>> 7a02841 [AIRFLOW-1074]
Don't count queued tasks for
>>>>>>>>>>> concurrency
>>>>>>>>>>>>>>> limits
>>>>>>>>>>>>>>>>>> a2c18a5 [AIRFLOW-1064]
Change default sort to job_id
>>>>>>>> for
>>>>>>>>>>>>>>>>>> TaskInstanceModelView
>>>>>>>>>>>>>>>>>> d1c64ab [AIRFLOW-1038]
Specify celery serialization
>>>>>>>>> options
>>>>>>>>>>>>>>> explicitly
>>>>>>>>>>>>>>>>>> b4ee88a [AIRFLOW-1036]
Randomize exponential backoff
>>>>>>>>>>>>>>>>>> 9fca409 [AIRFLOW-993]
Update date inference logic
>>>>>>>>>>>>>>>>>> 272c2f5 [AIRFLOW-1167]
Support microseconds in FTPHook
>>>>>>>>>>>>> modification
>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>> c7c0b72 [AIRFLOW-1179]
Fix Pandas 0.2x breaking Google
>>>>>>>>>>> BigQuery
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>> acd0166 [AIRFLOW-1263]
Dynamic height for charts
>>>>>>>>>>>>>>>>>> 7f33f6e [AIRFLOW-1266]
Increase width of gantt y axis
>>>>>>>>>>>>>>>>>> fc33c04 [AIRFLOW-1290]
set docs author to 'Apache
>>>>>>>>> Airflow'
>>>>>>>>>>>>>>>>>> 2e9eee3 [AIRFLOW-1282]
Fix known event column sorting
>>>>>>>>>>>>>>>>>> 2389a8a [AIRFLOW-1166]
Speed up
>>>>>>>>>>> _change_state_for_tis_without_
>>>>>>>>>>>>>> dagrun
>>>>>>>>>>>>>>>>>> bf966e6 [AIRFLOW-1192]
Some enhancements to
>>>>>>>>> qubole_operator
>>>>>>>>>>>>>>>>>> 57d5bcd [AIRFLOW-1281]
Sort variables by key field by
>>>>>>>>>> default
>>>>>>>>>>>>>>>>>> 802fc15 [AIRFLOW-1244]
Forbid creation of a pool with
>>>>>>>>> empty
>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>> 1232b6a [AIRFLOW-1243]
DAGs table has no default
>>>>>>>> entries
>>>>>>>>> to
>>>>>>>>>>>> show
>>>>>>>>>>>>>>>>>> b0ba3c9 [AIRFLOW-1227]
Remove empty column on the Logs
>>>>>>>>> view
>>>>>>>>>>>>>>>>>> c406652 [AIRFLOW-1226]
Remove empty column on the Jobs
>>>>>>>>> view
>>>>>>>>>>>>>>>>>> 51a83cc [AIRFLOW-1199]
Fix create modal
>>>>>>>>>>>>>>>>>> cac7d4c [AIRFLOW-1200]
Forbid creation of a variable
>>>>>>>> with
>>>>>>>>>> an
>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>> key
>>>>>>>>>>>>>>>>>> 5f3ee52 [AIRFLOW-1186]
Sort dag.get_task_instances by
>>>>>>>>>>>>>> execution_date
>>>>>>>>>>>>>>>>>> f446c08 [AIRFLOW-1145]
Fix closest_date_partition
>>>>>>>>> function
>>>>>>>>>>> with
>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>> to True If we're
looking for the closest date before,
>>>>>>>> we
>>>>>>>>>>> should
>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> latest date in the
list of date before.
>>>>>>>>>>>>>>>>>> 93b8e96 [AIRFLOW-1180]
Fix flask-wtf version for
>>>>>>>>>>>>>> test_csrf_rejection
>>>>>>>>>>>>>>>>>> bb56805 [AIRFLOW-1170]
DbApiHook insert_rows inserts
>>>>>>>>>>> parameters
>>>>>>>>>>>>>>>>> separately
>>>>>>>>>>>>>>>>>> 093b2f0 [AIRFLOW-1150]
Fix scripts execution in
>>>>>>>> sparksql
>>>>>>>>>>> hook[]
>>>>>>>>>>>>>>>>>> 777f181 [AIRFLOW-1168]
Add closing() to all connections
>>>>>>>>> and
>>>>>>>>>>>>> cursors
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Max
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message