airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2
Date Tue, 18 Jul 2017 07:01:31 GMT
Thanks Hitesh. We discussed it with John Ament on the IPMC. Python has the notion of 3 types of distributions, “source”, “sdist”, “bdist”, contrary to Java that knows only two (source, bdist). We used to vote on “sdist”, which was deemed incorrect.

So, Max, indeed we need to vote on a tar.gz that contains build instructions in INSTALL to get to “sdist”. The build instructions should also contain instruction how to run the license checks by Apache Rat. Most of the work probably goes in the build instructions and verifying they work, but it should not be much.

Any other clarification required?

Bolke


> On 18 Jul 2017, at 00:36, Hitesh Shah <hitesh@apache.org> wrote:
> 
> The main artifact to vote upon on is a source release. This implies just
> creating a clean tarball of the source codebase with an appropriate README,
> INSTALL, LICENSE, DISCLAIMER, NOTICE (all of these should be checked in to
> the codebase as the release should map to a specific commit hash which can
> be re-used to create the source tarball again if needed). License and
> Notice files should cover all files in the codebase.
> 
> You can also publish binary build(s) as a convenience ( assuming you want
> to publish to pypi) - this would be a separate tarball and likely different
> notice/license files (would suggest having these in the src repo too) as
> the binary tarball will likely have different files.
> 
> thanks
> -- Hitesh
> 
> 
> 
> On Mon, Jul 17, 2017 at 2:18 AM, Bolke de Bruin <bdbruin@gmail.com <mailto:bdbruin@gmail.com>> wrote:
> 
>> Great!
>> 
>> It is also a bit new to me so maybe @Hitesh @Jakob can help with some
>> guidance here?
>> 
>> But my assumption indeed is:
>> 
>> 1. Make a tarball from the repo with build instructions (including a
>> working License check!) -> Vote here and IPMC. This is the “official”
>> release.
>> 2. Make sdist for redistribution on PyPi
>> 
>> Bdist isn’t required.
>> 
>> Cheers
>> Bolke.
>> 
>>> On 17 Jul 2017, at 06:27, Maxime Beauchemin <maximebeauchemin@gmail.com>
>> wrote:
>>> 
>>> I've been slammed but skies are clearing up now I'm hoping.
>>> 
>>> Reading the general@ thread I'm unclear about the next steps, targz the
>>> whole repo and add build instructions? What should the file with the
>> build
>>> instructions be called? How to label that new tarbar? Can we skip the
>> bdist?
>>> 
>>> Max
>>> 
>>> On Sun, Jul 16, 2017 at 12:35 PM, Bolke de Bruin <bdbruin@gmail.com
>> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>>> wrote:
>>> 
>>>> Max, Ping? Do you need help?
>>>> 
>>>>> On 9 Jul 2017, at 14:30, Bolke de Bruin <bdbruin@gmail.com <mailto:bdbruin@gmail.com>> wrote:
>>>>> 
>>>>> Hi Max,
>>>>> 
>>>>> The canonical distribution would be what we have in git right now (ie.
>>>> before running python sdist). The rest is just convenience packages. So
>>>> npm, would solve the issue as long as we don’t rely on any non APL
>>>> compatible packages in core. I don’t think npm/yarn/webpack needs to be
>>>> done for 1.8.3, but considering the messy javascript that we currently
>> have
>>>> it would be nice to put it on the todo.
>>>>> 
>>>>> Cheers
>>>>> Bolke
>>>>> 
>>>>>> On 9 Jul 2017, at 06:46, Maxime Beauchemin <
>> maximebeauchemin@gmail.com <mailto:maximebeauchemin@gmail.com>
>>>> <mailto:maximebeauchemin@gmail.com <mailto:maximebeauchemin@gmail.com> <mailto:maximebeauchemin@gmail.com <mailto:maximebeauchemin@gmail.com>>>>
>> wrote:
>>>>>> 
>>>>>> As far as I understand npm would not solve the problem as typically
>> we'd
>>>>>> build our "entry" files and distribute that with Airflow as static
>>>> files.
>>>>>> Those entry files would contain these other npm packages, minified.
>>>> (from
>>>>>> my understanding that is the same issue as packaging the libs
>>>> themselves)
>>>>>> 
>>>>>> To make them runtime deps would be atypical and more complicated.
>>>> `airflow
>>>>>> webserver` would need to "build" (npm install/webpack) and the
>> webserver
>>>>>> would have to serve these static files out of some that temp location
>>>>>> (perhaps ~/.airflow/airflow.entry.js) as opposed to out of
>>>> `site-packages`.
>>>>>> 
>>>>>> Also note that Airflow's javascript is in pretty bad shape (scattered
>> in
>>>>>> jinja templates files) and it would take quite a significant amount of
>>>> work
>>>>>> to move to using npm/webpack.
>>>>>> 
>>>>>> I'm back from vacation and will have things to catch up on next week
>> but
>>>>>> I'll try to find time to look into some of this.
>>>>>> 
>>>>>> On Thu, Jul 6, 2017 at 1:10 PM, Bolke de Bruin <bdbruin@gmail.com <mailto:bdbruin@gmail.com>
>> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>>> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>>>> wrote:
>>>>>> 
>>>>>>> Hi Folks,
>>>>>>> 
>>>>>>> We probably need to adjust our release process as can be observed in
>>>> the
>>>>>>> IPMC thread. As we are packaging a “sdist” it does not pass license
>>>> checks
>>>>>>> and one cannot verify the validity of what we are doing. It was
>>>> suggested
>>>>>>> by one of the maintainers of another python project to create 3
>>>> different
>>>>>>> packages:
>>>>>>> 
>>>>>>> 1. A source tarball which is essentially a snapshot of the repository
>>>>>>> 2. A sdist
>>>>>>> 3. A bdist
>>>>>>> 
>>>>>>> 1 should then be the canonical Apache release. It should be
>>>> accompanied by
>>>>>>> build instructions and it should pass RAT checks. This is the package
>>>> we
>>>>>>> will vote on.
>>>>>>> 2 is what we have voted upon until now. It should contain (it does)
>>>>>>> LICENSE, NOTICE, and DISCLAIMER
>>>>>>> 3 bdist, wheel package. Same as 2. Not really required, but more a
>>>>>>> convenience package as is 2
>>>>>>> 
>>>>>>> 2 and 3 can be published to PyPi.
>>>>>>> 
>>>>>>> Max: can you take care of this? We need to vote on 1.  Build
>>>> instruction
>>>>>>> could be added to an INSTALL file or just to the README.md file? See
>>>> for
>>>>>>> inspiration the GitHub page of ariatosca: https://github.com/apache/ <https://github.com/apache/>
>> <https://github.com/apache/ <https://github.com/apache/>> <
>>>> https://github.com/apache/ <https://github.com/apache/> <https://github.com/apache/ <https://github.com/apache/>>>
>>>>>>> incubator-ariatosca <https://github.com/apache/incubator-ariatosca <https://github.com/apache/incubator-ariatosca> <
>> https://github.com/apache/incubator-ariatosca <https://github.com/apache/incubator-ariatosca>> <
>>>> https://github.com/apache/incubator-ariatosca <https://github.com/apache/incubator-ariatosca> <
>> https://github.com/apache/incubator-ariatosca <https://github.com/apache/incubator-ariatosca>>>>
>>>>>>> 
>>>>>>> In the meantime I am figuring out an issue with one of the
>>>> dependencies of
>>>>>>> nvd3 which might be or have been GPL3 which is incompatible with the
>>>> APL,
>>>>>>> which we are distributing together with our source. Ideally, we
>> should
>>>> move
>>>>>>> to a “yarn/webpack” build which will resolve those issues
>>>> automatically as
>>>>>>> these become runtime dependencies then in case of 1,2,3.
>>>>>>> 
>>>>>>> Cheers
>>>>>>> Bolke
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On 6 Jul 2017, at 05:20, Sumit Maheshwari <sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com>
>> <mailto:sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com>>
>>>> <mailto:sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com> <mailto:sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com>>>>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Awesome.. thanks a lot Max for taking the RM responsibility..
>>>>>>>> 
>>>>>>>> On Jul 5, 2017 11:10 PM, "Chris Riccomini" <criccomini@apache.org <mailto:criccomini@apache.org>
>> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>
>>>> <mailto:criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>>>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Nice, thanks Max!
>>>>>>>>> 
>>>>>>>>> On Mon, Jul 3, 2017 at 2:36 PM, Maxime Beauchemin <
>>>>>>>>> maximebeauchemin@gmail.com <mailto:maximebeauchemin@gmail.com> <mailto:maximebeauchemin@gmail.com <mailto:maximebeauchemin@gmail.com>>
>> <mailto:maximebeauchemin@gmail.com <mailto:maximebeauchemin@gmail.com> <mailto:maximebeauchemin@gmail.com <mailto:maximebeauchemin@gmail.com>>>>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hello,
>>>>>>>>>> 
>>>>>>>>>> Apache Airflow (incubating) 1.8.2 (RC2) has been accepted.
>>>>>>>>>> 
>>>>>>>>>> 5 “+1” votes received, 4 out of which are "binding":
>>>>>>>>>> 
>>>>>>>>>> - Maxime Beauchemin (binding)
>>>>>>>>>> - Bolke de Bruin (binding)
>>>>>>>>>> - Chris Riccomini (binding)
>>>>>>>>>> - Sumit Maheshwari  (binding)
>>>>>>>>>> 
>>>>>>>>>> - Kengo Seki  (non-binding)
>>>>>>>>>> 
>>>>>>>>>> Vote thread (start):
>>>>>>>>>> *http://mail-archives.apache.org/mod_mbox/incubator- <http://mail-archives.apache.org/mod_mbox/incubator-> <
>> http://mail-archives.apache.org/mod_mbox/incubator- <http://mail-archives.apache.org/mod_mbox/incubator->> <
>>>> http://mail-archives.apache.org/mod_mbox/incubator- <http://mail-archives.apache.org/mod_mbox/incubator-> <
>> http://mail-archives.apache.org/mod_mbox/incubator- <http://mail-archives.apache.org/mod_mbox/incubator->>>
>>>>>>>>>> airflow-dev/201706.mbox/%3CCAHEEp7XRRk__hGUMjOCw%2BTW0Gc--
>>>>>>>>>> 78qEewcsAsQBLrv1x6j71w%40mail.gmail.com <http://40mail.gmail.com/> <http://40mail.gmail.com/ <http://40mail.gmail.com/>>
>> <http://40mail.gmail.com/ <http://40mail.gmail.com/> <http://40mail.gmail.com/ <http://40mail.gmail.com/>>
>>>>> %3E
>>>>>>>>>> <http://mail-archives.apache.org/mod_mbox/incubator- <http://mail-archives.apache.org/mod_mbox/incubator-> <
>> http://mail-archives.apache.org/mod_mbox/incubator- <http://mail-archives.apache.org/mod_mbox/incubator->> <
>>>> http://mail-archives.apache.org/mod_mbox/incubator- <http://mail-archives.apache.org/mod_mbox/incubator-> <
>> http://mail-archives.apache.org/mod_mbox/incubator- <http://mail-archives.apache.org/mod_mbox/incubator->>>
>>>>>>>>>> airflow-dev/201706.mbox/%3CCAHEEp7XRRk__hGUMjOCw%2BTW0Gc--
>>>>>>>>>> 78qEewcsAsQBLrv1x6j71w%40mail.gmail.com <http://40mail.gmail.com/> <http://40mail.gmail.com/ <http://40mail.gmail.com/>>
>> <http://40mail.gmail.com/ <http://40mail.gmail.com/> <http://40mail.gmail.com/ <http://40mail.gmail.com/>>
>>>>> %3E>*
>>>>>>>>>> 
>>>>>>>>>> Next steps:
>>>>>>>>>> 1) will start the voting process at the IPMC mailinglist. I don’t
>>>>>>> expect
>>>>>>>>>> changes.
>>>>>>>>>> 2) Only after the positive voting on the IPMC and finalisation I
>>>> will
>>>>>>>>>> rebrand the RC to Release.
>>>>>>>>>> 3) I will upload it to the incubator release page, then the tar
>> ball
>>>>>>>>> needs
>>>>>>>>>> to propagate to the mirrors.
>>>>>>>>>> 4) Update the website
>>>>>>>>>> 5) Finally, will upload to Pypi.
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> 
>>>>>>>>>> Max
>>>>>>>>>> 
>>>>>>>>>> On Mon, Jul 3, 2017 at 12:11 PM, Bolke de Bruin <
>> bdbruin@gmail.com <mailto:bdbruin@gmail.com> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>>> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>>>>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Ping?
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On 30 Jun 2017, at 14:21, Bolke de Bruin <bdbruin@gmail.com <mailto:bdbruin@gmail.com>
>> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>>> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Max,
>>>>>>>>>>>> 
>>>>>>>>>>>> I think you can close the vote?
>>>>>>>>>>>> 
>>>>>>>>>>>> Bolke
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 27 Jun 2017, at 02:45, Kengo Seki <sekikn@apache.org <mailto:sekikn@apache.org>
>> <mailto:sekikn@apache.org <mailto:sekikn@apache.org>> <mailto:
>>>> sekikn@apache.org <mailto:sekikn@apache.org> <mailto:sekikn@apache.org <mailto:sekikn@apache.org>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> +1 (non-binding)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> - verified signatures and checksums
>>>>>>>>>>>>> - ran scheduler and webserver, confirmed they worked fine
>>>>>>>>>>>>> - confirmed the latest fix on v1.8 branch (AIRFLOW-809) is
>>>> included
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Kengo Seki <sekikn@apache.org <mailto:sekikn@apache.org> <mailto:sekikn@apache.org <mailto:sekikn@apache.org>>
>> <mailto:sekikn@apache.org <mailto:sekikn@apache.org> <mailto:sekikn@apache.org <mailto:sekikn@apache.org>>>>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2017-06-27 8:53 GMT+09:00 Chris Riccomini <
>> criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>
>>>> <mailto:criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>>>:
>>>>>>>>>>>>>> +1 (binding)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Been running in our dev env, and everything looks good.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 3:00 PM, Alex Guziel <
>>>>>>>>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>> <mailto:
>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>>>
>>>>>>>>>> .
>>>>>>>>>>> invalid
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Yeah that makes sense. It pages by default at 500 so it
>>>> explains
>>>>>>>>> why
>>>>>>>>>>> we saw
>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:53 PM, Chris Riccomini <
>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>> <mailto:
>> criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> In 1.8.1, the "DAGs" page has "Show <blank> entries". In
>>>> 1.8.2,
>>>>>>>>> it
>>>>>>>>>>> has
>>>>>>>>>>>>>>>> "Show <25> entries". So it looks like prior to 1.8.2, the
>>>>>>>>>> pagination
>>>>>>>>>>> was
>>>>>>>>>>>>>>>> broken in the sense that it defaulted to the whole list. We
>>>> have
>>>>>>>>>> 479
>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>> in one env, and it shows them all. It looks like someone
>> fixed
>>>>>>>>> the
>>>>>>>>>>> entry
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> default to 25 now, which exposed the problem for our
>>>>>>>>> environments.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:47 PM, Alex Guziel <
>>>>>>>>>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>> <mailto:
>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>>>
>>>>>>>>>>> .
>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> We're running 1.8.0 + some extras, and none of us added
>>>>>>>>> pagination
>>>>>>>>>>>>>>>>> recently, and our homepage is paginated. Are you sure it's
>>>> not
>>>>>>>>> the
>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>> of dags crossing the threshold? Maybe it's some Flask
>> version
>>>>>>>>>> thing?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:45 PM, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>
>> <mailto:criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Yes, I did the 1.8.1 release.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:44 PM, Alex Guziel <
>>>>>>>>>>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>> <mailto:
>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>>>
>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> There's no pagination in 1.8.1? Are you sure?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:37 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>
>> <mailto:criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> It's not happening on 1.8.1 (since there's no pagination
>>>> in
>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> version),
>>>>>>>>>>>>>>>>>>>> so I'd count this as a regression. I wouldn't say it's
>>>>>>>>>> blocking,
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>> pretty ugly.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:34 PM, Alex Guziel <
>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>>
>> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com> <mailto:alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>>>
>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I'm not so sure this is a new issue. I think we've seen
>>>> it
>>>>>>>>> on
>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>>>>>> production for quite a while.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:31 PM, Chris Riccomini <
>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>
>> <mailto:criccomini@apache.org <mailto:criccomini@apache.org> <mailto:criccomini@apache.org <mailto:criccomini@apache.org>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I am seeing a strange UI behavior on 1.8.2.RC2. I've
>>>>>>>>> opened a
>>>>>>>>>>>>>>>>> JIRA
>>>>>>>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1348 <https://issues.apache.org/jira/browse/AIRFLOW-1348> <
>> https://issues.apache.org/jira/browse/AIRFLOW-1348 <https://issues.apache.org/jira/browse/AIRFLOW-1348>> <
>>>> https://issues.apache.org/jira/browse/AIRFLOW-1348 <https://issues.apache.org/jira/browse/AIRFLOW-1348> <
>> https://issues.apache.org/jira/browse/AIRFLOW-1348 <https://issues.apache.org/jira/browse/AIRFLOW-1348>>>
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Has anyone else seen this?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 3:27 AM, Sumit Maheshwari <
>>>>>>>>>>>>>>>>>>>>> sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com> <mailto:sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com>>
>> <mailto:sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com> <mailto:sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com>>>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> +1, binding.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 3:49 PM, Bolke de Bruin <
>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <mailto:bdbruin@gmail.com> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>> <mailto:
>> bdbruin@gmail.com <mailto:bdbruin@gmail.com> <mailto:bdbruin@gmail.com <mailto:bdbruin@gmail.com>>>
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> We have been running it for the last couple of days.
>>>> No
>>>>>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>>>>>>>>> more responsive.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> +1, binding
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On 25 Jun 2017, at 01:10, Maxime Beauchemin <
>>>>>>>>>>>>>>>>>>>>>>> maximebeauchemin@gmail.com <mailto:maximebeauchemin@gmail.com> <mailto:maximebeauchemin@
>> gmail.com <http://gmail.com/>> <mailto:maximebeauchemin@
>>>> gmail.com <http://gmail.com/> <http://gmail.com/ <http://gmail.com/>>>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 1.8.2 RC2 is baked and available at:
>>>>>>>>>>>>>>>>>>>>>>>>> https://dist.apache.org/repos/ <https://dist.apache.org/repos/> <
>> https://dist.apache.org/repos/ <https://dist.apache.org/repos/>> <
>>>> https://dist.apache.org/repos/ <https://dist.apache.org/repos/> <https://dist.apache.org/repos/ <https://dist.apache.org/repos/>>>
>>>>>>>>>>>>>>> dist/dev/incubator/airflow
>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>>>>>>>> keys
>>>>>>>>>>>>>>>>>>>>>>>>> are available
>>>>>>>>>>>>>>>>>>>>>>>>> at https://dist.apache.org/repos/ <https://dist.apache.org/repos/> <
>> https://dist.apache.org/repos/ <https://dist.apache.org/repos/>> <
>>>> https://dist.apache.org/repos/ <https://dist.apache.org/repos/>>
>>>>>>>>>>>>>>>>>>> dist/release/incubator/airflow.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Note that RC1 was the first RC (skipped RC0) and
>> was
>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>>>>>> announced
>>>>>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>>>>> it had issues coming out of the oven, so RC2 is the
>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>>>>>>> RC.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 1.8.2 RC2 is build on to of 1.8.1 with these listed
>>>>>>>>>>>>>>>>>> "cherries"
>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>> top.
>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>> added the JIRAs that were identified blockers and
>>>>>>>>>>>>>>>> targeted
>>>>>>>>>>>>>>>>>>>> 1.8.2. I
>>>>>>>>>>>>>>>>>>>>>>>>> attempted to bring in all of the JIRAs that
>> targeted
>>>>>>>>>>>>>>>> 1.8.2
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> bailed
>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>> the ones that were generating merge conflicts. I
>> also
>>>>>>>>>>>>>>>> added
>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> JIRAs that we've been running in production at
>>>> Airbnb.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Issues fixed:
>>>>>>>>>>>>>>>>>>>>>>>>> 9a53e66 [AIRFLOW-809][AIRFLOW-1] Use __eq__
>>>>>>>>>>>>>>>> ColumnOperator
>>>>>>>>>>>>>>>>>> When
>>>>>>>>>>>>>>>>>>>>>> Testing
>>>>>>>>>>>>>>>>>>>>>>>>> Booleans
>>>>>>>>>>>>>>>>>>>>>>>>> 333e0b3 [AIRFLOW-1296] Propagate SKIPPED to all
>>>>>>>>>>>>>>>> downstream
>>>>>>>>>>>>>>>>>>> tasks
>>>>>>>>>>>>>>>>>>>>>>>>> 93825d5 [AIRFLOW-XXX] Re-enable caching for hadoop
>>>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>>>>>>>>>>> 33a9dcb [AIRFLOW-XXX] Pin Hive and Hadoop to a
>>>> specific
>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>>>>>>> writable warehouse dir
>>>>>>>>>>>>>>>>>>>>>>>>> 7cff6cd [AIRFLOW-1308] Disable nanny usage for Dask
>>>>>>>>>>>>>>>>>>>>>>>>> 570b2ed [AIRFLOW-1294] Backfills can loose tasks to
>>>>>>>>>>>>>>>> execute
>>>>>>>>>>>>>>>>>>>>>>>>> 3f48d48 [AIRFLOW-1291] Update NOTICE and LICENSE
>>>> files
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> match
>>>>>>>>>>>>>>>>>>>> ASF
>>>>>>>>>>>>>>>>>>>>>>>>> requirements
>>>>>>>>>>>>>>>>>>>>>>>>> 69bd269 [AIRFLOW-1160] Update Spark parameters for
>>>>>>>>>>>>>>> Mesos
>>>>>>>>>>>>>>>>>>>>>>>>> 9692510 [AIRFLOW 1149][AIRFLOW-1149] Allow for
>> custom
>>>>>>>>>>>>>>>>> filters
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> Jinja2
>>>>>>>>>>>>>>>>>>>>>>>>> templates
>>>>>>>>>>>>>>>>>>>>>>>>> 6de5330 [AIRFLOW-1119] Fix unload query so headers
>>>> are
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>>>> row[]
>>>>>>>>>>>>>>>>>>>>>>>>> b4e9eb8 [AIRFLOW-1089] Add Spark application
>>>> arguments
>>>>>>>>>>>>>>>>>>>>>>>>> a4083f3 [AIRFLOW-1078] Fix latest_runs endpoint for
>>>> old
>>>>>>>>>>>>>>>>> flask
>>>>>>>>>>>>>>>>>>>>>> versions
>>>>>>>>>>>>>>>>>>>>>>>>> 7a02841 [AIRFLOW-1074] Don't count queued tasks for
>>>>>>>>>>>>>>>>>> concurrency
>>>>>>>>>>>>>>>>>>>>>> limits
>>>>>>>>>>>>>>>>>>>>>>>>> a2c18a5 [AIRFLOW-1064] Change default sort to
>> job_id
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> TaskInstanceModelView
>>>>>>>>>>>>>>>>>>>>>>>>> d1c64ab [AIRFLOW-1038] Specify celery serialization
>>>>>>>>>>>>>>>> options
>>>>>>>>>>>>>>>>>>>>>> explicitly
>>>>>>>>>>>>>>>>>>>>>>>>> b4ee88a [AIRFLOW-1036] Randomize exponential
>> backoff
>>>>>>>>>>>>>>>>>>>>>>>>> 9fca409 [AIRFLOW-993] Update date inference logic
>>>>>>>>>>>>>>>>>>>>>>>>> 272c2f5 [AIRFLOW-1167] Support microseconds in
>>>> FTPHook
>>>>>>>>>>>>>>>>>>>> modification
>>>>>>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>>>>> c7c0b72 [AIRFLOW-1179] Fix Pandas 0.2x breaking
>>>> Google
>>>>>>>>>>>>>>>>>> BigQuery
>>>>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>>>> acd0166 [AIRFLOW-1263] Dynamic height for charts
>>>>>>>>>>>>>>>>>>>>>>>>> 7f33f6e [AIRFLOW-1266] Increase width of gantt y
>> axis
>>>>>>>>>>>>>>>>>>>>>>>>> fc33c04 [AIRFLOW-1290] set docs author to 'Apache
>>>>>>>>>>>>>>>> Airflow'
>>>>>>>>>>>>>>>>>>>>>>>>> 2e9eee3 [AIRFLOW-1282] Fix known event column
>> sorting
>>>>>>>>>>>>>>>>>>>>>>>>> 2389a8a [AIRFLOW-1166] Speed up
>>>>>>>>>>>>>>>>>> _change_state_for_tis_without_
>>>>>>>>>>>>>>>>>>>>> dagrun
>>>>>>>>>>>>>>>>>>>>>>>>> bf966e6 [AIRFLOW-1192] Some enhancements to
>>>>>>>>>>>>>>>> qubole_operator
>>>>>>>>>>>>>>>>>>>>>>>>> 57d5bcd [AIRFLOW-1281] Sort variables by key field
>> by
>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>> 802fc15 [AIRFLOW-1244] Forbid creation of a pool
>> with
>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>> 1232b6a [AIRFLOW-1243] DAGs table has no default
>>>>>>>>>>>>>>> entries
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> show
>>>>>>>>>>>>>>>>>>>>>>>>> b0ba3c9 [AIRFLOW-1227] Remove empty column on the
>>>> Logs
>>>>>>>>>>>>>>>> view
>>>>>>>>>>>>>>>>>>>>>>>>> c406652 [AIRFLOW-1226] Remove empty column on the
>>>> Jobs
>>>>>>>>>>>>>>>> view
>>>>>>>>>>>>>>>>>>>>>>>>> 51a83cc [AIRFLOW-1199] Fix create modal
>>>>>>>>>>>>>>>>>>>>>>>>> cac7d4c [AIRFLOW-1200] Forbid creation of a
>> variable
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>> key
>>>>>>>>>>>>>>>>>>>>>>>>> 5f3ee52 [AIRFLOW-1186] Sort dag.get_task_instances
>> by
>>>>>>>>>>>>>>>>>>>>> execution_date
>>>>>>>>>>>>>>>>>>>>>>>>> f446c08 [AIRFLOW-1145] Fix closest_date_partition
>>>>>>>>>>>>>>>> function
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>>>>>>>> to True If we're looking for the closest date
>> before,
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> latest date in the list of date before.
>>>>>>>>>>>>>>>>>>>>>>>>> 93b8e96 [AIRFLOW-1180] Fix flask-wtf version for
>>>>>>>>>>>>>>>>>>>>> test_csrf_rejection
>>>>>>>>>>>>>>>>>>>>>>>>> bb56805 [AIRFLOW-1170] DbApiHook insert_rows
>> inserts
>>>>>>>>>>>>>>>>>> parameters
>>>>>>>>>>>>>>>>>>>>>>>> separately
>>>>>>>>>>>>>>>>>>>>>>>>> 093b2f0 [AIRFLOW-1150] Fix scripts execution in
>>>>>>>>>>>>>>> sparksql
>>>>>>>>>>>>>>>>>> hook[]
>>>>>>>>>>>>>>>>>>>>>>>>> 777f181 [AIRFLOW-1168] Add closing() to all
>>>> connections
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> cursors
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Max


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message