airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.9.0 status
Date Mon, 30 Oct 2017 17:59:58 GMT
If anyone wants review those, they can be merged today. 

Bolke

Sent from my iPhone

> On 30 Oct 2017, at 18:58, Chris Riccomini <criccomini@apache.org> wrote:
> 
> Hey all,
> 
> Current blockers for 1.9.0:
> 
> https://issues.apache.org/jira/browse/AIRFLOW-1711
> https://issues.apache.org/jira/browse/AIRFLOW-1018
> 
> Cheers,
> Chris
> 
>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <criccomini@apache.org> wrote:
>> Hey all,
>> 
>> Talked with @Bolke on Gitter. Here are the currently blocking issues for 1.9.0:
>> 
>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries can be False
>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom config on PYTHONPATH
>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets stuck in queued state
>> 
>> PRs are out for them. After these three are merged, I'm planning to cut an RC.
>> 
>> Cheers,
>> Chris
>> 
>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>>> I upgraded our production environment today. Some observations
>>> 
>>> 1. Database migration (add max tries) didn’t workout of the box correctly for Postgres. “max_tries = False” didn’t work and needed to be casted to an int. Not sure how this gets to be false. This is a block imho
>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We needed to rework quite a lot of dags
>>> 3. We use LdAP for logins. We havent configured the ‘superuser’ ‘data_profiler’ groups and thus should have automatic superuser privileges. We don’t at the moment (eg. we cannot manage connections or see the admin screens).
>>> 
>>> After fixing #1, the rest seems quite normal. Happy with the new logging.
>>> 
>>> Bolke
>>> 
>>> 
>>> > On 20 Oct 2017, at 12:16, Driesprong, Fokko <fokko@driesprong.frl> wrote:
>>> >
>>> > Hi Bolke,
>>> >
>>> > This one is a blocker: https://issues.apache.org/jira/browse/AIRFLOW-1731.
>>> >
>>> > This requirers that we are sure that the /plugin/ folder is on the path:
>>> > https://github.com/apache/incubator-airflow/blob/eb2f589099b87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
>>> >
>>> > And maybe add an additional test and updating the docs a bit. Hopefully
>>> > somewhere in the next few days, however I'm a bit busy with the Spark
>>> > Summit.
>>> >
>>> > Cheers, Fokko
>>> >
>>> > 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bdbruin@gmail.com>:
>>> >
>>> >> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>>> >>
>>> >> What are the issues that are open?
>>> >>
>>> >> Cheers
>>> >> Bolke
>>> >>
>>> >>> On 9 Oct 2017, at 20:24, Chris Riccomini <criccomini@apache.org> wrote:
>>> >>>
>>> >>> K, I will plan on cutting an alpha1 later this week. Something for you
>>> >> guys to play with.
>>> >>>
>>> >>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
>>> >> <mailto:bdbruin@gmail.com>> wrote:
>>> >>> hi Chris,
>>> >>>
>>> >>> We are still running pre alpha. I am a bit preoccupied with preparing
>>> >> for a conference and the team is readying a release of one of our core
>>> >> products. So it will probably will be after this week when I get my hands
>>> >> dirty again.
>>> >>>
>>> >>> Cheers
>>> >>> Bolke
>>> >>>
>>> >>> Verstuurd vanaf mijn iPad
>>> >>>
>>> >>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
>>> >> <mailto:criccomini@apache.org>> het volgende geschreven:
>>> >>>>
>>> >>>> Hey all,
>>> >>>>
>>> >>>> Checking in. Has anyone tested this in their environments? Stable
>>> >> releases
>>> >>>> depend on community involvement.
>>> >>>>
>>> >>>> Cheers,
>>> >>>> Chris
>>> >>>>
>>> >>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>>> >> criccomini@apache.org <mailto:criccomini@apache.org>>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
>>> >> but
>>> >>>>> will be included in alpha1.
>>> >>>>>
>>> >>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fenglu@google.com.invalid>
>>> >> wrote:
>>> >>>>>>
>>> >>>>>> Hi Chris,
>>> >>>>>>
>>> >>>>>> I know it's annoying to have last minute commit com in, but this is a
>>> >>>>>> highly desirable feature for folks using GCP operators, is it
>>> >> possible to
>>> >>>>>> include AIRFLOW-1635
>>> >>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
>>> >> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>> >>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>> >>>>>> in?
>>> >>>>>> More than happy to offer help if there's something I can do.
>>> >>>>>> Thanks a lot.
>>> >>>>>>
>>> >>>>>> Feng
>>> >>>>>>
>>> >>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>>> >> criccomini@apache.org <mailto:criccomini@apache.org>>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Hey all,
>>> >>>>>>>
>>> >>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it
>>> >> here:
>>> >>>>>>>
>>> >>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>> >> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>>> >> airflow/1.9.0alpha0/>
>>> >>>>>>>
>>> >>>>>>> The bin tarball can be installed with pip:
>>> >>>>>>>
>>> >>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>>> >>>>>>>
>>> >>>>>>> The goal is to have the community install and run this to expose any
>>> >>>>>> bugs
>>> >>>>>>> before we move on to official release candidates.
>>> >>>>>>>
>>> >>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>> >>>>>>>
>>> >>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>> >>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>> >>>>>> marked
>>> >>>>>>> as
>>> >>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>> >> exception
>>> >>>>>>> for
>>> >>>>>>> @on
>>> >>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
>>> >>>>>> stdout
>>> >>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
>>> >>>>>> for
>>> >>>>>>> @once
>>> >>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>> >> fail
>>> >>>>>>>
>>> >>>>>>> Cheers,
>>> >>>>>>> Chris
>>> >>>>>>>
>>> >>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>>> >> criccomini@apache.org <mailto:criccomini@apache.org>
>>> >>>>>>>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
>>> >>>>>>>>
>>> >>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>> >>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>> >>>>>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>> >>>>>>>>>
>>> >>>>>>>>> Blockers for 1.9.0 are:
>>> >>>>>>>>>
>>> >>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>> >>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>> >>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>>> >>>>>>> marked
>>> >>>>>>>>> as
>>> >>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it to
>>> >>>>>> fail
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>>> >> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>> >>>>>>>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> Hi Chris
>>> >>>>>>>>>>
>>> >>>>>>>>>> Can I suggest releasing a beta? The stable branch is only cut at
>>> >> RC
>>> >>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
>>> >> point of
>>> >>>>>>>>>> reference.
>>> >>>>>>>>>>
>>> >>>>>>>>>> In addition the list below are mostly longer standing issues that
>>> >>>>>> are
>>> >>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525,
>>> >> 1258,
>>> >>>>>>> and
>>> >>>>>>>>>> 976 as blocker?
>>> >>>>>>>>>>
>>> >>>>>>>>>> Cheers
>>> >>>>>>>>>> Bolke
>>> >>>>>>>>>>
>>> >>>>>>>>>> Verstuurd vanaf mijn iPad
>>> >>>>>>>>>>
>>> >>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
>>> >>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>> >>>>>>>>>> het volgende geschreven:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Hey all,
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
>>> >>>>>> release,
>>> >>>>>>> but
>>> >>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
>>> >> delay.
>>> >>>>>>> Here
>>> >>>>>>>>>> are
>>> >>>>>>>>>>> the bugs that I'm tracking:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>> >>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>> >>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
>>> >> are
>>> >>>>>>>>>> marked as
>>> >>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>> >>>>>>> exception
>>> >>>>>>>>>> for
>>> >>>>>>>>>>> @on
>>> >>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
>>> >> to
>>> >>>>>>>>>> stdout
>>> >>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>> >>>>>> exception
>>> >>>>>>>>>> for @once
>>> >>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if
>>> >>>>>> Email
>>> >>>>>>> is
>>> >>>>>>>>>> Not
>>> >>>>>>>>>>> be
>>> >>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
>>> >> to
>>> >>>>>>> fail
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> These are the priority issues. Once they're merged, I'll cut the
>>> >>>>>>>>>>> v1-9-stable and beta release.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> If you can help clean this up, that would be really appreciated.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Cheers,
>>> >>>>>>>>>>> Chris
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>>> >>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> Marked it for 1.9.0.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
>>> >>>>>> cjones@simpli.fi <mailto:cjones@simpli.fi>>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
>>> >>>>>>>>>> callbacks
>>> >>>>>>>>>>>>> are not working correctly without emails... Its not a major
>>> >> bug,
>>> >>>>>>> but
>>> >>>>>>>>>> it
>>> >>>>>>>>>>>>> does cause us some annoyance in our current deployment.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Link to Jira:
>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Link to PR:
>>> >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
>>> >> https://github.com/apache/incubator-airflow/pull/2415>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Thanks!
>>> >>>>>>>>>>>>> Charlie Jones
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> CHARLIE JONES
>>> >>>>>>>>>>>>> Data Engineer
>>> >>>>>>>>>>>>> cjones@simpli.fi <mailto:cjones@simpli.fi>  |  M:
>>> >> 972.821.7631 <tel:972.821.7631>
>>> >>>>>>>>>>>>> __________________________________________________
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>> >>>>>>>>>>>>> __________________________________________________
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>> >>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
>>> >> http://www.simpli.fi/>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>>> >>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>> >>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Merged.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>> >>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore.com>>
>>> >>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
>>> >>>>>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0
>>> >>>>>> branch
>>> >>>>>>>>>> due
>>> >>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>> this issue.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 <
>>> >> https://github.com/apache/incubator-airflow/pull/2590>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Thanks,
>>> >>>>>>>>>>>>>>> Ryan
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>> >>>>>>>>>>>>> <fokko@driesprong.frl
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Hi All,
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>>> >>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 <
>>> >> https://github.com/apache/incubator-airflow/pull/2631>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Currently importing a custom logging configuration is not
>>> >>>>>> work
>>> >>>>>>> (as
>>> >>>>>>>>>>>>> far
>>> >>>>>>>>>>>>>>> as I
>>> >>>>>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Cheers, Fokko
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
>>> >>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
>>> >>>>>>>>>>> :
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> Done!
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>> >>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
>>> >> michael.crawford@modernizingmedicine.com>> wrote:
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in?
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>>> >> https://github.com/apache/incubator-airflow/pull/2626> <
>>> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>>> >> https://github.com/apache/incubator-airflow/pull/2626>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on
>>> >>>>>> edit
>>> >>>>>>>>>>>>> for
>>> >>>>>>>>>>>>>>> these
>>> >>>>>>>>>>>>>>>>>> types.
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> Thanks,
>>> >>>>>>>>>>>>>>>>>> Mike
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>> >>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>> >>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
>>> >> forward
>>> >>>>>> to
>>> >>>>>>>>>>>>>> these
>>> >>>>>>>>>>>>>>>>> fixes!
>>> >>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>> >>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>
>>> >>>>>>>>>>>>>>>> .
>>> >>>>>>>>>>>>>>>>>> invalid
>>> >>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> Can we get this in?
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>> >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 <
>>> >> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
>>> >> https://github.com/apache/incubator-airflow/commit/>
>>> >>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ <
>>> >> https://github.com/apache/incubator-airflow/commit/>
>>> >>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing
>>> >> to
>>> >>>>>>>>>>>>> have
>>> >>>>>>>>>>>>>> in
>>> >>>>>>>>>>>>>>>> 1.9.
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini <
>>> >>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>> >>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then.
>>> >>>>>>>>>>>>> Initial
>>> >>>>>>>>>>>>>>>>> warning
>>> >>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get
>>> >> into
>>> >>>>>>>>>>>>> 1.9.0
>>> >>>>>>>>>>>>>> at
>>> >>>>>>>>>>>>>>>>> that
>>> >>>>>>>>>>>>>>>>>>>>> point.
>>> >>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>> >>>>>>>>>>>>>>>> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>> >>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
>>> >> particular
>>> >>>>>>>>>>>>> fixed
>>> >>>>>>>>>>>>>>>> point
>>> >>>>>>>>>>>>>>>>> in
>>> >>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball
>>> >> than
>>> >>>>>> to
>>> >>>>>>> a
>>> >>>>>>>>>>>>>> git
>>> >>>>>>>>>>>>>>>>> pull.
>>> >>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>> Bolke
>>> >>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
>>> >>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
>>> >>>>>> different?
>>> >>>>>>>>>>>>>> IIRC,
>>> >>>>>>>>>>>>>>>> it's
>>> >>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
>>> >>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>> >>>>>>>>>>>>>>>>> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>> >>>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right
>>> >>>>>> away?
>>> >>>>>>>>>>>>>>> Isn’t a
>>> >>>>>>>>>>>>>>>>>>>> beta
>>> >>>>>>>>>>>>>>>>>>>>> a
>>> >>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
>>> >>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>> - Bolke
>>> >>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
>>> >>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to
>>> >>>>>> cut
>>> >>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>> stable
>>> >>>>>>>>>>>>>>>>>>>>> branch
>>> >>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once
>>> >> the
>>> >>>>>>>>>>>>> stable
>>> >>>>>>>>>>>>>>>> branch
>>> >>>>>>>>>>>>>>>>>>>> is
>>> >>>>>>>>>>>>>>>>>>>>>>>> cut, I
>>> >>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
>>> >> picked
>>> >>>>>>> into
>>> >>>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>>>>>> branch,
>>> >>>>>>>>>>>>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get
>>> >> the
>>> >>>>>>>>>>>>> release
>>> >>>>>>>>>>>>>>>> out.
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>> >>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini
>>> >> <
>>> >>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the
>>> >>>>>>>>>>>>> outstanding
>>> >>>>>>>>>>>>>> PRs
>>> >>>>>>>>>>>>>>>>> that
>>> >>>>>>>>>>>>>>>>>>>>> are
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS Vulnerability in
>>> >>>>>> Variable
>>> >>>>>>>>>>>>>>>> endpoint
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize logging in
>>> >>>>>> Airflow
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of local
>>> >>>>>>>>>>>>> loggers
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the logger to log
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor LICENSE &
>>> >> NOTICE
>>> >>>>>>>>>>>>> issue
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and
>>> >>>>>>>>>>>>> unneeded
>>> >>>>>>>>>>>>>>> code
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to operate
>>> >>>>>> HDFS
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
>>> >>>>>> |airflow/jobs.py:create_dag_ru
>>> >>>>>>>>>>>>> n()
>>> >>>>>>>>>>>>>>>>>>>> exception
>>> >>>>>>>>>>>>>>>>>>>>>> for
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> @on
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns shouldn't
>>> >>>>>>>>>>>>> include
>>> >>>>>>>>>>>>>>>> paused
>>> >>>>>>>>>>>>>>>>>>>>> DAGs
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG processes
>>> >>>>>> can
>>> >>>>>>>>>>>>> not
>>> >>>>>>>>>>>>>> log
>>> >>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>>>>>>> stdout
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView displayed over
>>> >>>>>> task
>>> >>>>>>>>>>>>>>>> instances
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
>>> >>>>>> |airflow/jobs.py:manage_slas()
>>> >>>>>>>>>>>>>>>> exception
>>> >>>>>>>>>>>>>>>>>>>> for
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> @once
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success running
>>> >> task
>>> >>>>>>>>>>>>> causes
>>> >>>>>>>>>>>>>> it
>>> >>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>>>>>> fail
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>> >>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>> >>>>>>>>>>>>>>>>>>>>>>>> examples
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>> >>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>> >>>>>>>>>>>>>>>>>>>> job
>>> >>>>>>>>>>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> real
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests and build
>>> >>>>>>>>>>>>> matrix
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should not
>>> >> push
>>> >>>>>>>>>>>>> XComs
>>> >>>>>>>>>>>>>> by
>>> >>>>>>>>>>>>>>>>>>>> default
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size for
>>> >> XComs
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow semantics
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context unexpectedly
>>> >>>>>> added to
>>> >>>>>>>>>>>>>> hive
>>> >>>>>>>>>>>>>>>> conf
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked
>>> >>>>>> into
>>> >>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>> 1.9.0
>>> >>>>>>>>>>>>>>>>>>>>>> branch
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in,
>>> >>>>>> please
>>> >>>>>>>>>>>>> set
>>> >>>>>>>>>>>>>>> the
>>> >>>>>>>>>>>>>>>>> fix
>>> >>>>>>>>>>>>>>>>>>>>>>>> version
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev
>>> >>>>>>> cluster,
>>> >>>>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>> it
>>> >>>>>>>>>>>>>>>>> has
>>> >>>>>>>>>>>>>>>>>>>>>> been
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability.
>>> >> If
>>> >>>>>> you
>>> >>>>>>>>>>>>> run
>>> >>>>>>>>>>>>>>>>>>>> Airflow,
>>> >>>>>>>>>>>>>>>>>>>>>> it's
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test
>>> >>>>>> branch
>>> >>>>>>>>>>>>>>>> somewhere,
>>> >>>>>>>>>>>>>>>>>>>> and
>>> >>>>>>>>>>>>>>>>>>>>>>>> verify
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Chris
>>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>
>>> >>
>>> >>
>>> 
>> 
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message