airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.9.0 status
Date Fri, 20 Oct 2017 14:49:10 GMT
I upgraded our production environment today. Some observations

1. Database migration (add max tries) didn’t workout of the box correctly for Postgres.
“max_tries = False” didn’t work and needed to be casted to an int. Not sure how this
gets to be false. This is a block imho
2. SSHExecuteOperator not having backwards compatibility sucks. We needed to rework quite
a lot of dags
3. We use LdAP for logins. We havent configured the ‘superuser’ ‘data_profiler’ groups
and thus should have automatic superuser privileges. We don’t at the moment (eg. we cannot
manage connections or see the admin screens).

After fixing #1, the rest seems quite normal. Happy with the new logging.

Bolke


> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fokko@driesprong.frl> wrote:
> 
> Hi Bolke,
> 
> This one is a blocker: https://issues.apache.org/jira/browse/AIRFLOW-1731.
> 
> This requirers that we are sure that the /plugin/ folder is on the path:
> https://github.com/apache/incubator-airflow/blob/eb2f589099b87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> 
> And maybe add an additional test and updating the docs a bit. Hopefully
> somewhere in the next few days, however I'm a bit busy with the Spark
> Summit.
> 
> Cheers, Fokko
> 
> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bdbruin@gmail.com>:
> 
>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
>> 
>> What are the issues that are open?
>> 
>> Cheers
>> Bolke
>> 
>>> On 9 Oct 2017, at 20:24, Chris Riccomini <criccomini@apache.org> wrote:
>>> 
>>> K, I will plan on cutting an alpha1 later this week. Something for you
>> guys to play with.
>>> 
>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <bdbruin@gmail.com
>> <mailto:bdbruin@gmail.com>> wrote:
>>> hi Chris,
>>> 
>>> We are still running pre alpha. I am a bit preoccupied with preparing
>> for a conference and the team is readying a release of one of our core
>> products. So it will probably will be after this week when I get my hands
>> dirty again.
>>> 
>>> Cheers
>>> Bolke
>>> 
>>> Verstuurd vanaf mijn iPad
>>> 
>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <criccomini@apache.org
>> <mailto:criccomini@apache.org>> het volgende geschreven:
>>>> 
>>>> Hey all,
>>>> 
>>>> Checking in. Has anyone tested this in their environments? Stable
>> releases
>>>> depend on community involvement.
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>>> wrote:
>>>> 
>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0,
>> but
>>>>> will be included in alpha1.
>>>>> 
>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <fenglu@google.com.invalid>
>> wrote:
>>>>>> 
>>>>>> Hi Chris,
>>>>>> 
>>>>>> I know it's annoying to have last minute commit com in, but this
is a
>>>>>> highly desirable feature for folks using GCP operators, is it
>> possible to
>>>>>> include AIRFLOW-1635
>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
>>>>>> in?
>>>>>> More than happy to offer help if there's something I can do.
>>>>>> Thanks a lot.
>>>>>> 
>>>>>> Feng
>>>>>> 
>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey all,
>>>>>>> 
>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download
it
>> here:
>>>>>>> 
>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
>> airflow/1.9.0alpha0/>
>>>>>>> 
>>>>>>> The bin tarball can be installed with pip:
>>>>>>> 
>>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>>>>>>> 
>>>>>>> The goal is to have the community install and run this to expose
any
>>>>>> bugs
>>>>>>> before we move on to official release candidates.
>>>>>>> 
>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
>>>>>>> 
>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
are
>>>>>> marked
>>>>>>> as
>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>> exception
>>>>>>> for
>>>>>>> @on
>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
to
>>>>>> stdout
>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
>>>>>> for
>>>>>>> @once
>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
to
>> fail
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>> 
>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
>> criccomini@apache.org <mailto:criccomini@apache.org>
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday.
:)
>>>>>>>> 
>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Works for me. Will try and cut a beta before end of week.
>>>>>>>>> 
>>>>>>>>> Blockers for 1.9.0 are:
>>>>>>>>> 
>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
issue
>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
are
>>>>>>> marked
>>>>>>>>> as
>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
causes it to
>>>>>> fail
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
>> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Chris
>>>>>>>>>> 
>>>>>>>>>> Can I suggest releasing a beta? The stable branch
is only cut at
>> RC
>>>>>>>>>> time. Betas allow us a broader exposure. It also
gives us a
>> point of
>>>>>>>>>> reference.
>>>>>>>>>> 
>>>>>>>>>> In addition the list below are mostly longer standing
issues that
>>>>>> are
>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider
1611, 1525,
>> 1258,
>>>>>>> and
>>>>>>>>>> 976 as blocker?
>>>>>>>>>> 
>>>>>>>>>> Cheers
>>>>>>>>>> Bolke
>>>>>>>>>> 
>>>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>>>> 
>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini
<
>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>>>>>>>>> het volgende geschreven:
>>>>>>>>>>> 
>>>>>>>>>>> Hey all,
>>>>>>>>>>> 
>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and
1.9.0 beta
>>>>>> release,
>>>>>>> but
>>>>>>>>>>> seeing as there are several outstanding bugs,
I'm going to
>> delay.
>>>>>>> Here
>>>>>>>>>> are
>>>>>>>>>>> the bugs that I'm tracking:
>>>>>>>>>>> 
>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging
in Airflow
>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE
& NOTICE issue
>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
SubDagOperator
>> are
>>>>>>>>>> marked as
>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
>>>>>>> exception
>>>>>>>>>> for
>>>>>>>>>>> @on
>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes
can not log
>> to
>>>>>>>>>> stdout
>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
>>>>>> exception
>>>>>>>>>> for @once
>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks
Are Repeated if
>>>>>> Email
>>>>>>> is
>>>>>>>>>> Not
>>>>>>>>>>> be
>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running
task causes it
>> to
>>>>>>> fail
>>>>>>>>>>> 
>>>>>>>>>>> These are the priority issues. Once they're merged,
I'll cut the
>>>>>>>>>>> v1-9-stable and beta release.
>>>>>>>>>>> 
>>>>>>>>>>> If you can help clean this up, that would be
really appreciated.
>>>>>>>>>>> 
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini
<
>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Marked it for 1.9.0.
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie
Jones <
>>>>>> cjones@simpli.fi <mailto:cjones@simpli.fi>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Is there any chance we could include
AIRFLOW-988 in 1.9.0? SLA
>>>>>>>>>> callbacks
>>>>>>>>>>>>> are not working correctly without emails...
Its not a major
>> bug,
>>>>>>> but
>>>>>>>>>> it
>>>>>>>>>>>>> does cause us some annoyance in our current
deployment.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Link to Jira:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988
<
>> https://issues.apache.org/jira/browse/AIRFLOW-988>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Link to PR:
>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415
<
>> https://github.com/apache/incubator-airflow/pull/2415>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> Charlie Jones
>>>>>>>>>>>>> 
>>>>>>>>>>>>> CHARLIE JONES
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> cjones@simpli.fi <mailto:cjones@simpli.fi>
 |  M:
>> 972.821.7631 <tel:972.821.7631>
>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Programmatic Performance.* Localized.*
>>>>>>>>>>>>> __________________________________________________
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort
Worth, TX 76102
>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>
 |  www.simpli.fi <
>> http://www.simpli.fi/>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris
Riccomini <
>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Merged.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM,
Ryan Buckley <
>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore.com>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Would it be possible to include
AIRFLOW-1587?
>>>>>>>>>>>>>>> Running dags from the UI is currently
broken on the 1.9.0
>>>>>> branch
>>>>>>>>>> due
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590
<
>> https://github.com/apache/incubator-airflow/pull/2590>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14
AM, Driesprong, Fokko
>>>>>>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611
in the 1.9.0 release:
>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631
<
>> https://github.com/apache/incubator-airflow/pull/2631>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Currently importing a custom
logging configuration is not
>>>>>> work
>>>>>>> (as
>>>>>>>>>>>>> far
>>>>>>>>>>>>>>> as I
>>>>>>>>>>>>>>>> know). Any feedback on the
PR would also be appreciated.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00
Chris Riccomini <
>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
>>>>>>>>>>> :
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Done!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017
at 5:11 AM, Michael Crawford <
>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com
<mailto:
>> michael.crawford@modernizingmedicine.com>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Can you slide the
aws and emr connection type fix in?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636
<
>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636
<
>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
<
>> https://github.com/apache/incubator-airflow/pull/2626> <
>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
<
>> https://github.com/apache/incubator-airflow/pull/2626>>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> It keeps the connection
type from getting blanked out on
>>>>>> edit
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>> types.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Sep 21, 2017,
at 1:27 PM, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Absolutely. Just
cherry-picked. I've been looking
>> forward
>>>>>> to
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>> fixes!
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, Sep 20,
2017 at 5:23 PM, Alex Guziel <
>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com>
>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Can we get
this in?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
<
>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
<
>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
<
>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
<
>> https://github.com/apache/incubator-airflow/commit/>
>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> It speeds
up loading times a lot, so it's a good thing
>> to
>>>>>>>>>>>>> have
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> 1.9.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Wed, Sep
20, 2017 at 11:14 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>> criccomini@apache.org
<mailto:criccomini@apache.org>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Sounds
good. I'll plan on stable+beta next week, then.
>>>>>>>>>>>>> Initial
>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>> stands,
that I will start locking down what can get
>> into
>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Wed,
Sep 20, 2017 at 11:10 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> No
vote indeed, just to gather feedback on a
>> particular
>>>>>>>>>>>>> fixed
>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> time.
It also gives a bit more trust to a tarball
>> than
>>>>>> to
>>>>>>> a
>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>> pull.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>
On 20 Sep 2017, at 20:09, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>
I can do a beta. Is the process significantly
>>>>>> different?
>>>>>>>>>>>>>> IIRC,
>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>
basically the same, just no vote, right?
>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>
On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin <
>>>>>>>>>>>>>>>>> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>
Are you sure you want to go ahead and do RCs right
>>>>>> away?
>>>>>>>>>>>>>>> Isn’t a
>>>>>>>>>>>>>>>>>>>> beta
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>
bit smarter?
>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>
- Bolke
>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>
On 20 Sep 2017, at 19:41, Chris Riccomini <
>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>
Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>
I want to send out a warning that I'm planning to
>>>>>> cut
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>
next week, and begin the RC1 release vote. Once
>> the
>>>>>>>>>>>>> stable
>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>
cut, I
>>>>>>>>>>>>>>>>>>>>>>>>>
will be locking down what commits get cherry
>> picked
>>>>>>> into
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> branch,
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>
will only be doing PRs that are required to get
>> the
>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>> out.
>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>
Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>
Chris
>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>
On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini
>> <
>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org
<mailto:criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>
wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>
Hey all,
>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>
An update on the 1.9.0 release. Here are the
>>>>>>>>>>>>> outstanding
>>>>>>>>>>>>>> PRs
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>
slated to be included into 1.9.0:
>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>
ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1617 |Open      |XSS Vulnerability in
>>>>>> Variable
>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1611 |Open      |Customize logging in
>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1605 |Reopened  |Fix log source of local
>>>>>>>>>>>>> loggers
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1604 |Open      |Rename the logger to log
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1525 |Open      |Fix minor LICENSE &
>> NOTICE
>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1499 |In Progres|Eliminate duplicate and
>>>>>>>>>>>>> unneeded
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1198 |Open      |HDFSOperator to operate
>>>>>> HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1055 |Open
>>>>>> |airflow/jobs.py:create_dag_ru
>>>>>>>>>>>>> n()
>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>
@on
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1019 |Open      |active_dagruns shouldn't
>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>> paused
>>>>>>>>>>>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1018 |Open      |Scheduler DAG processes
>>>>>> can
>>>>>>>>>>>>> not
>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1015 |Open      |TreeView displayed over
>>>>>> task
>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-1013 |Open
>>>>>> |airflow/jobs.py:manage_slas()
>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>
@once
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-976  |Open      |Mark success running
>> task
>>>>>>>>>>>>> causes
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> fail
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-914  |Open      |Refactor
>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>>>>>>>>>>>>
examples
>>>>>>>>>>>>>>>>>>>>>>>>>>
to
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-913  |Open      |Refactor
>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>>>>>>>>>>>> job
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>
real
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-912  |Open      |Refactor tests and build
>>>>>>>>>>>>> matrix
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-888  |Open      |Operators should not
>> push
>>>>>>>>>>>>> XComs
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-828  |Open      |Add maximum size for
>> XComs
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-825  |Open      |Add Dataflow semantics
>>>>>>>>>>>>>>>>>>>>>>>>>>
AIRFLOW-788  |Open      |Context unexpectedly
>>>>>> added to
>>>>>>>>>>>>>> hive
>>>>>>>>>>>>>>>> conf
>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>
I will be locking down what can get cherry-picked
>>>>>> into
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>>>>>>>>>>>>
shortly, so if you have something you want in,
>>>>>> please
>>>>>>>>>>>>> set
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>>>>>>>>>
version
>>>>>>>>>>>>>>>>>>>>>>>>>>
to 1.9.0.
>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>
We (at WePay) have deployed 1.9.0 into our dev
>>>>>>> cluster,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>
running smoothly for several days.
>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>
** I could really use help verifying stability.
>> If
>>>>>> you
>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>>
in your best interest to deploy the 1.9.0 test
>>>>>> branch
>>>>>>>>>>>>>>>> somewhere,
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>
verify
>>>>>>>>>>>>>>>>>>>>>>>>>>
it's working for your workload. **
>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>
Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>
Chris
>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 
>> 


Mime
View raw message