airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Airflow 1.9.0 status
Date Thu, 28 Sep 2017 20:32:54 GMT
Works for me. Will try and cut a beta before end of week.

Blockers for 1.9.0 are:

AIRFLOW-1611 |Bug         |Customize logging in Airflow
AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked as
AIRFLOW-976  |Bug         |Mark success running task causes it to fail


On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> Hi Chris
>
> Can I suggest releasing a beta? The stable branch is only cut at RC time.
> Betas allow us a broader exposure. It also gives us a point of reference.
>
> In addition the list below are mostly longer standing issues that are also
> part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, and 976 as
> blocker?
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <criccomini@apache.org>
> het volgende geschreven:
> >
> > Hey all,
> >
> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
> > seeing as there are several outstanding bugs, I'm going to delay. Here
> are
> > the bugs that I'm tracking:
> >
> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> marked as
> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
> for
> > @on
> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for
> @once
> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is
> Not
> > be
> > AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> >
> > These are the priority issues. Once they're merged, I'll cut the
> > v1-9-stable and beta release.
> >
> > If you can help clean this up, that would be really appreciated.
> >
> > Cheers,
> > Chris
> >
> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <criccomini@apache.org
> >
> > wrote:
> >
> >> Marked it for 1.9.0.
> >>
> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cjones@simpli.fi>
> wrote:
> >>>
> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
> callbacks
> >>> are not working correctly without emails... Its not a major bug, but it
> >>> does cause us some annoyance in our current deployment.
> >>>
> >>> Link to Jira:
> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
> >>>
> >>> Link to PR:
> >>> https://github.com/apache/incubator-airflow/pull/2415
> >>>
> >>> Thanks!
> >>> Charlie Jones
> >>>
> >>> CHARLIE JONES
> >>> Data Engineer
> >>> cjones@simpli.fi  |  M: 972.821.7631
> >>> __________________________________________________
> >>>
> >>>
> >>> Programmatic Performance.* Localized.*
> >>> __________________________________________________
> >>>
> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>> 800.840.0768  |  www.simpli.fi
> >>>
> >>>
> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> criccomini@apache.org>
> >>> wrote:
> >>>
> >>>> Merged.
> >>>>
> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>> ryan.buckley@bluecore.com>
> >>>> wrote:
> >>>>
> >>>>> Would it be possible to include AIRFLOW-1587?
> >>>>> Running dags from the UI is currently broken on the 1.9.0 branch
due
> >>> to
> >>>>> this issue.
> >>>>>
> >>>>> https://github.com/apache/incubator-airflow/pull/2590
> >>>>>
> >>>>> Thanks,
> >>>>> Ryan
> >>>>>
> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>> <fokko@driesprong.frl
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
> >>>>>>
> >>>>>> Currently importing a custom logging configuration is not work
(as
> >>> far
> >>>>> as I
> >>>>>> know). Any feedback on the PR would also be appreciated.
> >>>>>>
> >>>>>> Cheers, Fokko
> >>>>>>
> >>>>>>
> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <criccomini@apache.org>:
> >>>>>>
> >>>>>>> Done!
> >>>>>>>
> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
> >>>>>>>
> >>>>>>>> Can you slide the aws and emr connection type fix in?
> >>>>>>>>
> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
<
> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
> >>>>>>>>
> >>>>>>>> It keeps the connection type from getting blanked out
on edit
> >>> for
> >>>>> these
> >>>>>>>> types.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Mike
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>> criccomini@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking
forward to
> >>>> these
> >>>>>>> fixes!
> >>>>>>>>>
> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>> alex.guziel@airbnb.com
> >>>>>> .
> >>>>>>>> invalid
> >>>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Can we get this in?
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
> >>>>>>>>>>
> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>
> >>>>>>>>>> It speeds up loading times a lot, so it's a
good thing to
> >>> have
> >>>> in
> >>>>>> 1.9.
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini
<
> >>>>>>>> criccomini@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next
week, then.
> >>> Initial
> >>>>>>> warning
> >>>>>>>>>>> stands, that I will start locking down what
can get into
> >>> 1.9.0
> >>>> at
> >>>>>>> that
> >>>>>>>>>>> point.
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke
de Bruin <
> >>>>>> bdbruin@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> No vote indeed, just to gather feedback
on a particular
> >>> fixed
> >>>>>> point
> >>>>>>> in
> >>>>>>>>>>>> time. It also gives a bit more trust
to a tarball than to a
> >>>> git
> >>>>>>> pull.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Bolke
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris
Riccomini <
> >>>>> criccomini@apache.org
> >>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I can do a beta. Is the process
significantly different?
> >>>> IIRC,
> >>>>>> it's
> >>>>>>>>>>>>> basically the same, just no vote,
right?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM,
Bolke de Bruin <
> >>>>>>> bdbruin@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Are you sure you want to go
ahead and do RCs right away?
> >>>>> Isn’t a
> >>>>>>>>>> beta
> >>>>>>>>>>> a
> >>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41,
Chris Riccomini <
> >>>>>> criccomini@apache.org
> >>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I want to send out a warning
that I'm planning to cut
> >>> the
> >>>>>> stable
> >>>>>>>>>>> branch
> >>>>>>>>>>>>>>> next week, and begin the
RC1 release vote. Once the
> >>> stable
> >>>>>> branch
> >>>>>>>>>> is
> >>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>> will be locking down what
commits get cherry picked into
> >>>> the
> >>>>>>>>>> branch,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> will only be doing PRs that
are required to get the
> >>> release
> >>>>>> out.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at
11:19 AM, Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> An update on the 1.9.0
release. Here are the
> >>> outstanding
> >>>> PRs
> >>>>>>> that
> >>>>>>>>>>> are
> >>>>>>>>>>>>>>>> slated to be included
into 1.9.0:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS
   |DESCRIPTION
> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open 
    |XSS Vulnerability in Variable
> >>>>>> endpoint
> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open 
    |Customize logging in Airflow
> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened
 |Fix log source of local
> >>> loggers
> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open 
    |Rename the logger to log
> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open 
    |Fix minor LICENSE & NOTICE
> >>> issue
> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
duplicate and
> >>> unneeded
> >>>>> code
> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open 
    |HDFSOperator to operate HDFS
> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open 
    |airflow/jobs.py:create_dag_ru
> >>> n()
> >>>>>>>>>> exception
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open 
    |active_dagruns shouldn't
> >>> include
> >>>>>> paused
> >>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open 
    |Scheduler DAG processes can
> >>> not
> >>>> log
> >>>>>> to
> >>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open 
    |TreeView displayed over task
> >>>>>> instances
> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open 
    |airflow/jobs.py:manage_slas()
> >>>>>> exception
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open 
    |Mark success running task
> >>> causes
> >>>> it
> >>>>>> to
> >>>>>>>>>> fail
> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open 
    |Refactor
> >>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open 
    |Refactor
> >>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>> job
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open 
    |Refactor tests and build
> >>> matrix
> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open 
    |Operators should not push
> >>> XComs
> >>>> by
> >>>>>>>>>> default
> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open 
    |Add maximum size for XComs
> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open 
    |Add Dataflow semantics
> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open 
    |Context unexpectedly added to
> >>>> hive
> >>>>>> conf
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I will be locking down
what can get cherry-picked into
> >>> the
> >>>>>> 1.9.0
> >>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>> shortly, so if you have
something you want in, please
> >>> set
> >>>>> the
> >>>>>>> fix
> >>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> We (at WePay) have deployed
1.9.0 into our dev cluster,
> >>>> and
> >>>>> it
> >>>>>>> has
> >>>>>>>>>>>> been
> >>>>>>>>>>>>>>>> running smoothly for
several days.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ** I could really use
help verifying stability. If you
> >>> run
> >>>>>>>>>> Airflow,
> >>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>> in your best interest
to deploy the 1.9.0 test branch
> >>>>>> somewhere,
> >>>>>>>>>> and
> >>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>> it's working for your
workload. **
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message