airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Airflow 1.8.0 alpha 4
Date Wed, 11 Jan 2017 21:30:30 GMT
@bolke, this sounds like a good list.

On Wed, Jan 11, 2017 at 12:01 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> Ok.
>
> For now to call it “beta” 4 items seems to be left:
>
> Blocker:
> * retry_delay not respected
> * poison pill due to re-queue before process has finished (to be
> investigated)
>
> Features:
> * cgroups + impersonation
> * dag.catchup (Ben Tallman -> Only documentation is missing).
>
> PRs that contain documentation would really be appreciated. In my opinion
> we are lacking there. Think about docs covering:
> * new scheduler behaviour and options
> * task dependency engine
> * api / kerberized api
> * …
>
> Cheers
> Bolke
>
> > On 11 Jan 2017, at 18:59, Arthur Wiedmer <arthur.wiedmer@gmail.com>
> wrote:
> >
> > +1
> >
> > We can always think about different ways of doing this later (fair share
> > scheduling etc...)
> >
> > Best,
> > Arthur
> >
> > On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >
> >> Dear All,
> >>
> >> I would like to drop "Schedule all pending DAG runs in a single
> scheduler
> >> loop” from the 1.8.0 release (updated: https://github.com/apache/
> >> incubator-airflow/pull/1980 <https://github.com/apache/
> >> incubator-airflow/pull/1980>, original: https://github.com/apache/
> >> incubator-airflow/pull/1906 <https://github.com/apache/
> >> incubator-airflow/pull/1906>). The reason for this is that it, imho,
> >> biases the scheduler towards a single DAG as it fills the queue with
> tasks
> >> from one DAG and then goes to the next DAG. Starving DAGs that come
> after
> >> the first for resources. As such it should be updated and that will take
> >> time.
> >>
> >> Please let me know if I am incorrect.
> >>
> >> Thanks
> >> Bolke
> >>
> >>> On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbruin@gmail.com> wrote:
> >>>
> >>> Dear All,
> >>>
> >>> I have made Airflow 1.8.0 alpha 4 available at
> >> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/>
.
> >> Again no Apache release yet - this is for testing purposes. I consider
> this
> >> Alpha to be a Beta if not for the pending features. If the pending
> features
> >> are merged within a reasonable time frame (except for **, as no progress
> >> currently) then I am planning to mark the tarball as Beta and only allow
> >> bug fixes and (very) minor features. This week hopefully.
> >>>
> >>> Blockers:
> >>>
> >>> * None
> >>>
> >>> Fixed issues
> >>> * Regression in email
> >>> * LDAP case sensitivity
> >>> * one_failed task not being run: now seems to pass suddenly (so fixed?)
> >> -> need to investigate why
> >>> * Email attachments
> >>> * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression)
> >>> * Improve time units for task performance charts
> >>> * XCom throws an duplicate / locking error
> >>> * Add execution_date to trigger_dag
> >>>
> >>> Pending features:
> >>> * DAG.catchup : minor changes needed, documentation still required,
> >> integration tests seem to pass flawlessly
> >>> * Cgroups + impersonation: clean up of patches on going, more tests and
> >> more elaborate documentation required. Integration tests not executed
> yet
> >>> * Schedule all pending DAG runs in a single scheduler loop: no progress
> >> (**)
> >>>
> >>> Cheers!
> >>> Bolke
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message