airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Van Boxel <a...@vanboxel.be>
Subject Re: Airflow 1.8.0 alpha 4
Date Wed, 11 Jan 2017 14:39:35 GMT
Hey Bolke, I'll be of for a few days but I started looking at the
CHANGELIST to make sure we now have a list of changes for 1.8 (seems like a
big list...).

I'll will also drop the trigger issue for the 1.8 and see what we can do
for the next version. I'll make it into a documentation issue.

On Wed, Jan 11, 2017 at 1:46 PM Bolke de Bruin <bdbruin@gmail.com> wrote:

> Dear All,
>
> I would like to drop "Schedule all pending DAG runs in a single scheduler
> loop” from the 1.8.0 release (updated:
> https://github.com/apache/incubator-airflow/pull/1980 <
> https://github.com/apache/incubator-airflow/pull/1980>, original:
> https://github.com/apache/incubator-airflow/pull/1906 <
> https://github.com/apache/incubator-airflow/pull/1906>). The reason for
> this is that it, imho, biases the scheduler towards a single DAG as it
> fills the queue with tasks from one DAG and then goes to the next DAG.
> Starving DAGs that come after the first for resources. As such it should be
> updated and that will take time.
>
> Please let me know if I am incorrect.
>
> Thanks
> Bolke
>
> > On 10 Jan 2017, at 09:25, Bolke de Bruin <bdbruin@gmail.com> wrote:
> >
> > Dear All,
> >
> > I have made Airflow 1.8.0 alpha 4 available at
> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> .
> Again no Apache release yet - this is for testing purposes. I consider this
> Alpha to be a Beta if not for the pending features. If the pending features
> are merged within a reasonable time frame (except for **, as no progress
> currently) then I am planning to mark the tarball as Beta and only allow
> bug fixes and (very) minor features. This week hopefully.
> >
> > Blockers:
> >
> > * None
> >
> > Fixed issues
> > * Regression in email
> > * LDAP case sensitivity
> > * one_failed task not being run: now seems to pass suddenly (so fixed?)
> -> need to investigate why
> > * Email attachments
> > * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression)
> > * Improve time units for task performance charts
> > * XCom throws an duplicate / locking error
> > * Add execution_date to trigger_dag
> >
> > Pending features:
> > * DAG.catchup : minor changes needed, documentation still required,
> integration tests seem to pass flawlessly
> > * Cgroups + impersonation: clean up of patches on going, more tests and
> more elaborate documentation required. Integration tests not executed yet
> > * Schedule all pending DAG runs in a single scheduler loop: no progress
> (**)
> >
> > Cheers!
> > Bolke
>
> --
  _/
_/ Alex Van Boxel

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message