airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Airflow Releases
Date Fri, 30 Sep 2016 15:51:40 GMT
> I'm not sure how other projects do this, but I propose that we let the RC settle for a
week.

+1

> Who's on board!?

Me. :)

On Fri, Sep 30, 2016 at 7:56 AM, Maxime Beauchemin
<maximebeauchemin@gmail.com> wrote:
> As related note: the most people we can get to test this release candidate
> the better. Note that Airbnb uses only a subset of the Airflow codebase,
> there are many operators and features we do not use at all, so we encourage
> everyone to get on board with testing the RC.
>
> At Airbnb, our process starts with sending the RC into our staging cluster,
> let it settle and fix any bug over 48 hours or so. We look for changes
> around DagBag processing time, memory footprint on workers, smooth
> processing around our canaries.
>
> When the RC is ready to hit production, we take a full backup of the
> metadata DB, and make sure that the whole team is fully available to babysit
> the release and hotfix any issue that may pop up, with a clear, fast revert
> plan.
>
> As all of our tasks are assumed to be idempotent the worst case scenarios
> are around delayed SLAs and wasted resources which is all manageable.
>
> I'm not sure how other projects do this, but I propose that we let the RC
> settle for a week. I suggest that we reset the clock to 7 more days for each
> increment of RC, unless the hotfix is very minor.
>
> Who's on board!?
>
> Max
>
> On Fri, Sep 30, 2016 at 7:43 AM, Maxime Beauchemin
> <maximebeauchemin@gmail.com> wrote:
>>
>> Airbnb will cut a release candidate on the week of October 10th off of
>> master.
>>
>> Community, if you have outstanding PRs that you want to make this release,
>> now is the time! Though we do not want to rush-merge anything risky that may
>> make the release process harder.
>>
>> Max
>>
>> On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke
>> <luke.maycock@affiliate.oliverwyman.com> wrote:
>>>
>>> Hi Chris,
>>>
>>>
>>> We are currently running master in our environment and we'll be sure to
>>> report any issues.
>>>
>>>
>>> Do you know when the 1.8 release is due?
>>>
>>>
>>> Cheers,
>>>
>>> Luke Maycock
>>> OLIVER WYMAN
>>>
>>> luke.maycock@affiliate.oliverwyman.com<mailto:luke.maycock@affiliate.oliverwyman.com>
>>> www.oliverwyman.com<http://www.oliverwyman.com/>
>>>
>>>
>>> ________________________________
>>> From: Chris Riccomini <criccomini@apache.org>
>>> Sent: 29 September 2016 18:14:02
>>> To: dev@airflow.incubator.apache.org
>>> Subject: Re: Airflow Releases
>>>
>>> Hey Luke,
>>>
>>> > Is there anything we can do to help get the next release in place?
>>>
>>> One thing that would definitely help is running master somewhere in
>>> your environment, and reporting any issues that see. Over the next few
>>> weeks, AirBNB and a few other folks will be doing the same in an
>>> effort to harden the 1.8 release.
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke
>>> <luke.maycock@affiliate.oliverwyman.com> wrote:
>>> > Airflow Developers,
>>> >
>>> >
>>> > We were looking at writing a workflow framework in Python when we found
>>> > Airflow. We have carried out some proof of concept work for using Airflow
>>> > and wish to continue using it as it comes with lots of great features
>>> > out-of-the-box.
>>> >
>>> >
>>> > We have created our own fork here:
>>> >
>>> > https://github.com/owlabs/incubator-airflow
>>> >
>>> >
>>> > So far, the only thing we have committed back to the main repository is
>>> > the following fix to the mssql_hook:
>>> >
>>> > https://github.com/apache/incubator-airflow/pull/1626
>>> >
>>> >
>>> > Among other types of tasks, we wish to be able to run mssql tasks using
>>> > Airflow. In order to do so, the above and below fixes are required:
>>> >
>>> >
>>> > https://github.com/apache/incubator-airflow/pull/1458<https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed>
>>> >
>>> >
>>> > We have created a Chef cookbook for configuring VMs with Airflow and
>>> > its prerequisites. As part of this cookbook, we are installing the latest
>>> > release of Airflow. However, it appears that the latest release does not
>>> > have the aforementioned fixes.
>>> >
>>> > Do you know when the next release of Airflow is expected? Is there
>>> > anything we can do to help get the next release in place?
>>> >
>>> >
>>> > Luke Maycock
>>> > OLIVER WYMAN
>>> >
>>> > luke.maycock@affiliate.oliverwyman.com<mailto:luke.maycock@affiliate.oliverwyman.com>
>>> > www.oliverwyman.com<http://www.oliverwyman.com/>
>>> >
>>> >
>>> > ________________________________
>>> > This e-mail and any attachments may be confidential or legally
>>> > privileged. If you received this message in error or are not the intended
>>> > recipient, you should destroy the e-mail message and any attachments or
>>> > copies, and you are prohibited from retaining, distributing, disclosing
or
>>> > using any information contained herein. Please inform us of the erroneous
>>> > delivery by return e-mail. Thank you for your cooperation.
>>>
>>> ________________________________
>>> This e-mail and any attachments may be confidential or legally
>>> privileged. If you received this message in error or are not the intended
>>> recipient, you should destroy the e-mail message and any attachments or
>>> copies, and you are prohibited from retaining, distributing, disclosing or
>>> using any information contained herein. Please inform us of the erroneous
>>> delivery by return e-mail. Thank you for your cooperation.
>>
>>
>

Mime
View raw message