Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Wed, 15 Nov 2017 10:52:06 -0800
From: Scott Halgrim <scott.halgrim@zapier.com.INVALID>
To: dev@airflow.incubator.apache.org
Message-ID: <ea8decd6-c5f4-4cbe-a348-793f69a29e3a@Spark>
In-Reply-To: <CANZLcgh76Q9v0pMiHa2umMnw2coThxtHL43x8kSYMrMp17BJ3g@mail.gmail.com>
References: <CAEqTbYkcHq6H7vC0Z4TkOKpfC4rKM02or0L_cY+d1V7fegbwVw@mail.gmail.com>
 <CAMNEcwuOhrnyzez7Lk6XhFeMHBn2MxJZ4dsAUq=ysSKqaz-JhA@mail.gmail.com>
 <CAAO2x27d5DJspwwC_cnar4h_eawi3OoL7XJwXBzBW5iAqiDRVg@mail.gmail.com>
 <CAC15uRvzfEYV_ubd3WW4y4_u7NE3LOQ0CX_hX1jiPxbDe8iOYQ@mail.gmail.com>
 <CANZLcgh76Q9v0pMiHa2umMnw2coThxtHL43x8kSYMrMp17BJ3g@mail.gmail.com>
Subject: Re: Airflow Deployment tools
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="5a0c8ce9_41b71efb_382"
archived-at: Wed, 15 Nov 2017 18:52:44 -0000

--5a0c8ce9_41b71efb_382
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I bet the folks at astronomer.io would love to talk to you about Airflow as a service

On Nov 15, 2017, 4:31 AM -0800, Andrew Maguire <andrewm4894@gmail.com>, wrote:
> Is there any options at all out there for Airflow as a service type
> approach?
>
> I'd love to just be able to define my dags and load them to some cloud ui
> and not have to worry about anything else.
>
> This looks kinda interesting -
> http://docs.qubole.com/en/latest/user-guide/airflow/introduction-airflow.html
>
> Cheers,
> Andy
>
> On Wed, Nov 15, 2017 at 10:28 AM Driesprong, Fokko <fokko@driesprong.frl
> wrote:
>
> > I'm using Ansible to deploy the Airflow, the steps are:
> > - First install Airflow using pip (or a rc using curl)
> > - Do an `airflow version` to trigger the creation of the default config
> > - Set the config correctly variables in the config using Ansible.
> > - Deploy the supervisord files
> > - Start everything
> >
> > A separate role is there to deploy Postgres. But if you are working on a
> > cloud environment, you can also get Postgres/MySQL as a service. Hope this
> > helps.
> >
> > Cheers, Fokko
> >
> > 2017-11-15 3:19 GMT+01:00 Marc Bollinger <marc@lumoslabs.com>:
> >
> > > Samson <https://github.com/zendesk/samson> deploy that runs a script
> > > running a Broadside <https://github.com/lumoslabs/broadside> deploy for
> > > ECS, which bounces the Web and Scheduler workers, and updates the DAG
> > > directory on the workers. Docker images come from a Github -> Travis -
> > > Quay
> > > <https://quay.io/> CI setup.
> > >
> > > On Tue, Nov 14, 2017 at 10:18 AM, Alek Storm <alek.storm@gmail.com
> > wrote:
> > >
> > > > Our TeamCity server detects the master branch has changed, then
> > packages
> > > up
> > > > the repo containing our DAGs as an artifact. We then use SaltStack to
> > > > trigger a bash script on the targeted servers that downloads the
> > > artifact,
> > > > moves the files to the right place, and restarts the scheduler (on the
> > > > master).
> > > >
> > > > This allows us to easily revert changes by redeploying a particular
> > > > TeamCity artifact, without touching the git history.
> > > >
> > > > Alek
> > > >
> > > > On Nov 14, 2017 11:02 AM, "Andy Hadjigeorgiou" <andyxhadji@gmail.com
> > > > wrote:
> > > >
> > > > > Hey,
> > > > >
> > > > > Was just wondering what tools & services everyone uses to deploy new
> > > > > versions of their data pipelines (understandably this would vary
> > > greatly
> > > > > based on tech stack) but I'd love to hear what the community has been
> > > > > using.
> > > > >
> > > > > - Andy
> > > > >
> > > >
> > >
> >

--5a0c8ce9_41b71efb_382--