airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Halgrim <scott.halg...@zapier.com.INVALID>
Subject Re: Airflow Deployment tools
Date Wed, 15 Nov 2017 18:52:06 GMT
I bet the folks at astronomer.io would love to talk to you about Airflow as a service

On Nov 15, 2017, 4:31 AM -0800, Andrew Maguire <andrewm4894@gmail.com>, wrote:
> Is there any options at all out there for Airflow as a service type
> approach?
>
> I'd love to just be able to define my dags and load them to some cloud ui
> and not have to worry about anything else.
>
> This looks kinda interesting -
> http://docs.qubole.com/en/latest/user-guide/airflow/introduction-airflow.html
>
> Cheers,
> Andy
>
> On Wed, Nov 15, 2017 at 10:28 AM Driesprong, Fokko <fokko@driesprong.frl
> wrote:
>
> > I'm using Ansible to deploy the Airflow, the steps are:
> > - First install Airflow using pip (or a rc using curl)
> > - Do an `airflow version` to trigger the creation of the default config
> > - Set the config correctly variables in the config using Ansible.
> > - Deploy the supervisord files
> > - Start everything
> >
> > A separate role is there to deploy Postgres. But if you are working on a
> > cloud environment, you can also get Postgres/MySQL as a service. Hope this
> > helps.
> >
> > Cheers, Fokko
> >
> > 2017-11-15 3:19 GMT+01:00 Marc Bollinger <marc@lumoslabs.com>:
> >
> > > Samson <https://github.com/zendesk/samson> deploy that runs a script
> > > running a Broadside <https://github.com/lumoslabs/broadside> deploy for
> > > ECS, which bounces the Web and Scheduler workers, and updates the DAG
> > > directory on the workers. Docker images come from a Github -> Travis -
> > > Quay
> > > <https://quay.io/> CI setup.
> > >
> > > On Tue, Nov 14, 2017 at 10:18 AM, Alek Storm <alek.storm@gmail.com
> > wrote:
> > >
> > > > Our TeamCity server detects the master branch has changed, then
> > packages
> > > up
> > > > the repo containing our DAGs as an artifact. We then use SaltStack to
> > > > trigger a bash script on the targeted servers that downloads the
> > > artifact,
> > > > moves the files to the right place, and restarts the scheduler (on the
> > > > master).
> > > >
> > > > This allows us to easily revert changes by redeploying a particular
> > > > TeamCity artifact, without touching the git history.
> > > >
> > > > Alek
> > > >
> > > > On Nov 14, 2017 11:02 AM, "Andy Hadjigeorgiou" <andyxhadji@gmail.com
> > > > wrote:
> > > >
> > > > > Hey,
> > > > >
> > > > > Was just wondering what tools & services everyone uses to deploy
new
> > > > > versions of their data pipelines (understandably this would vary
> > > greatly
> > > > > based on tech stack) but I'd love to hear what the community has
been
> > > > > using.
> > > > >
> > > > > - Andy
> > > > >
> > > >
> > >
> >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message