airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerardo Curiel <gera...@gerar.do>
Subject Re: AIP-7 Simplified development workflow
Date Fri, 04 Jan 2019 00:40:16 GMT
I'll start adding some comments and updates to the proposal and
sub-projects.

On 3 January 2019 at 8:57:41 am, Daniel Imberman (daniel.imberman@gmail.com)
wrote:

Hi guys, I've set up a few sub-projects for this. @gerardo @fokko Lemme
know what you guys think

https://cwiki.apache.org/confluence/display/AIRFLOW/Optimizing+Docker+Image+Workflow
https://cwiki.apache.org/confluence/display/AIRFLOW/Kubernetes+Testing%3A+Using+GKE+instead+of+Minikube

On Tue, Jan 1, 2019 at 11:45 PM Driesprong, Fokko <fokko@driesprong.frl>
wrote:

> Hi Gerardo,
>
> Very valid points. I'm fully in favor of your proposal. To simplify the
> stack, I strongly believe we should also strip out tox and fully rely on
> Docker. Using tox will add another layer that doesn't add a lot of value
> from my perspective. Also, we should bake all the *.sh bootstrap scripts
> <https://github.com/apache/incubator-airflow/tree/master/scripts/ci> in
> the
> Docker container, instead of having to set this up before running the
> tests.
>
> In the upcoming months, I might have a bit more time to spend on Airflow,
> I'm happy to assist you on this one.
>
> Cheers, Fokko
>
> Op wo 2 jan. 2019 om 06:51 schreef Daniel Imberman <
> daniel.imberman@gmail.com>:
>
> > @gerardo thank you for setting this up.
> >
> > I've also been extremely interested in this as well. I've been messing
> with
> > GCP VM instances in the past few weeks to try to simplify my local
build
> as
> > well. Would definitely be interested in helping with the AIP +
> > implementation.
> >
> > One thing I believe we should do is set up the ci base-image with all
of
> > the pip dependencies pre-loaded. A lot of time is wasted pip installing
> > dependencies. We can auto-generate new images whenever a PR is
submitted
> to
> > this repository and then specify the tag in the .travis.yml when
> building.
> >
> > On the k8s side, I think we need to move away from minikube for k8s
> > testing. I discussed in a previous email setting travis to work with
GKE.
> > I'd be careful about coupling k8s stuff too tightly with a docker
> > infrastructure. That can get pretty dicey. I think as long as we're
> using a
> > separate k8s cluster the k8s executor tests only need to gather the IP
> > addresses + have access to the kubeconfig.
> >
> >
> > On Tue, Jan 1, 2019 at 8:10 PM Gerardo Curiel <gerardo@gerar.do> wrote:
> >
> > > Hi folks,
> > >
> > > I've created an AIP for simplifying Airflow's development workflow:
> > >
> > >
> >
>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow
> > >
> > > The goal of this proposal is to outline the work needed to make local
> > > testing significantly easier and standardise the best practices to
> > > contribute to the Airflow project.
> > >
> > > Any input on it would be greatly appreciated.
> > >
> > > Cheers,
> > >
> > > --
> > > Gerardo Curiel // https://gerar.do
> >
> >
> > On Tue, Jan 1, 2019 at 8:10 PM Gerardo Curiel <gerardo@gerar.do> wrote:
> >
> > > Hi folks,
> > >
> > > I've created an AIP for simplifying Airflow's development workflow:
> > >
> > >
> >
>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow
> > >
> > > The goal of this proposal is to outline the work needed to make local
> > > testing significantly easier and standardise the best practices to
> > > contribute to the Airflow project.
> > >
> > > Any input on it would be greatly appreciated.
> > >
> > > Cheers,
> > >
> > > --
> > > Gerardo Curiel // https://gerar.do
> > >
> >
>

--
Gerardo Curiel // https://gerar.do

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message