airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Van Boxel <a...@vanboxel.be>
Subject Re: Integration test env
Date Mon, 19 Dec 2016 22:56:29 GMT
I finally have some time to spend on my series of blog articles based on
the Google Cloud operator in Airflow. I started the GitHub repo with the
examples: https://github.com/alexvanboxel/airflow-gcp-examples

The main setup is that they should be *smokes tests* that would be able to
run *once a day *(because they are partitioned per day). Would the an idea
for the integration tests (and is once per day enough?)

On Mon, Dec 19, 2016 at 10:26 PM Chris Riccomini <criccomini@apache.org>
wrote:

> Hey Bolke,
>
> Thanks for stepping up. The "traditional" Apache infra is Jenkins with a
> pool of machines that they provide. That might or might not be satisfactory
> for us (it's certainly antiquated technology). If we decide we don't like
> it, I think that's OK, as long as we move forward knowing that any other
> testing solution can disappear at any time.
>
> My 2c. :)
>
> Cheers,
> Chris
>
> On Wed, Dec 14, 2016 at 11:11 AM, Dan Davydov <
> dan.davydov@airbnb.com.invalid> wrote:
>
> > This is extremely generous of you! I do agree with the approach of trying
> > to get funding from Apache and having shared resources (e.g. so that we
> > don't depend on any one company or individual for the uptime of the
> > integration environment, plus so we would have public cloud integration
> > potentially).
> >
> > On Wed, Dec 14, 2016 at 1:22 AM, Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > I have been thinking about an integration test environment. Aside from
> > any
> > > technical requirements we need a place to do it. I am willing to offer
> a
> > > place in Lab env I am running or to fund an environment in AWS/GCloud
> if
> > > Apache cannot make these kind of resources available.
> > >
> > > If running in our Lab there is virtually no restriction what we could
> do,
> > > however I will hand select people who have access to this environment.
> I
> > > will also hold ultimate power to remove access from anyone. I even
> might
> > > ask for a confirmation that you will behave when using our property
> > (don’t
> > > worry won’t cover it with legal wording). This is a IAAS service so we
> > need
> > > to cover the things we need ourselves, but the upside is we can and it
> is
> > > free. We could setup a Gitlab instance that mirrors from Apache a kicks
> > off
> > > runners to do testing. Downside 1) it might not be entirely Apache
> like.
> > > Sorry cant help that. 2) there is no guaranteed up time 3) I might need
> > to
> > > remove it in the future e.g. when I change jobs for example :). 4) No
> > > public cloud integration, it’s a private stack after all.
> > >
> > > I can also fund on AWS/GCloud. Again, I probably want to have ultimate
> > > power on access to this environment - it’s my company’s money on the
> line
> > > after all. Major downside to this is that it is dependent on and
> limited
> > by
> > > the budget I can make available. Upside is that it is not company
> > property.
> > > Also I personally have less exposure to public cloud environments due
> to
> > > company restrictions.
> > >
> > > Are there any other options? Any thoughts?
> > >
> > > Bolke
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
-- 
  _/
_/ Alex Van Boxel

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message