airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <>
Subject Re: Integration test env
Date Tue, 20 Dec 2016 16:18:20 GMT
Hi Chris,

Didn’t they offer a dedicated Travis thing as well? And can we setup long running services
(ie. Hadoop, KDC, Presto, MySQL, etc) in the Jenkins environment, because that would be sufficient
I guess? To my knowledge what Apache offers is more suited for unit tests (ie. the hadoop-qa
runs) than integration tests.

What are your thoughts?


> Op 19 dec. 2016, om 22:26 heeft Chris Riccomini <> het volgende
> Hey Bolke,
> Thanks for stepping up. The "traditional" Apache infra is Jenkins with a
> pool of machines that they provide. That might or might not be satisfactory
> for us (it's certainly antiquated technology). If we decide we don't like
> it, I think that's OK, as long as we move forward knowing that any other
> testing solution can disappear at any time.
> My 2c. :)
> Cheers,
> Chris
> On Wed, Dec 14, 2016 at 11:11 AM, Dan Davydov <
>> wrote:
>> This is extremely generous of you! I do agree with the approach of trying
>> to get funding from Apache and having shared resources (e.g. so that we
>> don't depend on any one company or individual for the uptime of the
>> integration environment, plus so we would have public cloud integration
>> potentially).
>> On Wed, Dec 14, 2016 at 1:22 AM, Bolke de Bruin <> wrote:
>>> Hi,
>>> I have been thinking about an integration test environment. Aside from
>> any
>>> technical requirements we need a place to do it. I am willing to offer a
>>> place in Lab env I am running or to fund an environment in AWS/GCloud if
>>> Apache cannot make these kind of resources available.
>>> If running in our Lab there is virtually no restriction what we could do,
>>> however I will hand select people who have access to this environment. I
>>> will also hold ultimate power to remove access from anyone. I even might
>>> ask for a confirmation that you will behave when using our property
>> (don’t
>>> worry won’t cover it with legal wording). This is a IAAS service so we
>> need
>>> to cover the things we need ourselves, but the upside is we can and it is
>>> free. We could setup a Gitlab instance that mirrors from Apache a kicks
>> off
>>> runners to do testing. Downside 1) it might not be entirely Apache like.
>>> Sorry cant help that. 2) there is no guaranteed up time 3) I might need
>> to
>>> remove it in the future e.g. when I change jobs for example :). 4) No
>>> public cloud integration, it’s a private stack after all.
>>> I can also fund on AWS/GCloud. Again, I probably want to have ultimate
>>> power on access to this environment - it’s my company’s money on the line
>>> after all. Major downside to this is that it is dependent on and limited
>> by
>>> the budget I can make available. Upside is that it is not company
>> property.
>>> Also I personally have less exposure to public cloud environments due to
>>> company restrictions.
>>> Are there any other options? Any thoughts?
>>> Bolke

View raw message