airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Coder <jcode...@gmail.com>
Subject Re: Tagging of the airflow images
Date Tue, 18 Jun 2019 09:16:25 GMT
Just my 2 cents, at work we tend to use “slim” to denote things that are pared down. 
How about “ci-slim”?

James Coder

> On Jun 18, 2019, at 4:06 AM, Jarek Potiuk <Jarek.Potiuk@polidea.com> wrote:
> 
> Ok so then next iteration of proposal: The only doubt I have myself is the
> "master" vs. "master-prod". Maybe we should rather have "master" for
> "production-ready" image and use a different name for the "small-ci"
> image". Maybe "trimci" or "ci-lite" or "liteci" ?
> What do you think?
> 
> *Versions from master (development use only):*
> 
>   - Main non-CI images (small) *: airflow:master-python3.5,
> airflow:**master-python3.6,
>   airflow:master*==airflow:master-python3.6
>   - CI images (big) *: airflow:master-ci-python3.5,
> airflow:**master-ci-python3.6,
>   airflow:master-ci*==airflow:master-ci-python3.6
>   - Production optimised images: (future): *airflow:master-prod-python3.5,
>   airflow:**master-prod-python3.6, airflow:master-prod*
>   ==airflow:master-prod-python3.6
> 
> *Release versions (future):*
> 
>   - Main non-CI images (small):  *airflow:1.10.4-python3.5, *
>   *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.5
>   - CI images (big): *airflow:1.10.4-ci-python3.5,
> **airflow:1.10.4**-ci-python3.6,
>   airflow:latest-ci*==airflow:1.10.4-ci-python3.6
>   - Production optimised images: *airflow:1.10.4-prod-python3.5, *
>   *airflow:1.10.4**-prod-python3.6, airflow:latest-prod*
>   ==airflow:1.10.4-prod-python3.6
> 
> 
> 
> On Mon, Jun 17, 2019 at 10:31 PM Philippe Gagnon <philgagnon1@gmail.com>
> wrote:
> 
>> That makes sense. The reason I had doubts is because of the way docker hub
>> lists image tags together -- there's no real differentiation between
>> pre-release and release builds. But then I suppose that if the tagging
>> scheme is explicit enough it shouldn't be an issue.
>> 
>> +1 on `:latest` being an official release.
>> 
>>> On Mon, Jun 17, 2019 at 10:25 AM Ash Berlin-Taylor <ash@apache.org> wrote:
>>> 
>>> That page does mention "Nightly" builds which is close to what building
>>> master would be. The other thing that matters is what we actual call A
>>> Release.
>>> 
>>>> Do not include any links on the project website that might encourage
>>> non-developers to download and use nightly builds, snapshots, release
>>> candidates, or any other similar package
>>> 
>>> I think we're find so long as we don't do that -- or in this case, since
>>> we will probably want to link to the docker hub page once we have
>> versioned
>>> images there if we make it clear that `:master` is not intended for end
>>> users, and by the same argument if we have anything as `:latest` it
>> should
>>> be a docker image relating to an official Release.
>>> 
>>> Jarek: no `latest` pointing at CI images please.
>>> 
>>> -a
>>> 
>>>> On 17 Jun 2019, at 15:04, Philippe Gagnon <philgagnon1@gmail.com>
>> wrote:
>>>> 
>>>> One thing: we talked about releasing images under a "master" tag
>>> (perhaps in another thread?), we should check if this is compatible with
>>> Apache's release policy [1]. It's not clear to me if this is allowable or
>>> not after a cursory reading.
>>>> 
>>>> [1] http://www.apache.org/legal/release-policy.html#what
>>>> 
>>>> On Mon, Jun 17, 2019 at 9:48 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
>>> 
>>> wrote:
>>>> Anyone has more comments. I think prevailing opnion is:
>>>> 1) To keep all images in one repo (apache/airflow)
>>>> 2) I am not sure about labelling but I might try to document all cases
>>> in a
>>>> "production" image proposal that I would like to start as soon as we
>>> merge
>>>> the current CI image (which I think is quite close to finalisation).
>>>> 
>>>> J.
>>>> 
>>>> On Tue, Jun 11, 2019 at 10:59 PM Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>
>>>> wrote:
>>>> 
>>>>> It's super easy to do :)
>>>>> 
>>>>> On Tue, Jun 11, 2019 at 10:33 PM Ash Berlin-Taylor <ash@apache.org>
>>> wrote:
>>>>> 
>>>>>> I'm fine with us just publishing release images using the newest
>>> python
>>>>>> release (i.e. 3.7) as the main reason we support older python
>>> versions is
>>>>>> to support distros thats ship those versions.(i.e. Deb stable), but
>> I
>>> don't
>>>>>> think we need to support that in docker.
>>>>>> 
>>>>>> (But if it's easy to do since we want them for ci then sure)
>>>>>> 
>>>>>> -ash
>>>>>> 
>>>>>> On 11 June 2019 21:21:28 BST, Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Yeah Kamil - python 3.5 is the default one for now. I think we
>>> should have
>>>>>>> another discussion here - how many versions to support. There
is
>> this
>>>>>>> ticket opened today :
>>> https://issues.apache.org/jira/browse/AIRFLOW-4762 about
>>>>>>> supporting python 3.6 and 3.7 in tests. Anyone has a strong opinion
>>> on
>>>>>>> this? I am for testing on all 3.5, 3.6 and 3.7 even if it increases
>>> the
>>>>>>> build/test time on Travis. There are a number of differences
>> between
>>> those
>>>>>>> major versions (I have a blog post about it in writing ) but
I
>> think
>>> there
>>>>>>> is concern about eating Apache Travis time.
>>>>>>> 
>>>>>>> Anyone against those three ?
>>>>>>> 
>>>>>>> On Tue, Jun 11, 2019 at 8:38 PM Kamil Breguła <
>>> kamil.bregula@polidea.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 1) I would prefer to use one repository.
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> 2) The presented schema looks logical to me. I had doubts
whether
>>>>>>>> Python 3.5 was a good choice for "latest" version, but I
checked
>>> that
>>>>>>>> travis uses only this version.
>>>>>>>> 
>>>>>>>> On Tue, Jun 11, 2019 at 3:04 PM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hello everyone,
>>>>>>>>> 
>>>>>>>>> We are close to finish AIP-10 (Airlfow image for CI)
and seems
>>> that we
>>>>>>>>> 
>>>>>>>> will
>>>>>>>> 
>>>>>>>>> start working soon on an official image AIP, but in the
meantime
>>> we have
>>>>>>>>> 1.10.4 release coming and we would like to agree tagging
scheme
>>> used for
>>>>>>>>> the current CI images. We discussed it a bit on Slack,
but it's
>>> time to
>>>>>>>>> bring it here. I created a JIRA issue for it:
>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4764  and
my
>>> proposals
>>>>>>>>> 
>>>>>>>> after
>>>>>>>> 
>>>>>>>>> the initial discussion are those:
>>>>>>>>> 
>>>>>>>>> First of all we have different images that we can talk
about :
>>>>>>>>> 
>>>>>>>>>    1. "base" one - with bare development-ready airflow
with
>>> minimum set
>>>>>>>>> 
>>>>>>>> of
>>>>>>>> 
>>>>>>>>>    dependencies
>>>>>>>>>    2. "CI" with all the tools packages that are needed
for CI
>>> tests
>>>>>>>>>    3. Soon we will likely have an "official" one which
might be
>>> used in
>>>>>>>>>    similar fashion as the "puckel" one.
>>>>>>>>> 
>>>>>>>>> There are two decisions to make:
>>>>>>>>> 
>>>>>>>>> 1) How to keep those images - in one repository or whether
we
>>> should have
>>>>>>>>> separate repos.
>>>>>>>>> 
>>>>>>>>> It is easier for now to keep all of them within apache/airflow
>>>>>>>>> <
>>> https://cloud.docker.com/u/apache/repository/docker/apache/airflow>
>>>>>>>>> 
>>>>>>>> repository
>>>>>>>> 
>>>>>>>>> it seems and use a labelling scheme to separate those
(there is
>>> nothing
>>>>>>>>> wrong with that but it might seem a bit hacky). It's
a bit
>> easier
>>> to
>>>>>>>>> maintain with access and CI.
>>>>>>>>> 
>>>>>>>>> We could also think about separate apache/airflow-ci,
>>> apache/airflow-dev,
>>>>>>>>> apache/airflow-prod or smth similar - that would require
some
>>>>>>>>> infrastructure tickets and is not very common.
>>>>>>>>> 
>>>>>>>>> 2) What labelling scheme to use(apache/airflow:label).
My
>>> proposal is
>>>>>>>>> similar to this (if we keep everything in the airflow
>> repository)
>>>>>>>>> 
>>>>>>>>>    - *latest* = latest released version (python 3.5)
 = *
>>>>>>>>> 
>>>>>>>> v1.10.3-python3.5*
>>>>>>>> 
>>>>>>>>> - *master* = latest master version (python 3.5)  =
>>>>>>>>> 
>>>>>>>> *v2.0.0dev0-python3.5*
>>>>>>>> 
>>>>>>>>>    - *v1.10.3-python3.5,v1.10.3-python3.6*  - released
1.10.3
>>> with python
>>>>>>>>>    3.5/3.6
>>>>>>>>>    - *latest-ci *= latest released version of CI variant
(python
>>> 3.5)
>>>>>>>>>    *v1.10.3-ci-python3.5*
>>>>>>>>>    - *master-ci* = latest master version of CI variant
(python
>>> 3.5)
>>>>>>>>>    *v2.0.0dev0-ci-python3.5*
>>>>>>>>>    - *v1.10.3-ci-python3.5, v1.10.3-ci-python3.6* - released
>>> 1.10.3 with
>>>>>>>>>    python 3.5/3.6
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> My preference is to keep all the images in one repo and
use
>>> labelling
>>>>>>>>> scheme as above,
>>>>>>>>> but I am open to discuss this.
>>>>>>>>> 
>>>>>>>>> J,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> Jarek Potiuk
>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
Software
>> Engineer
>>>>>>>>> 
>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>> 
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> 
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>> 
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>> 
>>> 
>> 
> 
> 
> -- 
> 
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
> 
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>

Mime
View raw message