airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: Tagging of the airflow images
Date Tue, 18 Jun 2019 13:00:14 GMT
I like this ordering and the reasoning given for it.

> On 18 Jun 2019, at 13:54, Philippe Gagnon <philgagnon1@gmail.com> wrote:
> 
> I know this is bikeshedding at this point but I think something more alone
> the lines of:
> 
> `apache/airflow:1.10.4-python3.5[-ci][-slim]`
> 
> would be more appropriate as a standard. Here is my rationale:
> 
> - The airflow version should definitely come first.
> - The python version is the second-most significant information (other than
> airflow itself's version)
> - Everything else are extra "flavor" tags which we can then combine to
> create more specialized images if the need arises. We could define that
> extra tags should be ordered alphabetically.
> 
> 
> On Tue, Jun 18, 2019 at 6:02 AM Jarek Potiuk <Jarek.Potiuk@polidea.com>
> wrote:
> 
>> Great :). I'd prefer to use single word rather than ci-slim (then we can
>> use "-" as separator when splitting image name). The "slimci" seem like
>> most appropriate :
>> 
>> I also think that it might make sense to build production-optimised images
>> all the time. This way we will notice when they break and will be able to
>> test them before they are actually released. It's usually much more painful
>> to see that something does not work *just* at the moment you do release.
>> The common theme which I learned - if something is painful in software
>> releasing - simply do it more often, then you learn how to cope with it. So
>> I'd leave it even in master. Looking at yesterday's discussion in slack
>> <https://apache-airflow.slack.com/archives/CCPRP7943/p1560712788074100>
>> and
>> the nice stats generated by Bas (attached
>> <
>> https://drive.google.com/file/d/1BVB_SqBDAqaxxTNOP2reTrgY_7C74xK0/view?usp=sharing
>>> )
>> I'd propose python 3.6 to become the default version (while we are still
>> supporting 3.5).
>> 
>> It looks like we are converging to this:
>> 
>> *Versions from master (development use only):*
>> 
>>   - CI slim image *: airflow:master-slimci-python3.5,
>> airflow:**master-slimci-python3.6,
>>   airflow:master-slimci*==airflow:master-slimci-python3.6
>>   - CI full image *: airflow:master-ci-python3.5,
>> airflow:**master-ci-python3.6,
>>   airflow:master-ci*==airflow:master-ci-python3.6
>>   - Production optimised images: (future): *airflow:master-python3.5,
>>   airflow:**master-python3.6, airflow:master*==airflow:master-python3.6
>> 
>> *Release versions (future):*
>> 
>>   - CI slim image:  *airflow:1.10.4-slimci-python3.5, *
>>   *airflow:1.10.4-slimci**-python3.6, airflow:latest-slimci*
>>   ==airflow:1.10.4-slimci-python3.5
>>   - CI full image: *airflow:1.10.4-ci-python3.5,
>> **airflow:1.10.4**-ci-python3.6,
>>   airflow:latest-ci*==airflow:1.10.4-ci-python3.6
>>   - Production optimised images (future): *airflow:1.10.4-python3.5, *
>>   *airflow:1.10.4**-python3.6, airflow:latest*==airflow:1.10.4-python3.6
>> 
>> If no-one objects, I will update AIP-10 and will take it into account when
>> creating AIP for "production" image.
>> 
>> J.
>> 
>> On Tue, Jun 18, 2019 at 11:24 AM Ash Berlin-Taylor <ash@apache.org> wrote:
>> 
>>> "slim" is common amongst docker, so that sounds good.
>>> 
>>> I think the "primary" docker images should be production focused, and
>>> anything else tagged (i.e. it is prod unless it says otherwise.) Since
>>> master is not meant for end use we could also _only_ have the CI versions
>>> of those images.
>>> 
>>> 
>>> *Versions from master (development use only):*
>>> 
>>>  - CI images (big) *: airflow:master-ci-python3.5,
>>> airflow:**master-ci-python3.6,
>>>  airflow:master-ci*==airflow:master-ci-python3.6
>>> 
>>> *Release versions (future):*
>>> 
>>>  - Main non-CI images (small):  *airflow:1.10.4-python3.5-slim, *
>>>  *airflow:1.10.4**-python3.6-slim,
>>> airflow:latest-slim*==airflow:1.10.4-python3.5-slim
>>>  - CI images (big): *airflow:1.10.4-ci-python3.5,
>>> **airflow:1.10.4**-ci-python3.6,
>>>  airflow:latest-ci*==airflow:1.10.4-ci-python3.6
>>>  - Production optimised images: *airflow:1.10.4-python3.5, *
>>>  *airflow:1.10.4**-python3.6, airflow:latest*
>>>  ==airflow:1.10.4-python3.6
>>> 
>>> 
>>>> On 18 Jun 2019, at 10:16, James Coder <jcoder01@gmail.com> wrote:
>>>> 
>>>> Just my 2 cents, at work we tend to use “slim” to denote things that
>> are
>>> pared down.
>>>> How about “ci-slim”?
>>>> 
>>>> James Coder
>>>> 
>>>>> On Jun 18, 2019, at 4:06 AM, Jarek Potiuk <Jarek.Potiuk@polidea.com>
>>> wrote:
>>>>> 
>>>>> Ok so then next iteration of proposal: The only doubt I have myself is
>>> the
>>>>> "master" vs. "master-prod". Maybe we should rather have "master" for
>>>>> "production-ready" image and use a different name for the "small-ci"
>>>>> image". Maybe "trimci" or "ci-lite" or "liteci" ?
>>>>> What do you think?
>>>>> 
>>>>> *Versions from master (development use only):*
>>>>> 
>>>>> - Main non-CI images (small) *: airflow:master-python3.5,
>>>>> airflow:**master-python3.6,
>>>>> airflow:master*==airflow:master-python3.6
>>>>> - CI images (big) *: airflow:master-ci-python3.5,
>>>>> airflow:**master-ci-python3.6,
>>>>> airflow:master-ci*==airflow:master-ci-python3.6
>>>>> - Production optimised images: (future):
>>> *airflow:master-prod-python3.5,
>>>>> airflow:**master-prod-python3.6, airflow:master-prod*
>>>>> ==airflow:master-prod-python3.6
>>>>> 
>>>>> *Release versions (future):*
>>>>> 
>>>>> - Main non-CI images (small):  *airflow:1.10.4-python3.5, *
>>>>> *airflow:1.10.4**-python3.6,
>> airflow:latest*==airflow:1.10.4-python3.5
>>>>> - CI images (big): *airflow:1.10.4-ci-python3.5,
>>>>> **airflow:1.10.4**-ci-python3.6,
>>>>> airflow:latest-ci*==airflow:1.10.4-ci-python3.6
>>>>> - Production optimised images: *airflow:1.10.4-prod-python3.5, *
>>>>> *airflow:1.10.4**-prod-python3.6, airflow:latest-prod*
>>>>> ==airflow:1.10.4-prod-python3.6
>>>>> 
>>>>> 
>>>>> 
>>>>> On Mon, Jun 17, 2019 at 10:31 PM Philippe Gagnon <
>> philgagnon1@gmail.com
>>>> 
>>>>> wrote:
>>>>> 
>>>>>> That makes sense. The reason I had doubts is because of the way
>> docker
>>> hub
>>>>>> lists image tags together -- there's no real differentiation between
>>>>>> pre-release and release builds. But then I suppose that if the
>> tagging
>>>>>> scheme is explicit enough it shouldn't be an issue.
>>>>>> 
>>>>>> +1 on `:latest` being an official release.
>>>>>> 
>>>>>>> On Mon, Jun 17, 2019 at 10:25 AM Ash Berlin-Taylor <ash@apache.org>
>>> wrote:
>>>>>>> 
>>>>>>> That page does mention "Nightly" builds which is close to what
>>> building
>>>>>>> master would be. The other thing that matters is what we actual
>> call A
>>>>>>> Release.
>>>>>>> 
>>>>>>>> Do not include any links on the project website that might
>> encourage
>>>>>>> non-developers to download and use nightly builds, snapshots,
>> release
>>>>>>> candidates, or any other similar package
>>>>>>> 
>>>>>>> I think we're find so long as we don't do that -- or in this
case,
>>> since
>>>>>>> we will probably want to link to the docker hub page once we
have
>>>>>> versioned
>>>>>>> images there if we make it clear that `:master` is not intended
for
>>> end
>>>>>>> users, and by the same argument if we have anything as `:latest`
it
>>>>>> should
>>>>>>> be a docker image relating to an official Release.
>>>>>>> 
>>>>>>> Jarek: no `latest` pointing at CI images please.
>>>>>>> 
>>>>>>> -a
>>>>>>> 
>>>>>>>> On 17 Jun 2019, at 15:04, Philippe Gagnon <philgagnon1@gmail.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>> One thing: we talked about releasing images under a "master"
tag
>>>>>>> (perhaps in another thread?), we should check if this is compatible
>>> with
>>>>>>> Apache's release policy [1]. It's not clear to me if this is
>>> allowable or
>>>>>>> not after a cursory reading.
>>>>>>>> 
>>>>>>>> [1] http://www.apache.org/legal/release-policy.html#what
>>>>>>>> 
>>>>>>>> On Mon, Jun 17, 2019 at 9:48 AM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com
>>>>>>> 
>>>>>>> wrote:
>>>>>>>> Anyone has more comments. I think prevailing opnion is:
>>>>>>>> 1) To keep all images in one repo (apache/airflow)
>>>>>>>> 2) I am not sure about labelling but I might try to document
all
>>> cases
>>>>>>> in a
>>>>>>>> "production" image proposal that I would like to start as
soon as
>> we
>>>>>>> merge
>>>>>>>> the current CI image (which I think is quite close to
>> finalisation).
>>>>>>>> 
>>>>>>>> J.
>>>>>>>> 
>>>>>>>> On Tue, Jun 11, 2019 at 10:59 PM Jarek Potiuk <
>>>>>> Jarek.Potiuk@polidea.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> It's super easy to do :)
>>>>>>>>> 
>>>>>>>>> On Tue, Jun 11, 2019 at 10:33 PM Ash Berlin-Taylor <
>> ash@apache.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I'm fine with us just publishing release images using
the newest
>>>>>>> python
>>>>>>>>>> release (i.e. 3.7) as the main reason we support
older python
>>>>>>> versions is
>>>>>>>>>> to support distros thats ship those versions.(i.e.
Deb stable),
>> but
>>>>>> I
>>>>>>> don't
>>>>>>>>>> think we need to support that in docker.
>>>>>>>>>> 
>>>>>>>>>> (But if it's easy to do since we want them for ci
then sure)
>>>>>>>>>> 
>>>>>>>>>> -ash
>>>>>>>>>> 
>>>>>>>>>> On 11 June 2019 21:21:28 BST, Jarek Potiuk <
>>>>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Yeah Kamil - python 3.5 is the default one for
now. I think we
>>>>>>> should have
>>>>>>>>>>> another discussion here - how many versions to
support. There is
>>>>>> this
>>>>>>>>>>> ticket opened today :
>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4762 about
>>>>>>>>>>> supporting python 3.6 and 3.7 in tests. Anyone
has a strong
>>> opinion
>>>>>>> on
>>>>>>>>>>> this? I am for testing on all 3.5, 3.6 and 3.7
even if it
>>> increases
>>>>>>> the
>>>>>>>>>>> build/test time on Travis. There are a number
of differences
>>>>>> between
>>>>>>> those
>>>>>>>>>>> major versions (I have a blog post about it in
writing ) but I
>>>>>> think
>>>>>>> there
>>>>>>>>>>> is concern about eating Apache Travis time.
>>>>>>>>>>> 
>>>>>>>>>>> Anyone against those three ?
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Jun 11, 2019 at 8:38 PM Kamil Breguła
<
>>>>>>> kamil.bregula@polidea.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 1) I would prefer to use one repository.
>>>>>>>>>>>> +1
>>>>>>>>>>>> 
>>>>>>>>>>>> 2) The presented schema looks logical to
me. I had doubts
>> whether
>>>>>>>>>>>> Python 3.5 was a good choice for "latest"
version, but I
>> checked
>>>>>>> that
>>>>>>>>>>>> travis uses only this version.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jun 11, 2019 at 3:04 PM Jarek Potiuk
<
>>>>>>> Jarek.Potiuk@polidea.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We are close to finish AIP-10 (Airlfow
image for CI) and seems
>>>>>>> that we
>>>>>>>>>>>>> 
>>>>>>>>>>>> will
>>>>>>>>>>>> 
>>>>>>>>>>>>> start working soon on an official image
AIP, but in the
>> meantime
>>>>>>> we have
>>>>>>>>>>>>> 1.10.4 release coming and we would like
to agree tagging
>> scheme
>>>>>>> used for
>>>>>>>>>>>>> the current CI images. We discussed it
a bit on Slack, but
>> it's
>>>>>>> time to
>>>>>>>>>>>>> bring it here. I created a JIRA issue
for it:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-4764
 and my
>>>>>>> proposals
>>>>>>>>>>>>> 
>>>>>>>>>>>> after
>>>>>>>>>>>> 
>>>>>>>>>>>>> the initial discussion are those:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> First of all we have different images
that we can talk about :
>>>>>>>>>>>>> 
>>>>>>>>>>>>>  1. "base" one - with bare development-ready
airflow with
>>>>>>> minimum set
>>>>>>>>>>>>> 
>>>>>>>>>>>> of
>>>>>>>>>>>> 
>>>>>>>>>>>>>  dependencies
>>>>>>>>>>>>>  2. "CI" with all the tools packages
that are needed for CI
>>>>>>> tests
>>>>>>>>>>>>>  3. Soon we will likely have an "official"
one which might be
>>>>>>> used in
>>>>>>>>>>>>>  similar fashion as the "puckel" one.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There are two decisions to make:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 1) How to keep those images - in one
repository or whether we
>>>>>>> should have
>>>>>>>>>>>>> separate repos.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It is easier for now to keep all of them
within apache/airflow
>>>>>>>>>>>>> <
>>>>>>> https://cloud.docker.com/u/apache/repository/docker/apache/airflow>
>>>>>>>>>>>>> 
>>>>>>>>>>>> repository
>>>>>>>>>>>> 
>>>>>>>>>>>>> it seems and use a labelling scheme to
separate those (there
>> is
>>>>>>> nothing
>>>>>>>>>>>>> wrong with that but it might seem a bit
hacky). It's a bit
>>>>>> easier
>>>>>>> to
>>>>>>>>>>>>> maintain with access and CI.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We could also think about separate apache/airflow-ci,
>>>>>>> apache/airflow-dev,
>>>>>>>>>>>>> apache/airflow-prod or smth similar -
that would require some
>>>>>>>>>>>>> infrastructure tickets and is not very
common.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2) What labelling scheme to use(apache/airflow:label).
My
>>>>>>> proposal is
>>>>>>>>>>>>> similar to this (if we keep everything
in the airflow
>>>>>> repository)
>>>>>>>>>>>>> 
>>>>>>>>>>>>>  - *latest* = latest released version
(python 3.5)  = *
>>>>>>>>>>>>> 
>>>>>>>>>>>> v1.10.3-python3.5*
>>>>>>>>>>>> 
>>>>>>>>>>>>> - *master* = latest master version (python
3.5)  =
>>>>>>>>>>>>> 
>>>>>>>>>>>> *v2.0.0dev0-python3.5*
>>>>>>>>>>>> 
>>>>>>>>>>>>>  - *v1.10.3-python3.5,v1.10.3-python3.6*
 - released 1.10.3
>>>>>>> with python
>>>>>>>>>>>>>  3.5/3.6
>>>>>>>>>>>>>  - *latest-ci *= latest released version
of CI variant
>> (python
>>>>>>> 3.5)
>>>>>>>>>>>>>  *v1.10.3-ci-python3.5*
>>>>>>>>>>>>>  - *master-ci* = latest master version
of CI variant (python
>>>>>>> 3.5)
>>>>>>>>>>>>>  *v2.0.0dev0-ci-python3.5*
>>>>>>>>>>>>>  - *v1.10.3-ci-python3.5, v1.10.3-ci-python3.6*
- released
>>>>>>> 1.10.3 with
>>>>>>>>>>>>>  python 3.5/3.6
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> My preference is to keep all the images
in one repo and use
>>>>>>> labelling
>>>>>>>>>>>>> scheme as above,
>>>>>>>>>>>>> but I am open to discuss this.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> J,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jarek Potiuk
>>>>>>>>>>>>> Polidea <https://www.polidea.com/>
| Principal Software
>>>>>> Engineer
>>>>>>>>>>>>> 
>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> Jarek Potiuk
>>>>>>>>> Polidea <https://www.polidea.com/> | Principal
Software Engineer
>>>>>>>>> 
>>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> 
>>>>>>>> Jarek Potiuk
>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software
Engineer
>>>>>>>> 
>>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>> 
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>> 
>>> 
>> 
>> --
>> 
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>> 
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>> 


Mime
View raw message