airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Tagging of the airflow images
Date Mon, 17 Jun 2019 15:20:52 GMT
OK. The next build should prepare "master" tags for all the "master"
images: Would such revised labelling make sense?

V*ersions from master (development use only):*

   - Main non-CI images (small) *: airflow:master-v2.0.0dev0-python3.5,
   airflow:**master-v2.0.0dev0-python3.6, airflow:master*
   ==airflow:master-v2.0.0dev0-python3.5
   - CI images (big) *: airflow:master-v2.0.0dev0-ci-python3.5,
airflow:**master-v2.0.0dev0-ci-python3.6,
   airflow:master-ci*==airflow:master-v2.0.0dev0-ci-python3.5
   - Production optimised images: (future):
*airflow:master-v2.0.0dev0-prod-python3.5,
   airflow:**master-v2.0.0dev0-prod-python3.6, airflow:master-prod*
   ==airflow:master-v2.0.0dev0-prod-python3.5

*Release versions (future):*

   - Main non-CI images (small):  *airflow:v1.10.4-python3.5, *
   *airflow:v1.10.4**-python3.6, airflow:latest*==*airflow:v1.10.4 (should
   we have them ? I think it might be useful to have a reference image for
   tests)*
   - CI images (big): *airflow:v1.10.4-ci-python3.5,
**airflow:v1.10.4**-ci-python3.6,
   airflow:latest-ci*==*airflow:v1.10.4-ci **(should we have them ? I think
   it might be useful to have a reference image for tests)*
   - Production optimised images: *airflow:v1.10.4-prod-python3.5, *
   *airflow:v1.10.4**-prod-python3.6, airflow:latest-prod*==
   *airflow:v1.10.4-pro*


*J.*

On Mon, Jun 17, 2019 at 4:56 PM Jarek Potiuk <Jarek.Potiuk@polidea.com>
wrote:

> Sure. I agree "latest" might be misleading until we work out how we
> release it so I am fine with changing to master :). It's super easy -
> merely changing tags in DockerHub.
>
> On Mon, Jun 17, 2019 at 4:25 PM Ash Berlin-Taylor <ash@apache.org> wrote:
>
>> That page does mention "Nightly" builds which is close to what building
>> master would be. The other thing that matters is what we actual call A
>> Release.
>>
>> > Do not include any links on the project website that might encourage
>> non-developers to download and use nightly builds, snapshots, release
>> candidates, or any other similar package
>>
>> I think we're find so long as we don't do that -- or in this case, since
>> we will probably want to link to the docker hub page once we have versioned
>> images there if we make it clear that `:master` is not intended for end
>> users, and by the same argument if we have anything as `:latest` it should
>> be a docker image relating to an official Release.
>>
>> Jarek: no `latest` pointing at CI images please.
>>
>> -a
>>
>> > On 17 Jun 2019, at 15:04, Philippe Gagnon <philgagnon1@gmail.com>
>> wrote:
>> >
>> > One thing: we talked about releasing images under a "master" tag
>> (perhaps in another thread?), we should check if this is compatible with
>> Apache's release policy [1]. It's not clear to me if this is allowable or
>> not after a cursory reading.
>> >
>> > [1] http://www.apache.org/legal/release-policy.html#what
>> >
>> > On Mon, Jun 17, 2019 at 9:48 AM Jarek Potiuk <Jarek.Potiuk@polidea.com>
>> wrote:
>> > Anyone has more comments. I think prevailing opnion is:
>> > 1) To keep all images in one repo (apache/airflow)
>> > 2) I am not sure about labelling but I might try to document all cases
>> in a
>> > "production" image proposal that I would like to start as soon as we
>> merge
>> > the current CI image (which I think is quite close to finalisation).
>> >
>> > J.
>> >
>> > On Tue, Jun 11, 2019 at 10:59 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
>> >
>> > wrote:
>> >
>> > > It's super easy to do :)
>> > >
>> > > On Tue, Jun 11, 2019 at 10:33 PM Ash Berlin-Taylor <ash@apache.org>
>> wrote:
>> > >
>> > >> I'm fine with us just publishing release images using the newest
>> python
>> > >> release (i.e. 3.7) as the main reason we support older python
>> versions is
>> > >> to support distros thats ship those versions.(i.e. Deb stable), but
>> I don't
>> > >> think we need to support that in docker.
>> > >>
>> > >> (But if it's easy to do since we want them for ci then sure)
>> > >>
>> > >> -ash
>> > >>
>> > >> On 11 June 2019 21:21:28 BST, Jarek Potiuk <Jarek.Potiuk@polidea.com
>> >
>> > >> wrote:
>> > >>>
>> > >>> Yeah Kamil - python 3.5 is the default one for now. I think we
>> should have
>> > >>> another discussion here - how many versions to support. There is
>> this
>> > >>> ticket opened today :
>> https://issues.apache.org/jira/browse/AIRFLOW-4762 about
>> > >>> supporting python 3.6 and 3.7 in tests. Anyone has a strong opinion
>> on
>> > >>> this? I am for testing on all 3.5, 3.6 and 3.7 even if it increases
>> the
>> > >>> build/test time on Travis. There are a number of differences
>> between those
>> > >>> major versions (I have a blog post about it in writing ) but I
>> think there
>> > >>> is concern about eating Apache Travis time.
>> > >>>
>> > >>> Anyone against those three ?
>> > >>>
>> > >>> On Tue, Jun 11, 2019 at 8:38 PM Kamil BreguĊ‚a <
>> kamil.bregula@polidea.com>
>> > >>> wrote:
>> > >>>
>> > >>>  1) I would prefer to use one repository.
>> > >>>>  +1
>> > >>>>
>> > >>>>  2) The presented schema looks logical to me. I had doubts
whether
>> > >>>>  Python 3.5 was a good choice for "latest" version, but I checked
>> that
>> > >>>>  travis uses only this version.
>> > >>>>
>> > >>>>  On Tue, Jun 11, 2019 at 3:04 PM Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>
>> > >>>>  wrote:
>> > >>>>
>> > >>>>>
>> > >>>>>  Hello everyone,
>> > >>>>>
>> > >>>>>  We are close to finish AIP-10 (Airlfow image for CI) and
seems
>> that we
>> > >>>>>
>> > >>>> will
>> > >>>>
>> > >>>>> start working soon on an official image AIP, but in the
meantime
>> we have
>> > >>>>> 1.10.4 release coming and we would like to agree tagging
scheme
>> used for
>> > >>>>> the current CI images. We discussed it a bit on Slack,
but it's
>> time to
>> > >>>>> bring it here. I created a JIRA issue for it:
>> > >>>>> https://issues.apache.org/jira/browse/AIRFLOW-4764  and
my
>> proposals
>> > >>>>>
>> > >>>> after
>> > >>>>
>> > >>>>>  the initial discussion are those:
>> > >>>>>
>> > >>>>>  First of all we have different images that we can talk
about :
>> > >>>>>
>> > >>>>>     1. "base" one - with bare development-ready airflow
with
>> minimum set
>> > >>>>>
>> > >>>> of
>> > >>>>
>> > >>>>>     dependencies
>> > >>>>>     2. "CI" with all the tools packages that are needed
for CI
>> tests
>> > >>>>>     3. Soon we will likely have an "official" one which
might be
>> used in
>> > >>>>>     similar fashion as the "puckel" one.
>> > >>>>>
>> > >>>>>  There are two decisions to make:
>> > >>>>>
>> > >>>>>  1) How to keep those images - in one repository or whether
we
>> should have
>> > >>>>>  separate repos.
>> > >>>>>
>> > >>>>>  It is easier for now to keep all of them within apache/airflow
>> > >>>>>  <
>> https://cloud.docker.com/u/apache/repository/docker/apache/airflow>
>> > >>>>>
>> > >>>> repository
>> > >>>>
>> > >>>>>  it seems and use a labelling scheme to separate those
(there is
>> nothing
>> > >>>>>  wrong with that but it might seem a bit hacky). It's a
bit
>> easier to
>> > >>>>>  maintain with access and CI.
>> > >>>>>
>> > >>>>>  We could also think about separate apache/airflow-ci,
>> apache/airflow-dev,
>> > >>>>>  apache/airflow-prod or smth similar - that would require
some
>> > >>>>>  infrastructure tickets and is not very common.
>> > >>>>>
>> > >>>>>  2) What labelling scheme to use(apache/airflow:label).
My
>> proposal is
>> > >>>>>  similar to this (if we keep everything in the airflow
repository)
>> > >>>>>
>> > >>>>>     - *latest* = latest released version (python 3.5) 
= *
>> > >>>>>
>> > >>>> v1.10.3-python3.5*
>> > >>>>
>> > >>>>> - *master* = latest master version (python 3.5)  =
>> > >>>>>
>> > >>>> *v2.0.0dev0-python3.5*
>> > >>>>
>> > >>>>>     - *v1.10.3-python3.5,v1.10.3-python3.6*  - released
1.10.3
>> with python
>> > >>>>>     3.5/3.6
>> > >>>>>     - *latest-ci *= latest released version of CI variant
(python
>> 3.5)
>> > >>>>>     *v1.10.3-ci-python3.5*
>> > >>>>>     - *master-ci* = latest master version of CI variant
(python
>> 3.5)
>> > >>>>>     *v2.0.0dev0-ci-python3.5*
>> > >>>>>     - *v1.10.3-ci-python3.5, v1.10.3-ci-python3.6* - released
>> 1.10.3 with
>> > >>>>>     python 3.5/3.6
>> > >>>>>
>> > >>>>>
>> > >>>>>  My preference is to keep all the images in one repo and
use
>> labelling
>> > >>>>>  scheme as above,
>> > >>>>>  but I am open to discuss this.
>> > >>>>>
>> > >>>>>  J,
>> > >>>>>
>> > >>>>>
>> > >>>>>  --
>> > >>>>>
>> > >>>>>  Jarek Potiuk
>> > >>>>>  Polidea <https://www.polidea.com/> | Principal Software
Engineer
>> > >>>>>
>> > >>>>>  M: +48 660 796 129 <+48660796129>
>> > >>>>>  [image: Polidea] <https://www.polidea.com/>
>> > >>>>>
>> > >>>>
>> > >>>>
>> > >>>
>> > >
>> > > --
>> > >
>> > > Jarek Potiuk
>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > >
>> > > M: +48 660 796 129 <+48660796129>
>> > > [image: Polidea] <https://www.polidea.com/>
>> > >
>> > >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >
>> > M: +48 660 796 129 <+48660796129>
>> > [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message