airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Helm Chart for Airflow
Date Fri, 19 Jun 2020 14:26:25 GMT
Congrats! Looking forward to start using it for tests (I am going to use it
also for some performance testing soon).

J.


On Fri, Jun 19, 2020 at 4:09 PM Greg Neiheisel <greg@astronomer.io> wrote:

> Hey everyone,
>
> If you haven't been following along on the PR
> <https://github.com/apache/airflow/pull/8777>, it was recently merged! The
> helm chart now lives under the chart directory inside the Airflow repo.
>
> Thanks to everyone who's contributed over the years to this chart or any of
> the other open source charts. I think they all ended up pulling parts from
> each other. Happy to have a place where we can all work together on this
> together now.
>
> There's plenty of work to do still, so if you have any improvements, please
> send a PR!
>
>
>
> On Fri, May 8, 2020 at 8:10 AM Kaxil Naik <kaxilnaik@gmail.com> wrote:
>
> > Thanks, Greg 🚀
> >
> > On Fri, May 8, 2020 at 4:19 AM Greg Neiheisel <greg@astronomer.io.invalid
> >
> > wrote:
> >
> > > Awesome Jarek. I just opened up a draft PR that adds the chart into the
> > > Airflow repo. https://github.com/apache/airflow/pull/8777
> > >
> > > It still contains some of our packaging, testing and releasing
> scripts. I
> > > figure they can serve as a starting point to discuss how we'd want to
> > > replicate into Airflow infrastructure.
> > >
> > > Looking forward to getting some feedback and moving this along!
> > >
> > > On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > > > Hey Greg,
> > > >
> > > > I just wanted to update you with some recent changes I've done to
> > > automate
> > > > most of the image builds stuff we have in Airflow. That might be
> useful
> > > for
> > > > your efforts.
> > > >
> > > > I am adding (and I will fix some of them very soon) issues based on
> > > > feedback from people on the #prod-docker-image channel:
> > > >
> > >
> >
> https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
> > > > But in the meantime all the build/CI infrastructure has nicely fallen
> > > into
> > > > place and It's almost complete (one thing missing but very simple is
> > > > automated builds of release images - for now this is manual but it is
> > > super
> > > > easy change and I will add it before 1.10.11 for sure).
> > > >
> > > > I described the whole build infrastructure here:
> > > > https://github.com/apache/airflow/blob/master/CI.rst including how
> > image
> > > > caching works, how nightly builds and images are triggered. I hope it
> > > will
> > > > be helpful in the helm-chart work. Any questions or asks for the
> image
> > -
> > > > please let me know, it is super easy to update the image now when we
> > have
> > > > infrastructure to build it automatically integrated well with Github
> > > > Actions and DockerHub.
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > > > wrote:
> > > >
> > > >> Awesome, thanks for the update!
> > > >>
> > > >> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <greg@astronomer.io>
> > > >> wrote:
> > > >>
> > > >>> Hey everyone, just wanted to post a quick update on the helm chart.
> > > >>> We're in the process of removing any references to Astronomer
> > specific
> > > >>> things and making them more general features. That work is
> happening
> > on
> > > >>> this PR here https://github.com/astronomer/airflow-chart/pull/66.
> > > >>> Hoping to wrap up the last few outstanding items and open up a PR
> to
> > > >>> Airflow next week sometime!
> > > >>>
> > > >>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <
> > > >>> daniel.imberman@gmail.com> wrote:
> > > >>>
> > > >>>> Hi Jarek,
> > > >>>>
> > > >>>> Congrats on getting the image out! This is going to make a lot of
> > > >>>> things easier :). We’ve actually been working on separating out
> KinD
> > > as
> > > >>>> well (and in fact moving it out of travis and into GitHub issues).
> > > Let me
> > > >>>> circle back with Steven (the engineer who’s been working on that)
> > and
> > > get a
> > > >>>> status report from him.
> > > >>>>
> > > >>>> I will also take some time today to get our helm chart to work
> with
> > > >>>> the airflow master image. It’s been a bit crazy around here in the
> > > last few
> > > >>>> weeks but I am consistently pushing to make sure we make the final
> > > changes
> > > >>>> of ripping out any remaining platform components before we donate
> > :).
> > > >>>>
> > > >>>> I agree that docker hub automation will be great. This will also
> > allow
> > > >>>> us to have a nightly build for any integration/UI tests we build.
> > > >>>>
> > > >>>> Thanks again Jarek!
> > > >>>>
> > > >>>> Daniel
> > > >>>>
> > > >>>> via Newton Mail
> > > >>>> <
> > >
> >
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > > >
> > > >>>>
> > > >>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>> Hello Greg and Daniel,
> > > >>>>
> > > >>>> Production image is merged and it's also going to be released in
> > > >>>> 1.10.10. For now, I will manually tag the release in DockerHub
> when
> > we
> > > >>>> release but we will automate that in the future. Currently, we
> have
> > > >>>> *airflow:master* and *airflow:v1-10-test* images (for all
> supported
> > > >>>> python versions) that you can use for testing with the Helm chart!
> > > >>>>
> > > >>>> The next things from my side:
> > > >>>>
> > > >>>> * I will automate DockerHub building for the prod image from
> master
> > -
> > > >>>> for now only CI is automated but we need to add prod image as well
> > > >>>> * I will move kind-cluster creation outside of the container and
> use
> > > >>>> production image for our CI Kubernetes tests. I have a WIP branch
> on
> > > it and
> > > >>>> I think I should have it early next week
> > > >>>> * The next step will be to switch our CI to Github Actions
> finally.
> > I
> > > >>>> think we will run both Travis and GA for a short time to make sure
> > it
> > > works
> > > >>>> * I will ask people for options they would like to have in the
> > image -
> > > >>>> additional packages etc. so that I can work on the next iteration
> of
> > > the
> > > >>>> image. Also if you have any needs to see for the Helm
> integration, I
> > > am
> > > >>>> super happy to incorporate them in the next version. I plan to
> keep
> > > the
> > > >>>> "master" and "v1-10-test" images updated with any new changes and
> > push
> > > >>>> images regularly
> > > >>>> * I am happy to help with the Helm Chart, so let me know if you
> need
> > > >>>> anything!
> > > >>>>
> > > >>>> J.
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hello Daniel and Greg.
> > > >>>>>
> > > >>>>> I think the production image for Airflow is quite ready for the
> > final
> > > >>>>> review and merge. I did some tests and I think it's quite
> complete
> > > now.
> > > >>>>>
> > > >>>>> The PR is running it's Travis build:
> > > >>>>> https://github.com/apache/airflow/pull/7832
> > > >>>>>
> > > >>>>> I already pushed the image to DockerHub - so that the next time
> it
> > is
> > > >>>>> merged It will use it as a base and will be rather quickly built
> > > using
> > > >>>>> those images as a cache. Once it is merged, I will add the
> > automated
> > > build
> > > >>>>> in DockerHub and merge it to 1.10. As the next step I also plan
> to
> > > use the
> > > >>>>> prod image for our Kubernetes tests (that will help with moving
> to
> > > the
> > > >>>>> GitHub Actions from Travis).
> > > >>>>>
> > > >>>>> I added quite a comprehensive documentation - explaining all ins-
> > and
> > > >>>>> outs- of both CI and PROD images
> > > >>>>>
> > >
> >
> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
> > > >>>>> .
> > > >>>>>
> > > >>>>> You can build it yourself using manual build process described in
> > the
> > > >>>>> doc or (easier) using Breeze. That includes using the same
> > > Dockerfile to
> > > >>>>> build older Airflow version from the 1.10.* line.
> > > >>>>>
> > > >>>>> Feel free to use the new image in the Helm Chart you have - happy
> > to
> > > >>>>> review the PRs. The image is available at
> > > 'apache/airflow:master-python3.6'
> > > >>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10
> > and
> > > >>>>> release 1.10.10 we will also have it automatically published for
> > > 1.10.10
> > > >>>>> and beyond.
> > > >>>>>
> > > >>>>> Happy to hear your review comments and hope to merge it very
> soon.
> > > >>>>> J.
> > > >>>>>
> > > >>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Absolutely! Please do :). The more eyes on this the better!
> > > >>>>>>
> > > >>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
> > > >>>>>> daniel.imberman@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> @Jarek Steven and I will look into getting KinD to work on
> GitHub
> > > >>>>>>> actions today. If we can get that working, we can donate the
> helm
> > > chart and
> > > >>>>>>> possibly even deprecate the old Kubernetes tests in the
> process.
> > > >>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <
> > > >>>>>>> Jarek.Potiuk@polidea.com>, wrote:
> > > >>>>>>> > We have A POC for Github Actions done by Tomek, and the
> blocker
> > > >>>>>>> there
> > > >>>>>>> > was precisely Kubernetes tests and specifically running kind
> > > >>>>>>> cluster
> > > >>>>>>> > in the Github Actions
> > > >>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the
> > > >>>>>>> meantime
> > > >>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed
> this
> > > very
> > > >>>>>>> > thing today at slack:
> > > >>>>>>> >
> > > >>>>>>>
> > > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
> > > >>>>>>> >
> > > >>>>>>> > For me this is the next thing I will look at after merging
> > > >>>>>>> > requirements PR and Prod image PR (and those two are a bit
> > > >>>>>>> > pre-requisites to make it stable and fast). I want to switch
> to
> > > the
> > > >>>>>>> > Production Image to run Kubernetes tests (they will be much
> > > faster)
> > > >>>>>>> > and then we can work out much faster how to run them on
> Github
> > > >>>>>>> > Actions. Then we can add Helm charts to run all those
> different
> > > >>>>>>> > deployments. I am really looking forward to it!
> > > >>>>>>> >
> > > >>>>>>> > I hope - this weekend I move forward all of that and have a
> PR
> > > >>>>>>> that we
> > > >>>>>>> > can start running tests on and replacing the resources with
> > helm
> > > >>>>>>> > chart.
> > > >>>>>>> >
> > > >>>>>>> > J.
> > > >>>>>>> >
> > > >>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
> > > >>>>>>> > <daniel.imberman@gmail.com> wrote:
> > > >>>>>>> > >
> > > >>>>>>> > > @Steven glad to help you out on that.
> > > >>>>>>> > >
> > > >>>>>>> > > For the moment we only use travis, but I believe @jarek has
> > > been
> > > >>>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our
> > > first
> > > >>>>>>> GitHub tests?
> > > >>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
> > > >>>>>>> <steven@astronomer.io.invalid>, wrote:
> > > >>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working
> to
> > > >>>>>>> test it. Where
> > > >>>>>>> > > > do we want to run it? Do you all just use Travis only?
> > > >>>>>>> > > >
> > > >>>>>>> > > > Steven
> > > >>>>>>> > > >
> > > >>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
> > > >>>>>>> daniel.imberman@gmail.com>
> > > >>>>>>> > > > wrote:
> > > >>>>>>> > > >
> > > >>>>>>> > > > > @Jarek I think with the helm chart + prod image we can
> go
> > > >>>>>>> even further
> > > >>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
> > > >>>>>>> autoscaling, and a
> > > >>>>>>> > > > > bunch of other configurations.
> > > >>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
> > > >>>>>>> Jarek.Potiuk@polidea.com>,
> > > >>>>>>> > > > > wrote:
> > > >>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my
> original
> > > >>>>>>> email :)
> > > >>>>>>> > > > > >
> > > >>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> > > >>>>>>> > > > > daniel.imberman@gmail.com> wrote:
> > > >>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment.
> > We
> > > >>>>>>> just have
> > > >>>>>>> > > > > deployment files. If anything having the helm chart as
> a
> > > >>>>>>> part of the
> > > >>>>>>> > > > > airflow repo could mean that the helm chart becomes the
> > > >>>>>>> defacto system for
> > > >>>>>>> > > > > testing airflow on kubernetes (we can get rid of all
> the
> > > >>>>>>> yams files and run
> > > >>>>>>> > > > > multiple k8s tests with different settings).
> > > >>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> > > >>>>>>> > > > > <greg@astronomer.io.invalid>, wrote:
> > > >>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow
> > > >>>>>>> repo. We've also
> > > >>>>>>> > > > > been
> > > >>>>>>> > > > > > > > > building up a test suite that currently runs on
> > > >>>>>>> CircleCI and uses
> > > >>>>>>> > > > > kind
> > > >>>>>>> > > > > > > > > (Kubernetes in Docker) to test several
> kubernetes
> > > >>>>>>> versions with
> > > >>>>>>> > > > > some
> > > >>>>>>> > > > > > > > > different settings. Right now we're mostly
> > testing
> > > >>>>>>> the different
> > > >>>>>>> > > > > executors
> > > >>>>>>> > > > > > > > > since that has the biggest impact on what gets
> > > >>>>>>> deployed, but that
> > > >>>>>>> > > > > can be
> > > >>>>>>> > > > > > > > > expanded.
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > > What CRDs are currently being used to run
> Airflow
> > > >>>>>>> for the tests?
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> > > >>>>>>> > > > > Jarek.Potiuk@polidea.com>
> > > >>>>>>> > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > > > One thing for the donation.
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > Did you you want to have separate repository
> > > Greg ?
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > I think we should simply create a folder in
> > > >>>>>>> Airflow repo and
> > > >>>>>>> > > > > keep it
> > > >>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am
> > > >>>>>>> going to switch our
> > > >>>>>>> > > > > > > > > > Kubernetes Tests to the production image
> (will
> > > >>>>>>> make the tests
> > > >>>>>>> > > > > much
> > > >>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
> > > >>>>>>> automatically in
> > > >>>>>>> > > > > CI -
> > > >>>>>>> > > > > > > > > > for now we are using some custom Resource
> > > >>>>>>> definitions to start
> > > >>>>>>> > > > > Airflow
> > > >>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we
> > could
> > > >>>>>>> switch to
> > > >>>>>>> > > > > using the
> > > >>>>>>> > > > > > > > > > helm chart - this way we can test all three
> > > things
> > > >>>>>>> at once:
> > > >>>>>>> > > > > > > > > > - Kubernetes Executor
> > > >>>>>>> > > > > > > > > > - Dockerfile
> > > >>>>>>> > > > > > > > > > - Helm Chart
> > > >>>>>>> > > > > > > > > > and we could also add more tests - for
> example
> > > >>>>>>> testing different
> > > >>>>>>> > > > > > > > > > deployment options for the helm chart.
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would
> > help
> > > >>>>>>> with that -
> > > >>>>>>> > > > > > > > > > especially in terms of future changes and
> > testing
> > > >>>>>>> them
> > > >>>>>>> > > > > automatically.
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > J.
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal
> > Nurmamat
> > > >>>>>>> kyzy
> > > >>>>>>> > > > > > > > > > <aizhamal@apache.org> wrote:
> > > >>>>>>> > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see
> more
> > > >>>>>>> useful stuff for
> > > >>>>>>> > > > > the
> > > >>>>>>> > > > > > > > > > > community :)
> > > >>>>>>> > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg
> > Neiheisel
> > > <
> > > >>>>>>> > > > > greg@astronomer.io>
> > > >>>>>>> > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up
> now
> > as
> > > >>>>>>> an optional
> > > >>>>>>> > > > > Kubernetes
> > > >>>>>>> > > > > > > > > > > > CronJob (
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > >
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > > >>>>>>> > > > > > > > > > )
> > > >>>>>>> > > > > > > > > > > > that we have run periodically to clean
> > failed
> > > >>>>>>> pods up and
> > > >>>>>>> > > > > could stay
> > > >>>>>>> > > > > > > > > > > > separate.
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > The wait_for_migrations script could
> > > >>>>>>> definitely be pulled
> > > >>>>>>> > > > > into Airflow.
> > > >>>>>>> > > > > > > > > > > > For context, we deploy an initContainer
> on
> > > the
> > > >>>>>>> scheduler (
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > >
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > > >>>>>>> > > > > > > > > > )
> > > >>>>>>> > > > > > > > > > > > that runs the upgradedb command before
> > > booting
> > > >>>>>>> the
> > > >>>>>>> > > > > scheduler. This new
> > > >>>>>>> > > > > > > > > > > > wait_for_migration script runs in an
> > > >>>>>>> initContainer on the
> > > >>>>>>> > > > > webserver and
> > > >>>>>>> > > > > > > > > > > > workers (
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > >
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > > >>>>>>> > > > > > > > > > )
> > > >>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a
> > > >>>>>>> potentially
> > > >>>>>>> > > > > long-running
> > > >>>>>>> > > > > > > > > > migration
> > > >>>>>>> > > > > > > > > > > > and attempt to operate on new or missing
> > > >>>>>>> columns/tables
> > > >>>>>>> > > > > before the
> > > >>>>>>> > > > > > > > > > > > migrations run. This prevents these pods
> > from
> > > >>>>>>> entering a
> > > >>>>>>> > > > > CrashLoop.
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek
> > > Potiuk <
> > > >>>>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
> > > >>>>>>> > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images
> are
> > > >>>>>>> currently
> > > >>>>>>> > > > > generated from
> > > >>>>>>> > > > > > > > > > > > > Dockerfiles
> > > >>>>>>> > > > > > > > > > > > > > in this repo
> > > >>>>>>> https://github.com/astronomer/ap-airflow
> > > >>>>>>> > > > > and get
> > > >>>>>>> > > > > > > > > > > > > published to
> > > >>>>>>> > > > > > > > > > > > > > DockerHub
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > For the most part those are typical
> > > >>>>>>> Airflow images.
> > > >>>>>>> > > > > There's an
> > > >>>>>>> > > > > > > > > > > > > entrypoint
> > > >>>>>>> > > > > > > > > > > > > > script that we include in the image
> > that
> > > >>>>>>> handles waiting
> > > >>>>>>> > > > > for the
> > > >>>>>>> > > > > > > > > > > > > database
> > > >>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which
> > is
> > > >>>>>>> pretty generic.
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > I already added waiting for the
> database
> > > >>>>>>> (both metadata
> > > >>>>>>> > > > > and celery
> > > >>>>>>> > > > > > > > > > URL) in
> > > >>>>>>> > > > > > > > > > > > > the PR:
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > >
> >
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > >>>>>>> > > > > > > > > > > > > .
> > > >>>>>>> > > > > > > > > > > > > It's functionally the same but more
> > > generic.
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > The only other
> > > >>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart
> uses
> > > >>>>>>> would be the
> > > >>>>>>> > > > > scripts in this
> > > >>>>>>> > > > > > > > > > repo
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> https://github.com/astronomer/astronomer-airflow-scripts.
> > > >>>>>>> > > > > Our
> > > >>>>>>> > > > > > > > > > > > > Dockerfiles
> > > >>>>>>> > > > > > > > > > > > > > pull this package in. These scripts
> are
> > > >>>>>>> used to
> > > >>>>>>> > > > > coordinate running
> > > >>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed
> pods.
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > I see two scripts:
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe)
> not
> > > >>>>>>> needed to run in
> > > >>>>>>> > > > > airflow -
> > > >>>>>>> > > > > > > > > > this
> > > >>>>>>> > > > > > > > > > > > > could be run as a separate
> pod/container?
> > > >>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think
> this
> > is
> > > >>>>>>> a good
> > > >>>>>>> > > > > candidate to add
> > > >>>>>>> > > > > > > > > > > > > *airflow
> > > >>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make
> > it
> > > >>>>>>> part of airflow
> > > >>>>>>> > > > > itself.
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > I think we also have to agree on the
> > > Airflow
> > > >>>>>>> version
> > > >>>>>>> > > > > supported by the
> > > >>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we
> > support
> > > >>>>>>> 1.10.10+ and we
> > > >>>>>>> > > > > > > > > > incorporate
> > > >>>>>>> > > > > > > > > > > > > all
> > > >>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the
> > "db
> > > >>>>>>> > > > > wait_for_migration")
> > > >>>>>>> > > > > > > > > > into 2.0
> > > >>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image
> and
> > > >>>>>>> helm chart for
> > > >>>>>>> > > > > those versions
> > > >>>>>>> > > > > > > > > > > > > only. That would help with people
> > migrating
> > > >>>>>>> to the latest
> > > >>>>>>> > > > > version.
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > WDYT?
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM
> Daniel
> > > >>>>>>> Imberman <
> > > >>>>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
> > > >>>>>>> > > > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think
> > that
> > > >>>>>>> pairing an
> > > >>>>>>> > > > > official helm
> > > >>>>>>> > > > > > > > > > chart
> > > >>>>>>> > > > > > > > > > > > > > > with the official image would make
> > for
> > > a
> > > >>>>>>> REALLY
> > > >>>>>>> > > > > powerful “up and
> > > >>>>>>> > > > > > > > > > > > > running
> > > >>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I
> > > have
> > > >>>>>>> also been
> > > >>>>>>> > > > > looking into
> > > >>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability
> to
> > > >>>>>>> create custom
> > > >>>>>>> > > > > controllers
> > > >>>>>>> > > > > > > > > > from
> > > >>>>>>> > > > > > > > > > > > > helm
> > > >>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a
> > 1-2
> > > >>>>>>> punch from the
> > > >>>>>>> > > > > same code
> > > >>>>>>> > > > > > > > > > base
> > > >>>>>>> > > > > > > > > > > > > :).
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if
> > there’s
> > > >>>>>>> no issues, can
> > > >>>>>>> > > > > we please
> > > >>>>>>> > > > > > > > > > start
> > > >>>>>>> > > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > process of donation?
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > Daniel
> > > >>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700,
> Jarek
> > > >>>>>>> Potiuk <
> > > >>>>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
> > > >>>>>>> > > > > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with
> > the
> > > >>>>>>> official image
> > > >>>>>>> > > > > we have
> > > >>>>>>> > > > > > > > > > work in
> > > >>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at
> the
> > > >>>>>>> Astronomer's
> > > >>>>>>> > > > > image while
> > > >>>>>>> > > > > > > > > > > > > preparing
> > > >>>>>>> > > > > > > > > > > > > > my
> > > >>>>>>> > > > > > > > > > > > > > > > draft and we can make any
> > adjustments
> > > >>>>>>> needed to make
> > > >>>>>>> > > > > it works
> > > >>>>>>> > > > > > > > > > with
> > > >>>>>>> > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > helm
> > > >>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
> > > >>>>>>> collaborate on that.
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > PR here:
> > > >>>>>>> https://github.com/apache/airflow/pull/7832
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > J.
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM
> > Kaxil
> > > >>>>>>> Naik <
> > > >>>>>>> > > > > kaxilnaik@gmail.com
> > > >>>>>>> > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
> > > >>>>>>> tomasz.urbaszek@polidea.com> :
> > > >>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
> > > >>>>>>> > > > > https://github.com/astronomer/airflow-chart
> > > >>>>>>> > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM
> > > >>>>>>> Tomasz Urbaszek <
> > > >>>>>>> > > > > > > > > > > > > > turbaszek@apache.org>
> > > >>>>>>> > > > > > > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is
> > > >>>>>>> something our
> > > >>>>>>> > > > > community needs!
> > > >>>>>>> > > > > > > > > > Using
> > > >>>>>>> > > > > > > > > > > > > your
> > > >>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a
> > lot
> > > >>>>>>> of sens to me
> > > >>>>>>> > > > > because as
> > > >>>>>>> > > > > > > > > > you
> > > >>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle
> tested.
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow
> > image
> > > >>>>>>> do you use?
> > > >>>>>>> > > > > Also, would you
> > > >>>>>>> > > > > > > > > > > > > mind
> > > >>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > Tomek
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07
> PM
> > > >>>>>>> Greg Neiheisel
> > > >>>>>>> > > > > > > > > > > > > > > > > > <greg@astronomer.io.invalid>
> > > >>>>>>> wrote:
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
> > > >>>>>>> Astronomer, we’ve
> > > >>>>>>> > > > > created,
> > > >>>>>>> > > > > > > > > > managed,
> > > >>>>>>> > > > > > > > > > > > > > and
> > > >>>>>>> > > > > > > > > > > > > > > > > > hardened
> > > >>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm
> Chart
> > > >>>>>>> for Airflow (
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> https://github.com/astronomer/airflow-chart)
> > > >>>>>>> > > > > that is
> > > >>>>>>> > > > > > > > > > being
> > > >>>>>>> > > > > > > > > > > > > used
> > > >>>>>>> > > > > > > > > > > > > > by
> > > >>>>>>> > > > > > > > > > > > > > > > > both
> > > >>>>>>> > > > > > > > > > > > > > > > > > our
> > > >>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise
> > customers.
> > > >>>>>>> This chart is
> > > >>>>>>> > > > > > > > > > battle-tested and
> > > >>>>>>> > > > > > > > > > > > > > > running
> > > >>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow
> > deployments
> > > >>>>>>> of varying
> > > >>>>>>> > > > > sizes and
> > > >>>>>>> > > > > > > > > > runtime
> > > >>>>>>> > > > > > > > > > > > > > > > > > environments.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to
> > > >>>>>>> encapsulate the issues
> > > >>>>>>> > > > > that Airflow
> > > >>>>>>> > > > > > > > > > > > > users
> > > >>>>>>> > > > > > > > > > > > > > run
> > > >>>>>>> > > > > > > > > > > > > > > > > into
> > > >>>>>>> > > > > > > > > > > > > > > > > > in
> > > >>>>>>> > > > > > > > > > > > > > > > > > > the real world.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > While this chart was
> > originally
> > > >>>>>>> developed
> > > >>>>>>> > > > > internally for
> > > >>>>>>> > > > > > > > > > our
> > > >>>>>>> > > > > > > > > > > > > > > Astronomer
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently
> > > >>>>>>> decoupled the chart
> > > >>>>>>> > > > > from the
> > > >>>>>>> > > > > > > > > > rest of
> > > >>>>>>> > > > > > > > > > > > > our
> > > >>>>>>> > > > > > > > > > > > > > > > > > platform
> > > >>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the
> > > greater
> > > >>>>>>> Airflow
> > > >>>>>>> > > > > community. With
> > > >>>>>>> > > > > > > > > > these
> > > >>>>>>> > > > > > > > > > > > > > > changes
> > > >>>>>>> > > > > > > > > > > > > > > > > in
> > > >>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
> > > >>>>>>> conversation about
> > > >>>>>>> > > > > donating this
> > > >>>>>>> > > > > > > > > > > > > chart
> > > >>>>>>> > > > > > > > > > > > > > to
> > > >>>>>>> > > > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Airflow community.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features
> of
> > > the
> > > >>>>>>> chart are:
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box.
> > With
> > > >>>>>>> zero
> > > >>>>>>> > > > > configuration, a user
> > > >>>>>>> > > > > > > > > > > > > will
> > > >>>>>>> > > > > > > > > > > > > > get
> > > >>>>>>> > > > > > > > > > > > > > > > > a
> > > >>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a
> default
> > > >>>>>>> user and the
> > > >>>>>>> > > > > > > > > > KubernetesExecutor
> > > >>>>>>> > > > > > > > > > > > > > ready
> > > >>>>>>> > > > > > > > > > > > > > > > > to
> > > >>>>>>> > > > > > > > > > > > > > > > > > run
> > > >>>>>>> > > > > > > > > > > > > > > > > > > DAGs.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery
> > (w/
> > > >>>>>>> optional KEDA
> > > >>>>>>> > > > > > > > > > autoscaling) and
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Support for optional
> > pgbouncer.
> > > >>>>>>> We use this to
> > > >>>>>>> > > > > share a
> > > >>>>>>> > > > > > > > > > > > > > configurable
> > > >>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per
> > > >>>>>>> deployment. Useful
> > > >>>>>>> > > > > for limiting
> > > >>>>>>> > > > > > > > > > > > > > > connections to
> > > >>>>>>> > > > > > > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > > > > > metadata database.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration
> support.
> > A
> > > >>>>>>> user can push a
> > > >>>>>>> > > > > newer
> > > >>>>>>> > > > > > > > > > version
> > > >>>>>>> > > > > > > > > > > > > of
> > > >>>>>>> > > > > > > > > > > > > > > > > > Airflow
> > > >>>>>>> > > > > > > > > > > > > > > > > > > into an existing release
> and
> > > >>>>>>> migrations will
> > > >>>>>>> > > > > > > > > > automatically run
> > > >>>>>>> > > > > > > > > > > > > > > > > > cleanly.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support.
> > > Optionally
> > > >>>>>>> install and
> > > >>>>>>> > > > > configure a
> > > >>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter
> > > >>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics
> and
> > > >>>>>>> expose them to
> > > >>>>>>> > > > > Prometheus
> > > >>>>>>> > > > > > > > > > > > > > > > > automatically.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Resource control.
> > Optionally
> > > >>>>>>> control the
> > > >>>>>>> > > > > ResourceQuotas
> > > >>>>>>> > > > > > > > > > and
> > > >>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each
> > deployment
> > > >>>>>>> so that no
> > > >>>>>>> > > > > deployment can
> > > >>>>>>> > > > > > > > > > > > > > overload
> > > >>>>>>> > > > > > > > > > > > > > > a
> > > >>>>>>> > > > > > > > > > > > > > > > > > > cluster.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional
> > Elasticsearch
> > > >>>>>>> support.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace
> cleanup.
> > > >>>>>>> Sometimes
> > > >>>>>>> > > > > > > > > > KubernetesExecutor and
> > > >>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods
> > fail
> > > >>>>>>> for reasons
> > > >>>>>>> > > > > other than the
> > > >>>>>>> > > > > > > > > > > > > actual
> > > >>>>>>> > > > > > > > > > > > > > > > > > task.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep
> > things
> > > >>>>>>> clean in
> > > >>>>>>> > > > > Kubernetes.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Support for running
> locally
> > > in
> > > >>>>>>> KIND
> > > >>>>>>> > > > > (Kubernetes in
> > > >>>>>>> > > > > > > > > > Docker).
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested
> across
> > > >>>>>>> many Kubernetes
> > > >>>>>>> > > > > versions
> > > >>>>>>> > > > > > > > > > with
> > > >>>>>>> > > > > > > > > > > > > Helm
> > > >>>>>>> > > > > > > > > > > > > > 2
> > > >>>>>>> > > > > > > > > > > > > > > > > > and 3
> > > >>>>>>> > > > > > > > > > > > > > > > > > > support.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the
> cleanest
> > > >>>>>>> and most
> > > >>>>>>> > > > > reliable way to
> > > >>>>>>> > > > > > > > > > deploy
> > > >>>>>>> > > > > > > > > > > > > > DAGs
> > > >>>>>>> > > > > > > > > > > > > > > to
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them
> at
> > > >>>>>>> scale is to
> > > >>>>>>> > > > > package them
> > > >>>>>>> > > > > > > > > > into
> > > >>>>>>> > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > actual
> > > >>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have
> > geared
> > > >>>>>>> this chart
> > > >>>>>>> > > > > towards that
> > > >>>>>>> > > > > > > > > > > > > method of
> > > >>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding
> > other
> > > >>>>>>> methods should
> > > >>>>>>> > > > > be
> > > >>>>>>> > > > > > > > > > > > > straightforward.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from
> > the
> > > >>>>>>> community and
> > > >>>>>>> > > > > would love
> > > >>>>>>> > > > > > > > > > to
> > > >>>>>>> > > > > > > > > > > > > see
> > > >>>>>>> > > > > > > > > > > > > > > this
> > > >>>>>>> > > > > > > > > > > > > > > > > > chart
> > > >>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and
> > > >>>>>>> running on
> > > >>>>>>> > > > > Kubernetes!
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief
> > > >>>>>>> Architect
> > > >>>>>>> > > > > Astronomer.io
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk
> > > >>>>>>> > > > > > > > > > > > > > > > Polidea <
> https://www.polidea.com/>
> > |
> > > >>>>>>> Principal
> > > >>>>>>> > > > > Software
> > > >>>>>>> > > > > > > > > > Engineer
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129
> > > >>>>>>> <+48%20660%20796%20129>
> > > >>>>>>> > > > > <+48660796129
> > > >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
> > > >>>>>>> > > > > > > > > > > > > > > > [image: Polidea] <
> > > >>>>>>> https://www.polidea.com/>
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > > >>>>>>> Astronomer.io
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > Jarek Potiuk
> > > >>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
> > > >>>>>>> Principal Software
> > > >>>>>>> > > > > Engineer
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > M: +48 660 796 129
> > <+48%20660%20796%20129>
> > > >>>>>>> <+48660796129
> > > >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
> > > >>>>>>> > > > > > > > > > > > > [image: Polidea] <
> > https://www.polidea.com/
> > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > > >>>>>>> Astronomer.io
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > --
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > Jarek Potiuk
> > > >>>>>>> > > > > > > > > > Polidea | Principal Software Engineer
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > M: +48 660 796 129
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > > --
> > > >>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect
> Astronomer.io
> > > >>>>>>> > > > > >
> > > >>>>>>> > > > > >
> > > >>>>>>> > > > > > --
> > > >>>>>>> > > > > > Jarek Potiuk
> > > >>>>>>> > > > > > Polidea | Principal Software Engineer
> > > >>>>>>> > > > > > M: +48 660 796 129
> > > >>>>>>> > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>> >
> > > >>>>>>> >
> > > >>>>>>> >
> > > >>>>>>> > --
> > > >>>>>>> >
> > > >>>>>>> > Jarek Potiuk
> > > >>>>>>> > Polidea | Principal Software Engineer
> > > >>>>>>> >
> > > >>>>>>> > M: +48 660 796 129
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>>
> > > >>>>>> Jarek Potiuk
> > > >>>>>> Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > > >>>>>>
> > > >>>>>> M: +48 660 796 129 <+48660796129>
> > > >>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>>
> > > >>>>> Jarek Potiuk
> > > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>>>>
> > > >>>>> M: +48 660 796 129 <+48660796129>
> > > >>>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>>
> > > >>>> Jarek Potiuk
> > > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>>>
> > > >>>> M: +48 660 796 129 <+48660796129>
> > > >>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>> --
> > > >>> *Greg Neiheisel* / Chief Architect Astronomer.io
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Jarek Potiuk
> > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>
> > > >> M: +48 660 796 129 <+48660796129>
> > > >> [image: Polidea] <https://www.polidea.com/>
> > > >>
> > > >>
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > > >
> > >
> > > --
> > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > >
> >
>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message