airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Travis builds in a queue for hours
Date Fri, 12 Jul 2019 19:47:49 GMT
Just FYI - I opened a ticket to get stats of the machine usage for Travis
to infra: https://issues.apache.org/jira/browse/INFRA-18742

On Thu, Jul 11, 2019 at 1:48 PM Jarek Potiuk <Jarek.Potiuk@polidea.com>
wrote:

> Absolutely! I thought about it today and GKE cluster would be perfect for
> us - especially that we can also use it to run Kubernetes tests on it !
> That's still a major pain having to setup minikube for the tests and having
> a GKE cluster that we can simply use would simplify this part a LOT.
>
> Principal Software Engineer
> Phone: +48660796129
>
> czw., 11 lip 2019, 09:26 użytkownik Driesprong, Fokko <fokko@driesprong.frl>
> napisał:
>
>> Yes, Gitlab works very well with GCP. A Kubernetes cluster with
>> autoscaling
>> for the runners would be perfect, and will also minimize the resources
>> provided by Google.
>>
>> Cheers, Fokko
>>
>> Op do 11 jul. 2019 om 07:13 schreef Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>
>>
>> > Since more than few people (including myself) are in favour of GitLab
>> CI,
>> > and since Apache Infra is talking to GitLab CI, I will make sure to
>> check
>> > if we can combine the two approaches - workers from Google and managed,
>> > central GitlabCI interface to manage it (likely managed by the Infra
>> team).
>> > Airflow can easily be a  "guinea pig" for GitLab CI / Apache
>> integration.
>> > We also have quite an expertise in managin GitLab in my company (we use
>> > GitLab in Polidea for most of our mobile project CI and all the cloud
>> > builds that we run internally).
>> >
>> > I will make an AIP for that soon and involve the right people :).
>> >
>> > J.
>> >
>> > On Thu, Jul 11, 2019 at 8:01 AM Driesprong, Fokko <fokko@driesprong.frl
>> >
>> > wrote:
>> >
>> > > Regardings the numbers, I believe that INFRA has an overview of the
>> usage
>> > > per project. I think they are happy to share these numbers with you.
>> > Also,
>> > > it seems like there is also a queue in Jenkins:
>> > https://status.apache.org/
>> > >
>> > > Talking about Jenkins. I'm not a big fan of it. For example, Spark
>> uses
>> > it,
>> > > and it is rather difficult to set up the environment yourself, in
>> > contrast
>> > > with Travis. I also have good experiences with Gitlab since that is
>> the
>> > > only Docker native CI in my personal opinion.
>> > >
>> > > > But we can try both of course. And even switch later.
>> > > There is nothing as permanent as a temporary solution :-) However, I'm
>> > not
>> > > against trying. I've checked the beam project, and the integration
>> with
>> > > Github looks good.
>> > >
>> > > Thanks again Jarek and Aizhamal for all the work an effort.
>> > >
>> > > Cheers, Fokko
>> > >
>> > >
>> > >
>> > >
>> > > Op wo 10 jul. 2019 om 23:11 schreef Aizhamal Nurmamat kyzy <
>> > > aizhamal@apache.org>:
>> > >
>> > > > Hi all,
>> > > >
>> > > > I am still working on trying to get approvals for this, so this is
>> not
>> > > yet
>> > > > a done deal. I'll keep y'all updated.
>> > > >
>> > > > As for the CI solution to use, we have no particular inclination.
As
>> > long
>> > > > as the community supports it, and it is consistent with any Apache
>> > > > guidelines for CI from their projects. Jenkins and GitLab CI both
>> sound
>> > > > sensible.
>> > > >
>> > > > The email from INFRA says that Airflow runs 2600 hours of tests per
>> > > month,
>> > > > or the equivalent of about 4 machines. Can the community help with
a
>> > > > reasonable estimate for this, so I can use it as a reference for the
>> > > > request?
>> > > >
>> > > > Thanks!
>> > > >
>> > > > On Wed, Jul 10, 2019 at 2:43 PM Jarek Potiuk <
>> Jarek.Potiuk@polidea.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Yeah. Gitlab CI is definitely what I would prefer as well from
the
>> > > > > "modernity" point of view (and one of my very close friends is
>> Gitlab
>> > > CI
>> > > > > maintainer and actually The person who introduced CI to GitLab
>> > > > offering). I
>> > > > > also actually already catalysed discussion between GitLab and
>> Apache
>> > > > > infrastructure to introduce GitLab CI on the "Apache" level (they
>> are
>> > > > > talking about it now I believe).
>> > > > >
>> > > > > But from Google <> Apache/Procedural point of view it might
>> simply be
>> > > > > easier to follow footsteps of Apache Beam. It might simply be
few
>> > > clicks
>> > > > > away for the Apache Infrastructure to add more machines and
>> connect
>> > > them
>> > > > to
>> > > > > the Apache Jenkins for our project. If we have a path cleared
by
>> > > others,
>> > > > > following it might be simply much faster.
>> > > > >
>> > > > > But we can try both of course. And even switch later. The Docker
>> CI
>> > > > > approach I am about to merge is designed to be super-easy to
>> switch
>> > > > betwen
>> > > > > CI systems. Virtually ALL the build logic is in scripts  in shared
>> > > Docker
>> > > > > images. There is basically one file per CI system to add and
we
>> can
>> > > > support
>> > > > > Travis/Jenkins/CloudBuild/CircleCI - whatever we imaging. We
can
>> even
>> > > > > support all of them at the same time :)
>> > > > >
>> > > > > J.
>> > > > >
>> > > > > On Wed, Jul 10, 2019 at 11:32 PM Bolke de Bruin <
>> bdbruin@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > If you need an alternative why not use a couple of gitlab-ci
>> > runners?
>> > > > > Much
>> > > > > > easier to maintain, light weight, and much closer to what
we use
>> > now.
>> > > > > >
>> > > > > > B.
>> > > > > >
>> > > > > > Verstuurd vanaf mijn iPad
>> > > > > >
>> > > > > > > Op 10 jul. 2019 om 23:27 heeft Bolke de Bruin <
>> bdbruin@gmail.com
>> > >
>> > > > het
>> > > > > > volgende geschreven:
>> > > > > > >
>> > > > > > > Awesome! But I hope you are not serious about using
Jenkins
>> > right?
>> > > > If I
>> > > > > > need to start a Holy War it would be against Jenkins.
>> > > > > > >
>> > > > > > > B.
>> > > > > > >
>> > > > > > > Verstuurd vanaf mijn iPad
>> > > > > > >
>> > > > > > >> Op 10 jul. 2019 om 22:55 heeft Jarek Potiuk <
>> > > > Jarek.Potiuk@polidea.com
>> > > > > >
>> > > > > > het volgende geschreven:
>> > > > > > >>
>> > > > > > >> Hello Everyone,
>> > > > > > >>
>> > > > > > >> I have some really good news. I just had a call
with Google
>> OSS
>> > > team
>> > > > > > (Gris,
>> > > > > > >> Aizhamal) and they are willing to donate VMs on
Google Cloud
>> > > > Platform
>> > > > > to
>> > > > > > >> run CI for Airflow. In order to simplify the setup
(and make
>> > sure
>> > > it
>> > > > > is
>> > > > > > ok
>> > > > > > >> according to Apache regulations) we think we should
go
>> exactly
>> > the
>> > > > > same
>> > > > > > >> route as Apache Beam project (Google donated 16x
16CPU
>> machines
>> > > for
>> > > > > > them).
>> > > > > > >> The route of Apache Beam is to use the machines
as workers
>> for
>> > > > Apache
>> > > > > > >> Jenkins (https://builds.apache.org/). Apache Jenkins
is one
>> of
>> > > the
>> > > > > > >> encouraged CI solutions by Apache and if we can
have workers
>> > > > connected
>> > > > > > to
>> > > > > > >> the existing Jenkins master of Apache, it means
that the
>> > > maintenance
>> > > > > > >> overhead will be pretty minimal. And we can follow
Apache
>> Beam
>> > > setup
>> > > > > so
>> > > > > > I
>> > > > > > >> do not expect any legal problems.
>> > > > > > >>
>> > > > > > >> I also work very closely with the team that uses
Apache Beam
>> > > Jenkins
>> > > > > > >> heavily so I have access to all the necessary experts
to help
>> > with
>> > > > the
>> > > > > > >> setup (and I am happy to help with that).
>> > > > > > >>
>> > > > > > >> I really hope everyone in the community will be
really happy
>> to
>> > go
>> > > > in
>> > > > > > that
>> > > > > > >> direction - it's. Please let me know if you have
any
>> concerns !
>> > > > > > >>
>> > > > > > >> We do not need as many machines as Beam for sure
(Beam uses
>> the
>> > > > > > machines to
>> > > > > > >> process a lot of data for tests including some
load testing)
>> but
>> > > we
>> > > > > > need to
>> > > > > > >> estimate the number/types of machines that we are
going to
>> need.
>> > > > > > >> Fokko, Ash, others - do you have some recent numbers
for the
>> > > current
>> > > > > > usage
>> > > > > > >> or should I open an Infrastructure ticket for it?
>> > > > > > >>
>> > > > > > >> J
>> > > > > > >>
>> > > > > > >> On Fri, Jun 28, 2019 at 4:50 PM Jarek Potiuk <
>> > > > > Jarek.Potiuk@polidea.com>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >>> Thanks Aizhamal! I spoke already to Gris and
she confirmed
>> that
>> > > as
>> > > > > well
>> > > > > > >>> and the 8th of July date is ok for us as we
will have to
>> > evaluate
>> > > > and
>> > > > > > >>> prepare as well. Have a nice trip.
>> > > > > > >>>
>> > > > > > >>> J.
>> > > > > > >>>
>> > > > > > >>> On Fri, Jun 28, 2019 at 4:25 PM Aizhamal Nurmamat
kyzy
>> > > > > > >>> <aizhamal@google.com.invalid> wrote:
>> > > > > > >>>
>> > > > > > >>>> Hi all,
>> > > > > > >>>>
>> > > > > > >>>> On Thu, Jun 27, 2019 at 15:28 Jarek Potiuk
<
>> > > > > Jarek.Potiuk@polidea.com>
>> > > > > > >>>> wrote:
>> > > > > > >>>>
>> > > > > > >>>>> Yeah. I also have a working version
of Cloud build
>> > > configuration
>> > > > > and
>> > > > > > we
>> > > > > > >>>> can
>> > > > > > >>>>> run the tests on cloud build if we
can get some credits
>> from
>> > > > > Google.
>> > > > > > >>>>
>> > > > > > >>>>
>> > > > > > >>>> I can look into getting a small amount
of credits approved
>> for
>> > > > this,
>> > > > > > to
>> > > > > > >>>> see
>> > > > > > >>>> if it’s useful to offload some tests
to Cloud Build, or to
>> > > > provision
>> > > > > > some
>> > > > > > >>>> VMs to run on Apache Infra.
>> > > > > > >>>>
>> > > > > > >>>> I am traveling at the moment, but I’ll
be back in the
>> office
>> > on
>> > > > July
>> > > > > > 8,
>> > > > > > >>>> and
>> > > > > > >>>> I’ll try to get this done.
>> > > > > > >>>>
>> > > > > > >>>>
>> > > > > > >>>> Thanks,
>> > > > > > >>>> Aizhamal
>> > > > > > >>>>
>> > > > > > >>>> And
>> > > > > > >>>>> the changes from the upcoming CI image
will make it much
>> > easier
>> > > > to
>> > > > > > run
>> > > > > > >>>>> tests on any CI provider. Except Kubernetes
tests they are
>> > > pretty
>> > > > > > much
>> > > > > > >>>>> CI-agnostic. Kubernetes tests will
likely be also fixed
>> soon.
>> > > > > > >>>>>
>> > > > > > >>>>> Another idea: I thought that in the
future we can also run
>> > only
>> > > > > > subset
>> > > > > > >>>> of
>> > > > > > >>>>> postgres/mysql/sqlite tests on all
combinations. I think
>> > there
>> > > > are
>> > > > > > just
>> > > > > > >>>>> handful of tests that are specific
for backend (and we
>> > already
>> > > > know
>> > > > > > >>>> which
>> > > > > > >>>>> ones they are - they are skipped-if).
>> > > > > > >>>>>
>> > > > > > >>>>> J.
>> > > > > > >>>>>
>> > > > > > >>>>> Principal Software Engineer
>> > > > > > >>>>> Phone: +48660796129
>> > > > > > >>>>>
>> > > > > > >>>>> czw., 27 cze 2019, 15:12 użytkownik
Philippe Gagnon <
>> > > > > > >>>> philgagnon1@gmail.com
>> > > > > > >>>>>>
>> > > > > > >>>>> napisał:
>> > > > > > >>>>>
>> > > > > > >>>>>> I think the combinations that you
are proposing are
>> sensible
>> > > for
>> > > > > > >>>>> pre-merge
>> > > > > > >>>>>> checks.
>> > > > > > >>>>>>
>> > > > > > >>>>>> I am working on a proposal to offload
extra combinations
>> to
>> > > > > another
>> > > > > > CI
>> > > > > > >>>>>> provider (Azure DevOps specifically
seems like a good
>> > > > candidate),
>> > > > > > >>>> either
>> > > > > > >>>>>> pre or post merge. Ideally I'd
like to run more
>> combinations
>> > > > > > pre-merge
>> > > > > > >>>>> but
>> > > > > > >>>>>> there is a trade-off to be conscious
of here between
>> > > development
>> > > > > > >>>> velocity
>> > > > > > >>>>>> and quality assurance, which I
think this issue
>> highlights
>> > > quite
>> > > > > > well.
>> > > > > > >>>>>>
>> > > > > > >>>>>> Please let me know your thoughts
>> > > > > > >>>>>>
>> > > > > > >>>>>> Philippe
>> > > > > > >>>>>>
>> > > > > > >>>>>> On Thu, Jun 27, 2019 at 9:05 AM
Jarek Potiuk <
>> > > > > > >>>> Jarek.Potiuk@polidea.com>
>> > > > > > >>>>>> wrote:
>> > > > > > >>>>>>
>> > > > > > >>>>>>> Agree that we should be thoughtful
about others as
>> well: In
>> > > the
>> > > > > > >>>> latest
>> > > > > > >>>>>> push
>> > > > > > >>>>>>> (few minutes ago) of the upcoming
official CI image i
>> > > > implemented
>> > > > > > >>>> the
>> > > > > > >>>>>>> change we discussed in the
Github where we limit the
>> number
>> > > of
>> > > > > > >>>>>> combinations
>> > > > > > >>>>>>> we test:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> You can see it yourself:
>> > > > > > >>>>>>> https://travis-ci.org/apache/airflow/builds/551305240
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Those are the combinations
I propose:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Python: 3.6
>> > > > > > >>>>>>> BACKEND=mysql ENV=docker
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Python: 3.6
>> > > > > > >>>>>>> BACKEND=postgres ENV=docker
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Python: 3.5
>> > > > > > >>>>>>> BACKEND=sqlite ENV=docker
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Python: 3.6
>> > > > > > >>>>>>> BACKEND=postgres ENV=kubernetes
>> KUBERNETES_VERSION=v1.13.0
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> J,
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> On Thu, Jun 27, 2019 at 11:00
AM Driesprong, Fokko
>> > > > > > >>>>> <fokko@driesprong.frl
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> We got this message last
year:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> Hello, Airflow PPMC.
>> > > > > > >>>>>>>>> While going through
the usage statistics for our
>> Travis
>> > CI
>> > > > > > >>>>> service, I
>> > > > > > >>>>>>>>> have noticed that the
Airflow project is using an
>> > > abnormally
>> > > > > > >>>> large
>> > > > > > >>>>>>>>> amount of resources,
2600 hours per month or the
>> > equivalent
>> > > > of
>> > > > > > >>>>> having
>> > > > > > >>>>>>>>> almost 4 machines building
airflow non-stop 24/7. As
>> this
>> > > is
>> > > > > not
>> > > > > > >>>>>> free,
>> > > > > > >>>>>>>>> but rather costing
us money, I'm contacting you with
>> the
>> > > > > > >>>> intention
>> > > > > > >>>>> of
>> > > > > > >>>>>>>>> figuring out ways to
reduce the use of Travis for the
>> > > > project.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> We would greatly prefer
that the project itself comes
>> up
>> > > > with a
>> > > > > > >>>>>>> solution
>> > > > > > >>>>>>>>> to lower the usage
of Travis, as we'd hate to simply
>> turn
>> > > it
>> > > > > off
>> > > > > > >>>>> for
>> > > > > > >>>>>>>>> you, but the usage
is at a rather severe level,
>> totaling
>> > > more
>> > > > > > >>>> than
>> > > > > > >>>>>> 21%
>> > > > > > >>>>>>>>> of the total build
time of all projects using Travis,
>> so
>> > > > > > >>>> something
>> > > > > > >>>>>>>>> actionable should be
decided upon and (preferably)
>> > > completed
>> > > > by
>> > > > > > >>>> the
>> > > > > > >>>>>> end
>> > > > > > >>>>>>>>> of May that will reduce
the consumption of Travis
>> > > resources.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> Alternately, if you
are unable to lower the pressure
>> on
>> > > > Travis,
>> > > > > > >>>> the
>> > > > > > >>>>>>>>> podling and/or IPMC
may ask the board of directors
>> for a
>> > > > > > >>>> separate
>> > > > > > >>>>>>> budget
>> > > > > > >>>>>>>>> for additional build
nodes to cope with the added
>> load -
>> > > I'll
>> > > > > > >>>> leave
>> > > > > > >>>>>>> this
>> > > > > > >>>>>>>>> for the podling and
IPMC to decide on.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> Please let us know
when you have decided on a plan to
>> > > remedy
>> > > > > > >>>> this
>> > > > > > >>>>>>>> situation.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> With regards,
>> > > > > > >>>>>>>>> Daniel on behalf of
ASF Infrastructure.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> I think more and more projects
are still migrating to
>> the
>> > > ASF
>> > > > > > >>>> Travis,
>> > > > > > >>>>>> so
>> > > > > > >>>>>>> I
>> > > > > > >>>>>>>> think natural that there
is more load. However, this
>> still
>> > > > > leaves
>> > > > > > >>>> the
>> > > > > > >>>>>>>> question if we have to
run the full matrix.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Cheers, Fokko
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Op do 27 jun. 2019 om 10:56
schreef Jarek Potiuk <
>> > > > > > >>>>>>> Jarek.Potiuk@polidea.com
>> > > > > > >>>>>>>>> :
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> I think we should really
involve infra to increase the
>> > slot
>> > > > > > >>>> number
>> > > > > > >>>>> or
>> > > > > > >>>>>>>> maybe
>> > > > > > >>>>>>>>> even somehow allocate
slots per project.
>> > > > > > >>>>>>>>> The problem is that
we cannot control what other
>> apache
>> > > > > projects
>> > > > > > >>>>> are
>> > > > > > >>>>>>>> doing,
>> > > > > > >>>>>>>>> so even if we decrease
our runtime, it's the other
>> > projects
>> > > > > that
>> > > > > > >>>>>> might
>> > > > > > >>>>>>>> hold
>> > > > > > >>>>>>>>> us in the queue :(
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> J.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> On Thu, Jun 27, 2019
at 10:19 AM Driesprong, Fokko
>> > > > > > >>>>>>> <fokko@driesprong.frl
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> wrote:
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> I've noticed this
at other Apache projects as well,
>> > > > sometimes
>> > > > > > >>>> it
>> > > > > > >>>>>>> takes
>> > > > > > >>>>>>>> up
>> > > > > > >>>>>>>>>> to 7-8 hours. The
only thing we can do, is reduce the
>> > > > runtime
>> > > > > > >>>> of
>> > > > > > >>>>>> the
>> > > > > > >>>>>>>> jobs
>> > > > > > >>>>>>>>>> so we take less
slots :-)
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> Cheers, Fokko
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> Op wo 26 jun. 2019
om 21:59 schreef Jarek Potiuk <
>> > > > > > >>>>>>>>> Jarek.Potiuk@polidea.com
>> > > > > > >>>>>>>>>>> :
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>>> Yep. That's
what I suggested as the reason in the
>> > ticket
>> > > -
>> > > > I
>> > > > > > >>>>>> guess
>> > > > > > >>>>>>>>> INFRA
>> > > > > > >>>>>>>>>>> are the only
people who can do anything about it
>> > > (increase
>> > > > > > >>>>>>>> concurrency
>> > > > > > >>>>>>>>> ?
>> > > > > > >>>>>>>>>>> pay more for
Travis :)? ).
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> On Wed, Jun
26, 2019 at 9:51 PM Ash Berlin-Taylor <
>> > > > > > >>>>>> ash@apache.org>
>> > > > > > >>>>>>>>>> wrote:
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> I asked
Travis on twitter and they said it was due
>> to
>> > > the
>> > > > > > >>>>>> Apache
>> > > > > > >>>>>>>>> other
>> > > > > > >>>>>>>>>>>> projects
build queues
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>
>> > https://twitter.com/travisci/status/1143893051460526080
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> -ash
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> On 26 June
2019 20:48:33 BST, Jarek Potiuk <
>> > > > > > >>>>>>>> Jarek.Potiuk@polidea.com
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>>>> wrote:
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Hello
everyone,
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> For
the last few days the Travis builds for
>> > > > > > >>>> apache/airflow
>> > > > > > >>>>>>> project
>> > > > > > >>>>>>>>> are
>> > > > > > >>>>>>>>>>>>> waiting
in a queue for hours. This is not a normal
>> > > > > > >>>>> situation.
>> > > > > > >>>>>>> I've
>> > > > > > >>>>>>>>>>> opened
>> > > > > > >>>>>>>>>>>>> INFRA
ticket for that:
>> > > > > > >>>>>>>>>>> https://issues.apache.org/jira/browse/INFRA-18657
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> J.
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> --
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> Jarek Potiuk
>> > > > > > >>>>>>>>>>> Polidea <https://www.polidea.com/>
| Principal
>> > Software
>> > > > > > >>>>> Engineer
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> M: +48 660
796 129 <+48660796129>
>> > > > > > >>>>>>>>>>> [image: Polidea]
<https://www.polidea.com/>
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> --
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> Jarek Potiuk
>> > > > > > >>>>>>>>> Polidea <https://www.polidea.com/>
| Principal
>> Software
>> > > > > > >>>> Engineer
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> M: +48 660 796 129
<+48660796129>
>> > > > > > >>>>>>>>> [image: Polidea] <https://www.polidea.com/>
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> --
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Jarek Potiuk
>> > > > > > >>>>>>> Polidea <https://www.polidea.com/>
| Principal Software
>> > > > Engineer
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> M: +48 660 796 129 <+48660796129>
>> > > > > > >>>>>>> [image: Polidea] <https://www.polidea.com/>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>
>> > > > > > >>>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>> --
>> > > > > > >>>
>> > > > > > >>> Jarek Potiuk
>> > > > > > >>> Polidea <https://www.polidea.com/> |
Principal Software
>> > Engineer
>> > > > > > >>>
>> > > > > > >>> M: +48 660 796 129 <+48660796129>
>> > > > > > >>> [image: Polidea] <https://www.polidea.com/>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>
>> > > > > > >> --
>> > > > > > >>
>> > > > > > >> Jarek Potiuk
>> > > > > > >> Polidea <https://www.polidea.com/> | Principal
Software
>> > Engineer
>> > > > > > >>
>> > > > > > >> M: +48 660 796 129 <+48660796129>
>> > > > > > >> [image: Polidea] <https://www.polidea.com/>
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Jarek Potiuk
>> > > > > Polidea <https://www.polidea.com/> | Principal Software
Engineer
>> > > > >
>> > > > > M: +48 660 796 129 <+48660796129>
>> > > > > [image: Polidea] <https://www.polidea.com/>
>> > > > >
>> > > >
>> > >
>> >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >
>> > M: +48 660 796 129 <+48660796129>
>> > [image: Polidea] <https://www.polidea.com/>
>> >
>>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message