beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Kirpichov <kirpic...@google.com>
Subject Re: How to cope with Maven transient network issues?
Date Tue, 05 Dec 2017 17:18:18 GMT
Thank you!

On Tue, Dec 5, 2017 at 3:11 AM Romain Manni-Bucau <rmannibucau@gmail.com>
wrote:

> FYI https://github.com/apache/maven-wagon/pull/37
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-12-05 6:54 GMT+01:00 Romain Manni-Bucau <rmannibucau@gmail.com>:
> >
> >
> > Le 5 déc. 2017 06:20, "Eugene Kirpichov" <kirpichov@google.com> a écrit
> :
> >
> >
> >
> > On Mon, Dec 4, 2017 at 1:45 PM Romain Manni-Bucau <rmannibucau@gmail.com
> >
> > wrote:
> >>
> >>
> >>
> >> Le 4 déc. 2017 21:45, "Eugene Kirpichov" <kirpichov@google.com> a
> écrit :
> >>
> >> Romain - as far as I understand, Maven *does* have a retry strategy, but
> >> it is a poor retry strategy and there is no way to tweak it. In
> particular,
> >> if it uses
> https://maven.apache.org/guides/mini/guide-http-settings.html ,
> >> that means it uses Apache Http Client 4.1.2 whose default retry
> settings are
> >> extremely conservative
> >>
> >>
> https://github.com/apache/httpcomponents-client/blob/4.1.x/httpclient/src/main/java/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.java#L93
> >>
> >>
> >> Right
> >>
> >>
> >>
> >> The errors we're hitting with Maven are some of those: "connection
> >> refused" and alike.
> >>
> >>
> >>
> >> Would a retry guarantee you it works. Should be quite easy to switch
> wagon
> >> with a configurable retry handler instance through mvnwrapper or custom
> mvn
> >> setup and test it but the logic looks sane to me.
> >
> > Retries indeed don't give any guarantees, but they're very good at
> reducing
> > error rate, and our builds could definitely use some of that.
> >
> > I tried to figure out how to make Maven use a configurable http wagon
> with a
> > custom retry handler, but I couldn't - if you know more, could you give
> some
> > pointers? I think we should definitely do that.
> >
> >
> > It is not configurable but to ensure it is worth a mvn patch we can add a
> > mvn wrapper script where we patch wagon and use that on the CI.
> >
> >
> >
> >>
> >>
> >> If it is on asf repo we must work with infra to fix it rather than
> working
> >> around the clients. If on a repo we dont control we can try to get rid
> of it
> >> maybe?
> >
> > It happens often enough with the maven central repo
> repo.maven.apache.org.
> > At Beam's (moderate) scale of hundreds of builds per day resolving
> hundreds
> > of dependencies each, even 99.99% availability with a poor retry strategy
> > would translate to builds failing regularly.
> >
> >
> > Hmm, this is an asf server and behing should be repo1 and repo2 so either
> > there is a huge client issue or the proxy can be better configure. If you
> > have some failure logs, do you mind pinging infra with them?
> >
> > There's also multiple parts that can fail - Maven Central itself, network
> > issues between Jenkins and Maven Central, network issues on the Jenkins
> > machine itself and so on - I doubt that ASF can fix all of this for us.
> > I think this sort of issue *has* to be resolved on the client.
> >
> >
> > Not all kind of failures, connection breakdown during the transfert yes
> but
> > other kind means the server is down so retry shouldnt help.
> >
> > Let me know if you nees help with wagon patching/testing. Can have a few
> > cycles end of the week to give some help.
> >
> >
> >>
> >>
> >>
> >>
> >> On Mon, Dec 4, 2017 at 12:15 PM Romain Manni-Bucau <
> rmannibucau@gmail.com>
> >> wrote:
> >>>
> >>> 2017-12-04 18:58 GMT+01:00 Kenneth Knowles <klk@google.com>:
> >>> > On Sat, Dec 2, 2017 at 3:06 AM, Romain Manni-Bucau
> >>> > <rmannibucau@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> Need to check but if plugin dependencies were not tuned it should
be
> >>> >> the
> >>> >> default with a retry of 3 IIRC.
> >>> >>
> >>> >> But arent you sure it was a repo/server issue any client cant solve?
> >>> >
> >>> >
> >>> > Not sure what you mean here. What we need is for mvn and gradle to
> >>> > succeed
> >>> > even though there are frequent transient failures of the repo.
> Anything
> >>> > that
> >>> > makes this happen is success.
> >>>
> >>> What I meant is you will likely never get it with whatever build
> >>> system until you go 100% local which wouldn't make the build
> >>> reproducible.
> >>>
> >>> Concretely you can do
> >>>
> >>> 1.
> >>>
> >>> for (int i = 0; i < 3; i++) {
> >>>   if (download()) break;
> >>> }
> >>>
> >>>
> >>> 2.
> >>>
> >>> for (int i = 0; i < 3; i++) {
> >>>   if (download()) break;
> >>>   exponentialBackoff();
> >>> }
> >>>
> >>>
> >>> Or other wait strategy with a timeout.
> >>>  but you can't have:
> >>>
> >>> while (!download()) {
> >>>   waitUntilItWorks();
> >>> }
> >>>
> >>>
> >>> So concretely if your down time > reasonnable time you will see the
> >>> failure anyway.
> >>>
> >>> Maven default has a retry (you see it sometimes in the logs) so should
> >>> tolerante a restart or small downtime. Ivy (gradle) uses httpclient as
> >>> well so same kind of config is available but with the same limitation
> >>> by design.
> >>>
> >>> Having a local cache is the best way to avoid these issues but it
> >>> requires at least one green build for dependencies.
> >>>
> >>> What I often saw was to do a pre-build step just resolving
> >>> dependencies+plugins and use a .m2/repository caching.
> >>>
> >>> >
> >>> > Kenn
> >>> >
> >>> >>
> >>> >>
> >>> >> Le 1 déc. 2017 23:27, "Kenneth Knowles" <klk@google.com>
a écrit :
> >>> >>>
> >>> >>> How do you instruct maven or gradle to use that all the time?
> >>> >>>
> >>> >>> On Fri, Dec 1, 2017 at 1:58 PM, Romain Manni-Bucau
> >>> >>> <rmannibucau@gmail.com> wrote:
> >>> >>>>
> >>> >>>> If you use wagon http - and not lightweigh - it should
have
> retries
> >>> >>>> by
> >>> >>>> default.
> >>> >>>>
> >>> >>>> Le 1 déc. 2017 22:31, "Valentyn Tymofieiev" <valentyn@google.com>
> a
> >>> >>>> écrit :
> >>> >>>>>
> >>> >>>>> Has this ever been brought up in Maven dev community?
Perhaps
> they
> >>> >>>>> have
> >>> >>>>> some suggestions. It sounds like a reasonable feature
request.
> >>> >>>>>
> >>> >>>>> On Fri, Dec 1, 2017 at 1:18 PM, Kenneth Knowles <klk@google.com>
> >>> >>>>> wrote:
> >>> >>>>>>
> >>> >>>>>> I've repeatedly searched around for a way to just
add proper
> retry
> >>> >>>>>> to
> >>> >>>>>> maven or gradle, and haven't found anything :-(
> >>> >>>>>>
> >>> >>>>>> I had thought that we altered our builds in such
a way that the
> >>> >>>>>> .m2
> >>> >>>>>> directory was permitted to survive across builds.
True that it
> >>> >>>>>> isn't
> >>> >>>>>> hermetic, precisely, but it is pretty safe to treat
as a cache
> of
> >>> >>>>>> immutable
> >>> >>>>>> data, which is no more dangerous than having a
caching
> incremental
> >>> >>>>>> build
> >>> >>>>>> system.
> >>> >>>>>>
> >>> >>>>>> Kenn
> >>> >>>>>>
> >>> >>>>>> On Wed, Nov 29, 2017 at 3:39 PM, Eugene Kirpichov
> >>> >>>>>> <kirpichov@google.com> wrote:
> >>> >>>>>>>
> >>> >>>>>>> Our builds often hit transient Maven network
issues, e.g. this
> >>> >>>>>>> one
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/5331/consoleFull
> >>> >>>>>>>
> >>> >>>>>>> 2017-11-29T02:18:02.936 [ERROR] Failed to execute
goal on
> project
> >>> >>>>>>> beam-sdks-java-io-hadoop-jdk1.8-tests: Could
not resolve
> >>> >>>>>>> dependencies for
> >>> >>>>>>> project
> >>> >>>>>>>
> >>> >>>>>>>
> org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:2.3.0-SNAPSHOT:
> >>> >>>>>>> Could not transfer artifact
> org.apache.derby:derby:jar:10.10.2.0
> >>> >>>>>>> from/to
> >>> >>>>>>> central (https://repo.maven.apache.org/maven2):
GET request
> of:
> >>> >>>>>>> org/apache/derby/derby/10.10.2.0/derby-10.10.2.0.jar
from
> central
> >>> >>>>>>> failed:
> >>> >>>>>>> Connection reset -> [Help 1]
> >>> >>>>>>>
> >>> >>>>>>> It'd be good to increase reliability of our
builds.
> >>> >>>>>>> repo.maven.apache.org seems quite unreliable.
> >>> >>>>>>>
> >>> >>>>>>> I tried finding a way to configure Maven to
retry such network
> >>> >>>>>>> errors
> >>> >>>>>>> and it appears to be impossible [will be happy
if someone
> proves
> >>> >>>>>>> me wrong].
> >>> >>>>>>>
> >>> >>>>>>> Would this issue be resolved if we used multiple
mirrors?
> >>> >>>>>>>
> https://maven.apache.org/guides/mini/guide-mirror-settings.html
> >>> >>>>>>> Any other
> >>> >>>>>>> suggestions?
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>>
> >>> >
> >>
> >>
> >
>

Mime
View raw message