beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Romain Manni-Bucau <rmannibu...@gmail.com>
Subject Re: How to cope with Maven transient network issues?
Date Tue, 05 Dec 2017 11:11:17 GMT
FYI https://github.com/apache/maven-wagon/pull/37

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-12-05 6:54 GMT+01:00 Romain Manni-Bucau <rmannibucau@gmail.com>:
>
>
> Le 5 déc. 2017 06:20, "Eugene Kirpichov" <kirpichov@google.com> a écrit :
>
>
>
> On Mon, Dec 4, 2017 at 1:45 PM Romain Manni-Bucau <rmannibucau@gmail.com>
> wrote:
>>
>>
>>
>> Le 4 déc. 2017 21:45, "Eugene Kirpichov" <kirpichov@google.com> a écrit :
>>
>> Romain - as far as I understand, Maven *does* have a retry strategy, but
>> it is a poor retry strategy and there is no way to tweak it. In particular,
>> if it uses https://maven.apache.org/guides/mini/guide-http-settings.html ,
>> that means it uses Apache Http Client 4.1.2 whose default retry settings are
>> extremely conservative
>>
>> https://github.com/apache/httpcomponents-client/blob/4.1.x/httpclient/src/main/java/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.java#L93
>>
>>
>> Right
>>
>>
>>
>> The errors we're hitting with Maven are some of those: "connection
>> refused" and alike.
>>
>>
>>
>> Would a retry guarantee you it works. Should be quite easy to switch wagon
>> with a configurable retry handler instance through mvnwrapper or custom mvn
>> setup and test it but the logic looks sane to me.
>
> Retries indeed don't give any guarantees, but they're very good at reducing
> error rate, and our builds could definitely use some of that.
>
> I tried to figure out how to make Maven use a configurable http wagon with a
> custom retry handler, but I couldn't - if you know more, could you give some
> pointers? I think we should definitely do that.
>
>
> It is not configurable but to ensure it is worth a mvn patch we can add a
> mvn wrapper script where we patch wagon and use that on the CI.
>
>
>
>>
>>
>> If it is on asf repo we must work with infra to fix it rather than working
>> around the clients. If on a repo we dont control we can try to get rid of it
>> maybe?
>
> It happens often enough with the maven central repo repo.maven.apache.org.
> At Beam's (moderate) scale of hundreds of builds per day resolving hundreds
> of dependencies each, even 99.99% availability with a poor retry strategy
> would translate to builds failing regularly.
>
>
> Hmm, this is an asf server and behing should be repo1 and repo2 so either
> there is a huge client issue or the proxy can be better configure. If you
> have some failure logs, do you mind pinging infra with them?
>
> There's also multiple parts that can fail - Maven Central itself, network
> issues between Jenkins and Maven Central, network issues on the Jenkins
> machine itself and so on - I doubt that ASF can fix all of this for us.
> I think this sort of issue *has* to be resolved on the client.
>
>
> Not all kind of failures, connection breakdown during the transfert yes but
> other kind means the server is down so retry shouldnt help.
>
> Let me know if you nees help with wagon patching/testing. Can have a few
> cycles end of the week to give some help.
>
>
>>
>>
>>
>>
>> On Mon, Dec 4, 2017 at 12:15 PM Romain Manni-Bucau <rmannibucau@gmail.com>
>> wrote:
>>>
>>> 2017-12-04 18:58 GMT+01:00 Kenneth Knowles <klk@google.com>:
>>> > On Sat, Dec 2, 2017 at 3:06 AM, Romain Manni-Bucau
>>> > <rmannibucau@gmail.com>
>>> > wrote:
>>> >>
>>> >> Need to check but if plugin dependencies were not tuned it should be
>>> >> the
>>> >> default with a retry of 3 IIRC.
>>> >>
>>> >> But arent you sure it was a repo/server issue any client cant solve?
>>> >
>>> >
>>> > Not sure what you mean here. What we need is for mvn and gradle to
>>> > succeed
>>> > even though there are frequent transient failures of the repo. Anything
>>> > that
>>> > makes this happen is success.
>>>
>>> What I meant is you will likely never get it with whatever build
>>> system until you go 100% local which wouldn't make the build
>>> reproducible.
>>>
>>> Concretely you can do
>>>
>>> 1.
>>>
>>> for (int i = 0; i < 3; i++) {
>>>   if (download()) break;
>>> }
>>>
>>>
>>> 2.
>>>
>>> for (int i = 0; i < 3; i++) {
>>>   if (download()) break;
>>>   exponentialBackoff();
>>> }
>>>
>>>
>>> Or other wait strategy with a timeout.
>>>  but you can't have:
>>>
>>> while (!download()) {
>>>   waitUntilItWorks();
>>> }
>>>
>>>
>>> So concretely if your down time > reasonnable time you will see the
>>> failure anyway.
>>>
>>> Maven default has a retry (you see it sometimes in the logs) so should
>>> tolerante a restart or small downtime. Ivy (gradle) uses httpclient as
>>> well so same kind of config is available but with the same limitation
>>> by design.
>>>
>>> Having a local cache is the best way to avoid these issues but it
>>> requires at least one green build for dependencies.
>>>
>>> What I often saw was to do a pre-build step just resolving
>>> dependencies+plugins and use a .m2/repository caching.
>>>
>>> >
>>> > Kenn
>>> >
>>> >>
>>> >>
>>> >> Le 1 déc. 2017 23:27, "Kenneth Knowles" <klk@google.com> a écrit
:
>>> >>>
>>> >>> How do you instruct maven or gradle to use that all the time?
>>> >>>
>>> >>> On Fri, Dec 1, 2017 at 1:58 PM, Romain Manni-Bucau
>>> >>> <rmannibucau@gmail.com> wrote:
>>> >>>>
>>> >>>> If you use wagon http - and not lightweigh - it should have
retries
>>> >>>> by
>>> >>>> default.
>>> >>>>
>>> >>>> Le 1 déc. 2017 22:31, "Valentyn Tymofieiev" <valentyn@google.com>
a
>>> >>>> écrit :
>>> >>>>>
>>> >>>>> Has this ever been brought up in Maven dev community? Perhaps
they
>>> >>>>> have
>>> >>>>> some suggestions. It sounds like a reasonable feature request.
>>> >>>>>
>>> >>>>> On Fri, Dec 1, 2017 at 1:18 PM, Kenneth Knowles <klk@google.com>
>>> >>>>> wrote:
>>> >>>>>>
>>> >>>>>> I've repeatedly searched around for a way to just add
proper retry
>>> >>>>>> to
>>> >>>>>> maven or gradle, and haven't found anything :-(
>>> >>>>>>
>>> >>>>>> I had thought that we altered our builds in such a way
that the
>>> >>>>>> .m2
>>> >>>>>> directory was permitted to survive across builds. True
that it
>>> >>>>>> isn't
>>> >>>>>> hermetic, precisely, but it is pretty safe to treat
as a cache of
>>> >>>>>> immutable
>>> >>>>>> data, which is no more dangerous than having a caching
incremental
>>> >>>>>> build
>>> >>>>>> system.
>>> >>>>>>
>>> >>>>>> Kenn
>>> >>>>>>
>>> >>>>>> On Wed, Nov 29, 2017 at 3:39 PM, Eugene Kirpichov
>>> >>>>>> <kirpichov@google.com> wrote:
>>> >>>>>>>
>>> >>>>>>> Our builds often hit transient Maven network issues,
e.g. this
>>> >>>>>>> one
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/5331/consoleFull
>>> >>>>>>>
>>> >>>>>>> 2017-11-29T02:18:02.936 [ERROR] Failed to execute
goal on project
>>> >>>>>>> beam-sdks-java-io-hadoop-jdk1.8-tests: Could not
resolve
>>> >>>>>>> dependencies for
>>> >>>>>>> project
>>> >>>>>>>
>>> >>>>>>> org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:2.3.0-SNAPSHOT:
>>> >>>>>>> Could not transfer artifact org.apache.derby:derby:jar:10.10.2.0
>>> >>>>>>> from/to
>>> >>>>>>> central (https://repo.maven.apache.org/maven2):
GET request of:
>>> >>>>>>> org/apache/derby/derby/10.10.2.0/derby-10.10.2.0.jar
from central
>>> >>>>>>> failed:
>>> >>>>>>> Connection reset -> [Help 1]
>>> >>>>>>>
>>> >>>>>>> It'd be good to increase reliability of our builds.
>>> >>>>>>> repo.maven.apache.org seems quite unreliable.
>>> >>>>>>>
>>> >>>>>>> I tried finding a way to configure Maven to retry
such network
>>> >>>>>>> errors
>>> >>>>>>> and it appears to be impossible [will be happy if
someone proves
>>> >>>>>>> me wrong].
>>> >>>>>>>
>>> >>>>>>> Would this issue be resolved if we used multiple
mirrors?
>>> >>>>>>> https://maven.apache.org/guides/mini/guide-mirror-settings.html
>>> >>>>>>> Any other
>>> >>>>>>> suggestions?
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>
>>> >
>>
>>
>

Mime
View raw message