Mailing-List: contact dev-help@maven.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Maven Developers List" <dev@maven.apache.org>
Date: Sun, 19 Feb 2017 05:11:57 -0700 (MST)
From: Tibor Digana <tibordigana@apache.org>
To: dev@maven.apache.org
Message-ID: <CAKuVzBRAKSnML+tqbm7LwOBG6+qC92ioxJbB2FzriYZ0rDrecA@mail.gmail.com>
In-Reply-To: <692075fe-3d78-ba5c-d400-4be294d2a61c@apache.org>
References: <6e497d13-0c12-14f3-98b7-e0cbef8c79a6@apache.org> <a124d948-39f7-2dab-e4ff-d60a8a85a735@schulte.it> <0db8050b-736a-e601-d527-cfd9b184cc02@apache.org> <CAKuVzBQm20oubryykEkHfBooHw-B6_fcE9fvOm=FCM75ydJTow@mail.gmail.com> <CA+nPnMzMjP=_=N9N3F0itddKD52T8mzBMVx7PPuJoAp5sUADXg@mail.gmail.com> <692075fe-3d78-ba5c-d400-4be294d2a61c@apache.org>
Subject: Re: Recent issues found in Surefire master/Maven Clean Plugin 3.0.0
 with Maven master
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_152214_507033630.1487506317950"
archived-at: Sun, 19 Feb 2017 12:12:03 -0000

------=_Part_152214_507033630.1487506317950
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Michael, pls try to build now. It might be multiple issues we are facing.
The threading issue workaround was pushed to 2.19.2-experimental.

On Sun, Feb 19, 2017 at 1:07 PM, Michael Osipov-2 [via Maven] <
ml-node+s40175n5899236h56@n5.nabble.com> wrote:

> Socket give you, of course, a lot of control, but they also impose an
> overhead compares to named pipes (mkfifo, CreateNamePipe).
>
> I think our biggest problem is that stdout is subject to buffering
> issues and in case if System#exit() status Z is the buffer, never
> written out.
> Surefire thinks that the crashed.
>
> Am 2017-02-19 um 12:32 schrieb Stephen Connolly:
>
> > What about using a local host bound socket rather than studio?
> >
> > In fact a socket could open up other options like forking the tests
> inside
> > docker, or running the surefire forks on remote JVMs.
> >
> > Plus we gain more control (modulo TCP retry/timeouts)
> >
> > On Sun 19 Feb 2017 at 11:16, Tibor Digana <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5899236&i=0>>
> > wrote:
> >
> >> Currently I placed sleep(1sec) before child process exit(0). If the
> reader
> >> process ThreadedStreamConsumer would read buffer entirely within 1 sec
> and
> >> the build pass then I have to add a confirmation mechanism where the
> exit
> >> event must be confirmed by plugin until short period of time.
> >> Later we should not use these pipes but standard file system, but we
> have
> >> to take care of SecurityException thrown by JUnit3.
> >>
> >> On Sun, Feb 19, 2017 at 11:37 AM, Michael Osipov <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5899236&i=1>>
> >> wrote:
> >>
> >>> Am 2017-02-18 um 23:19 schrieb Christian Schulte:
> >>>
> >>>> Am 02/17/17 um 21:48 schrieb Michael Osipov:
> >>>>
> >>>>> Hi folks,
> >>>>>
> >>>>> Christian Schulte asked me a couple of days ago wether I am able to
> >>>>> build Surefire master with Maven master. It was constantly failing
> for
> >>>>> him on his OpenBSD machines. Since I have several real servers with
> >>>>> FreeBSD 10.3-STABLE at hand I did run all Surefire ITs and I was
> able
> >> to
> >>>>> reproduce it. Our entire test infrastructure wasn't unfortunately!
> >>>>>
> >>>>> @Tibor: correct me if something is wrong or missing!
> >>>>>
> >>>>> After several days of heavy testing, thread dumps and log file
> analysis
> >>>>> with Tibor Digana and various Maven combinations (3.3.9, master,
> >>>>> MNG-6169, MNG-6169 + MCLEAN 3.0.0) we figured out that there are
> >> several
> >>>>> serious bugs in Surefire master, Maven Shared Utils 0.9/3.1.0 and
> >> likely
> >>>>> Maven Clean Plugin 3.0.0.
> >>>>>
> >>>>> Since crucial parts of Surefire rely on native code in the JVM
> (forks,
> >>>>> streams), our code was not robust enough.
> >>>>>
> >>>>> As of today we have found:
> >>>>>
> >>>>> * Missing flushes in streams caused forked VM to be apparently
> >>>>> non-responsive
> >>>>> * TestNG tests mostly failed due to duplicate contradicting
> properties
> >>>>> passed to forked VMs
> >>>>> * Uninitialized/too early attributes made daemon threads to kill
> forked
> >>>>> VMs
> >>>>> * Code or dependency change from MCLEAN 2.6.1 to 3.0.0 cause
> repeated
> >>>>> failures of a handful ITs
> >>>>> @Karl Heinz: were you able to figure out something here?
> >>>>>
> >>>>> Issues in JIRA are pending...
> >>>>>
> >>>>> Everyone's invited to take a look at the log output as well as the
> >>>>> target directory of surefire-integration-tests and contribute:
> >>>>> http://home.apache.org/~michaelo/maven/surefire/. The filenames
> should
> >>>>> be pretty much self-explanatory.
> >>>>>
> >>>>> My big question is: how can we improve our test infrastructure? Can
> we
> >>>>> raise with INFRA to get at least one FreeBSD and Solaris node for
> >>>>> Jenkins? I consider coverage on Windows and Ubuntu way to small, we
> do
> >>>>> not even have a CentOS node. Surefire ITs and Maven ITs are
> paramount
> >>>>> for us, we should treat them as such!
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>
> >>>> Does any of your findings solve the following already? This is what
> >>>> makes the Jenkins build jobs appear unreliable. It's sending out an
> >>>> email about a failed job and all you see is something like the
> following
> >>>> sporadically. I am having the same issue locally sporadically.
> >>>>
> >>>> [ERROR] Failed to execute goal
> >>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test
> >>>> (default-test) on project child2: Execution default-test of goal
> >>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test failed:
> The
> >>>> forked VM terminated without saying properly goodbye. VM crash or
> >>>> System.exit called ? -> [Help 1]
> >>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> >>>> execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test
>
> >>>> (default-test) on project child2: Execution default-test of goal
> >>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test failed:
> The
> >>>> forked VM terminated without saying properly goodbye. VM crash or
> >>>> System.exit called ?
> >>>>
> >>>
> >>> We were able to solve only a few cases so far: 10%. The rest is still
> >>> failing.
> >>>
> >>> I made some digging: it seems to be a very common problem in NodeJS
> with
> >>> async and forking:
> >>> https://github.com/nodejs/node/issues/2972
> >>> https://github.com/nodejs/node/issues/6456
> >>> http://stackoverflow.com/q/11138355/696632
> >>> http://stackoverflow.com/q/1716296/696632
> >>> https://github.com/nodejs/node-v0.x-archive/issues/8329
> >>> https://github.com/nodejs/node-v0.x-archive/issues/1669
> >>> https://github.com/nodejs/node-v0.x-archive/issues/3737
> >>>
> >>>
> >>> Especially this answer is our case:
> >>>
> >>>> That's not a bug - process.exit() is an imperative that says "exit
> >> now!".
> >>>>
> >>>> When stdout/stderr is a pipe (as is the case with child processes)
> and
> >>>> said pipe is full, pending data gets lost. Waiting until the pipe
> drains
> >>>> is not an option, that could hang the process indefinitely.
> >>>>
> >>>
> >>> I think if we don't gain more control over stdio, we are lost...
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5899236&i=2>
> >>> For additional commands, e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5899236&i=3>
> >>>
> >>>
> >>
> >>
> >> --
> >> Cheers
> >> Tibor
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5899236&i=4>
> For additional commands, e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5899236&i=5>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://maven.40175.n5.nabble.com/Recent-issues-found-in-
> Surefire-master-Maven-Clean-Plugin-3-0-0-with-Maven-
> master-tp5898937p5899236.html
> To start a new topic under Maven Developers, email
> ml-node+s40175n142166h86@n5.nabble.com
> To unsubscribe from Maven Developers, click here
> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=142166&code=dGlib3JkaWdhbmFAYXBhY2hlLm9yZ3wxNDIxNjZ8LTI4OTQ5MjEwMg==>
> .
> NAML
> <http://maven.40175.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>


--
View this message in context: http://maven.40175.n5.nabble.com/Recent-issues-found-in-Surefire-master-Maven-Clean-Plugin-3-0-0-with-Maven-master-tp5898937p5899239.html
Sent from the Maven Developers mailing list archive at Nabble.com.
------=_Part_152214_507033630.1487506317950--