infra-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Wegner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (INFRA-16380) Nodes Failing With "Backing Channel '<slave_name>' is disconnected.
Date Wed, 18 Apr 2018 23:19:00 GMT

    [ https://issues.apache.org/jira/browse/INFRA-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443322#comment-16443322
] 

Scott Wegner commented on INFRA-16380:
--------------------------------------

[~cml] thanks for the details, and setting up a monitoring dashboard sounds perfect. I'm not
familiar with the capabilities of Data Dog, but as Jason mentioned: the more data the better.

For the current task of build migration, here's what I'm grappling with: Gradle does much
more task parallelization than we saw with our previous build, and is spinning up so many
workers that it exhausts machine memory. We have a number of different Jenkins jobs that all
do different subsets of build/testing and will each have its own performance characteristics.
I need to figure out the right tuning parameters for the build to make sure memory is safely
constrained while not over-throttling build performance.

For the above problem, here's an ideal workflow:
a) Kick off a particular Jenkins job
b) While it's running, view all processes and subprocesses launched as part of the job, with
commandline parameters and a timeline of the current and historical memory usage
c) Look for times where memory usage peaked above some limit, and drill into what was going
on with as much correlated context as possible, such as heap dump, gradle tasks status, log
files, etc.
d) Based on investigation, tweak some build parameters and iterate until we're happy.

I realize that's probably a pipe dream, but any portion that you could enable would be amazing
:)

> Nodes Failing With "Backing Channel '<slave_name>' is disconnected.
> -------------------------------------------------------------------
>
>                 Key: INFRA-16380
>                 URL: https://issues.apache.org/jira/browse/INFRA-16380
>             Project: Infrastructure
>          Issue Type: Bug
>          Components: Jenkins
>            Reporter: Jason Kuster
>            Priority: Major
>
> We've seen a couple of Jenkins builds dying with the cited error. They generally look
like the following:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4179/console
> {code}
> FATAL: command execution failed
> java.io.IOException: Backing channel 'beam2' is disconnected.
> 	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:212)
> 	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:281)
> 	at com.sun.proxy.$Proxy131.isAlive(Unknown Source)
> 	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1138)
> 	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1130)
> 	at hudson.Launcher$ProcStarter.join(Launcher.java:465)
> 	at hudson.plugins.gradle.Gradle.performTask(Gradle.java:333)
> 	at hudson.plugins.gradle.Gradle.perform(Gradle.java:225)
> 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
> 	at hudson.model.Build$BuildExecution.build(Build.java:206)
> 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
> 	at hudson.model.Run.execute(Run.java:1724)
> 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 	at hudson.model.ResourceController.execute(ResourceController.java:97)
> 	at hudson.model.Executor.run(Executor.java:429)
> Caused by: java.io.IOException: Unexpected termination of the channel
> 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
> Caused by: java.io.EOFException
> 	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
> 	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
> 	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
> 	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
> 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
> 	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
> 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
> Build step 'Invoke Gradle script' changed build result to FAILURE
> Build step 'Invoke Gradle script' marked build as failure
> ERROR: Step ‘Publish JUnit test result report’ failed: no workspace for beam_PreCommit_Java_GradleBuild
#4179
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> Setting status of 1073baaaa633dd34ed552812e65108944eb92ac6 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4179/
and message: 'FAILURE
>  '
> Using context: Jenkins: ./gradlew :javaPreCommit
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> Finished: FAILURE
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message