infra-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Thistlethwaite (JIRA)" <>
Subject [jira] [Commented] (INFRA-16380) Nodes Failing With "Backing Channel '<slave_name>' is disconnected.
Date Thu, 19 Apr 2018 21:05:00 GMT


Chris Thistlethwaite commented on INFRA-16380:

Ran into a bit of a snag with the dashboard. Seems that publicly shared datadog dashboards
can't use template variables. My original idea was to have nice drop downs so you could select
a specific node and show you system data along with builds that ran on that node.  

Here's what it would look like for beam8
notice the flat lines, that's when the node wasn't sending data to datadog for some reason
(OOM, reboot, offline, network, etc).

There might also be some missing data as I had to create new labels to map Jenkins node names
to actual hostnames (beam8 vs, but those will fill in as the dashboard

> Nodes Failing With "Backing Channel '<slave_name>' is disconnected.
> -------------------------------------------------------------------
>                 Key: INFRA-16380
>                 URL:
>             Project: Infrastructure
>          Issue Type: Bug
>          Components: Jenkins
>            Reporter: Jason Kuster
>            Priority: Major
> We've seen a couple of Jenkins builds dying with the cited error. They generally look
like the following:
> {code}
> FATAL: command execution failed
> Backing channel 'beam2' is disconnected.
> 	at hudson.remoting.RemoteInvocationHandler.channelOrFail(
> 	at hudson.remoting.RemoteInvocationHandler.invoke(
> 	at com.sun.proxy.$Proxy131.isAlive(Unknown Source)
> 	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(
> 	at hudson.Launcher$RemoteLauncher$ProcImpl.join(
> 	at hudson.Launcher$ProcStarter.join(
> 	at hudson.plugins.gradle.Gradle.performTask(
> 	at hudson.plugins.gradle.Gradle.perform(
> 	at hudson.tasks.BuildStepMonitor$1.perform(
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(
> 	at hudson.model.Build$
> 	at hudson.model.Build$BuildExecution.doRun(
> 	at hudson.model.AbstractBuild$
> 	at hudson.model.Run.execute(
> 	at
> 	at hudson.model.ResourceController.execute(
> 	at
> Caused by: Unexpected termination of the channel
> 	at hudson.remoting.SynchronousCommandTransport$
> Caused by:
> 	at$PeekInputStream.readFully(
> 	at$BlockDataInputStream.readShort(
> 	at
> 	at<init>(
> 	at hudson.remoting.ObjectInputStreamEx.<init>(
> 	at
> 	at hudson.remoting.SynchronousCommandTransport$
> Build step 'Invoke Gradle script' changed build result to FAILURE
> Build step 'Invoke Gradle script' marked build as failure
> ERROR: Step ‘Publish JUnit test result report’ failed: no workspace for beam_PreCommit_Java_GradleBuild
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> Setting status of 1073baaaa633dd34ed552812e65108944eb92ac6 to FAILURE with url
and message: 'FAILURE
>  '
> Using context: Jenkins: ./gradlew :javaPreCommit
> ERROR: beam2 is offline; cannot locate JDK 1.8 (latest)
> Finished: FAILURE
> {code}

This message was sent by Atlassian JIRA

View raw message