www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Demers <brian.dem...@gmail.com>
Subject Re: jenkins emergency restart notice - Tues 00:00 UTC
Date Thu, 20 Jul 2017 16:29:35 GMT
Is this failure related to this? or is there a node that is down?

https://builds.apache.org/job/Shiro/502/console

[EnvInject] - Loading node environment variables.
ERROR: SEVERE ERROR occurs
org.jenkinsci.lib.envinject.EnvInjectException:
hudson.remoting.ChannelClosedException: channel is already closed
	at org.jenkinsci.plugins.envinject.service.EnvironmentVariablesNodeLoader.gatherEnvVarsForNode(EnvironmentVariablesNodeLoader.java:95)
	at org.jenkinsci.plugins.envinject.EnvInjectListener.loadEnvironmentVariablesNode(EnvInjectListener.java:80)
	at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironment(EnvInjectListener.java:43)
	at hudson.model.AbstractBuild$AbstractBuildExecution.createLauncher(AbstractBuild.java:528)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:448)
	at hudson.model.Run.execute(Run.java:1735)
	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:542)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:405)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
	at hudson.remoting.Channel.send(Channel.java:605)
	at hudson.remoting.Request.call(Request.java:130)
	at hudson.remoting.Channel.call(Channel.java:829)
	at hudson.FilePath.act(FilePath.java:1081)
	at org.jenkinsci.plugins.envinject.service.EnvironmentVariablesNodeLoader.gatherEnvVarsForNode(EnvironmentVariablesNodeLoader.java:64)
	... 8 more
Caused by: java.io.IOException
	at hudson.remoting.Channel.close(Channel.java:1292)
	at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:180)
	at hudson.remoting.PingThread.ping(PingThread.java:130)
	at hudson.remoting.PingThread.run(PingThread.java:86)
Caused by: java.util.concurrent.TimeoutException: Ping started at
1500559004189 hasn't completed by 1500559244191
	... 2 more
[EnvInject] - [ERROR] - SEVERE ERROR occurs: channel is already closed


On Wed, Jul 19, 2017 at 11:50 AM, Chris Lambertus <cml@apache.org> wrote:

> I just commented on that ticket. Looks like this was primarily due to the
> switch to systemd startup in ubuntu 16.04 and the need to update
> DefaultLimitNOFILE in /etc/systemd/system.conf (grr) to affect processes
> started via systemctl. Thanks Joan Touzet for pointing this out.
>
> -Chris
>
>
>
> > On Jul 19, 2017, at 8:47 AM, Geoff Macartney <
> geoff.macartney@cloudsoft.io> wrote:
> >
> > Apache Brooklyn builds seem to be ok now, thanks.
> >
> > On Wed, 19 Jul 2017, 16:09 Dominik Psenner, <dpsenner@gmail.com> wrote:
> >
> >> I filed https://issues.apache.org/jira/browse/INFRA-14628 which is
> >> probably
> >> related to this since the exception message says "Too many open files".
> >>
> >> 2017-07-19 17:01 GMT+02:00 Mike Drob <mdrob@apache.org>:
> >>
> >>> Have the changes resulted in any noticeable improvements? Is there a
> JIRA
> >>> interested folks can follow along on?
> >>>
> >>> On Tue, Jul 18, 2017 at 6:40 PM, Chris Lambertus <cml@apache.org>
> wrote:
> >>>
> >>>> We’re trying a suggested downgrade of the credentials binding plugin
> >>> which
> >>>> may be related to this. No guarantees, though. I will be restarting
> >> again
> >>>> tonight at 0100 UTC 19 July.
> >>>>
> >>>> -Chris
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On Jul 18, 2017, at 9:00 AM, Chris Lambertus <cml@apache.org>
wrote:
> >>>>>
> >>>>> This appears to be a bug in one of the new versions of either jenkins
> >>> or
> >>>> a plugin. We are still looking into the issue. The limit was raised
> >>>> yesterday by a factor of 4.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Jul 18, 2017, at 4:08 AM, Geoff Macartney <
> >>>> geoff.macartney@cloudsoft.io> wrote:
> >>>>>>
> >>>>>> We're seeing them in Apache Brooklyn builds too, e.g.
> >>>>>> https://builds.apache.org/job/brooklyn-server-master/664.
> >>>>>>
> >>>>>> regards
> >>>>>> Geoff
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, 18 Jul 2017 at 12:03 Ismael Juma <ismael@juma.me.uk>
wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> Thanks for the heads up. Was this done? We are still seeing
"too
> >> many
> >>>> open
> >>>>>>> files" issues in Kafka builds. One example of many:
> >>>>>>>
> >>>>>>> https://builds.apache.org/job/kafka-trunk-jdk7/
> >>>> lastCompletedBuild/console
> >>>>>>>
> >>>>>>> Ismael
> >>>>>>>
> >>>>>>> On Mon, Jul 17, 2017 at 2:53 PM, Chris Lambertus <cml@apache.org>
> >>>> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Jenkins will be restarted at midnight tonight UTC to
address
> >>> problems
> >>>>>>>> related to ‘too many open files’ and to attempt
to correct a
> >> problem
> >>>> with
> >>>>>>>> some windows agents. I will be stopping new builds so
that the
> >> queue
> >>>> can
> >>>>>>>> empty over the next 2 hours.
> >>>>>>>>
> >>>>>>>> -Chris
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Dominik Psenner
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message