www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Lambertus <...@apache.org>
Subject Re: jenkins emergency restart notice - Tues 00:00 UTC
Date Wed, 19 Jul 2017 15:50:12 GMT
I just commented on that ticket. Looks like this was primarily due to the switch to systemd
startup in ubuntu 16.04 and the need to update DefaultLimitNOFILE in /etc/systemd/system.conf
(grr) to affect processes started via systemctl. Thanks Joan Touzet for pointing this out.

-Chris



> On Jul 19, 2017, at 8:47 AM, Geoff Macartney <geoff.macartney@cloudsoft.io> wrote:
> 
> Apache Brooklyn builds seem to be ok now, thanks.
> 
> On Wed, 19 Jul 2017, 16:09 Dominik Psenner, <dpsenner@gmail.com> wrote:
> 
>> I filed https://issues.apache.org/jira/browse/INFRA-14628 which is
>> probably
>> related to this since the exception message says "Too many open files".
>> 
>> 2017-07-19 17:01 GMT+02:00 Mike Drob <mdrob@apache.org>:
>> 
>>> Have the changes resulted in any noticeable improvements? Is there a JIRA
>>> interested folks can follow along on?
>>> 
>>> On Tue, Jul 18, 2017 at 6:40 PM, Chris Lambertus <cml@apache.org> wrote:
>>> 
>>>> We’re trying a suggested downgrade of the credentials binding plugin
>>> which
>>>> may be related to this. No guarantees, though. I will be restarting
>> again
>>>> tonight at 0100 UTC 19 July.
>>>> 
>>>> -Chris
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Jul 18, 2017, at 9:00 AM, Chris Lambertus <cml@apache.org> wrote:
>>>>> 
>>>>> This appears to be a bug in one of the new versions of either jenkins
>>> or
>>>> a plugin. We are still looking into the issue. The limit was raised
>>>> yesterday by a factor of 4.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Jul 18, 2017, at 4:08 AM, Geoff Macartney <
>>>> geoff.macartney@cloudsoft.io> wrote:
>>>>>> 
>>>>>> We're seeing them in Apache Brooklyn builds too, e.g.
>>>>>> https://builds.apache.org/job/brooklyn-server-master/664.
>>>>>> 
>>>>>> regards
>>>>>> Geoff
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, 18 Jul 2017 at 12:03 Ismael Juma <ismael@juma.me.uk>
wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> Thanks for the heads up. Was this done? We are still seeing "too
>> many
>>>> open
>>>>>>> files" issues in Kafka builds. One example of many:
>>>>>>> 
>>>>>>> https://builds.apache.org/job/kafka-trunk-jdk7/
>>>> lastCompletedBuild/console
>>>>>>> 
>>>>>>> Ismael
>>>>>>> 
>>>>>>> On Mon, Jul 17, 2017 at 2:53 PM, Chris Lambertus <cml@apache.org>
>>>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> Jenkins will be restarted at midnight tonight UTC to address
>>> problems
>>>>>>>> related to ‘too many open files’ and to attempt to correct
a
>> problem
>>>> with
>>>>>>>> some windows agents. I will be stopping new builds so that
the
>> queue
>>>> can
>>>>>>>> empty over the next 2 hours.
>>>>>>>> 
>>>>>>>> -Chris
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Dominik Psenner
>> 


Mime
View raw message