www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: ulimit changed with Apache Jenkins upgrade?
Date Wed, 03 Sep 2014 17:20:28 GMT
I've opened a BUILDS jira for this:
https://issues.apache.org/jira/browse/BUILDS-17

On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <phunt@apache.org> wrote:
> Giri could you or one of the other folks look into this? Our tests
> have been broken for some time.
>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>
> The same test is still failing with the same error (too many open
> files) and afaict the ulimit is still just set to 4k. I'm printing the
> ulimit info at the start of the job, here it is:
>
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 386177
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 4096
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 386177
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
>
> Patrick
>
> On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <phunt@apache.org> wrote:
>> Does someone in builds@ have the ability to address this? (restart the
>> jenkins slaves so that the new ulimit limits will take effect)
>>
>> Thanks!
>>
>> Patrick
>>
>> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <phunt@apache.org> wrote:
>>> Giri do you mean me? I don't have access to that afaict.
>>>
>>> Patrick
>>>
>>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>>> <gkesavan@hortonworks.com> wrote:
>>>> jenkins slaves might need a restart.
>>>>
>>>> Could you pls re-launch the slaves from the jenkins UI configuration page?
>>>>
>>>> -giri
>>>>
>>>>
>>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <phunt@apache.org> wrote:
>>>>>
>>>>> Thanks Giri! Unfortunately though it seems to not have taken effect,
I
>>>>> just kicked off a precommit build and I see
>>>>>
>>>>> core file size          (blocks, -c) 0
>>>>> data seg size           (kbytes, -d) unlimited
>>>>> scheduling priority             (-e) 0
>>>>> file size               (blocks, -f) unlimited
>>>>> pending signals                 (-i) 386178
>>>>> max locked memory       (kbytes, -l) 64
>>>>> max memory size         (kbytes, -m) unlimited
>>>>> open files                      (-n) 4096
>>>>> pipe size            (512 bytes, -p) 8
>>>>> POSIX message queues     (bytes, -q) 819200
>>>>> real-time priority              (-r) 0
>>>>> stack size              (kbytes, -s) 8192
>>>>> cpu time               (seconds, -t) unlimited
>>>>> max user processes              (-u) 386178
>>>>> virtual memory          (kbytes, -v) unlimited
>>>>> file locks                      (-x) unlimited
>>>>>
>>>>> more details here:
>>>>>
>>>>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>>>>>
>>>>> Patrick
>>>>>
>>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>>>>> <gkesavan@hortonworks.com> wrote:
>>>>> >
>>>>> >
>>>>> > jenkins@asf901:~$ ulimit -a
>>>>> > core file size          (blocks, -c) 0
>>>>> > data seg size           (kbytes, -d) unlimited
>>>>> > scheduling priority             (-e) 0
>>>>> > file size               (blocks, -f) unlimited
>>>>> > pending signals                 (-i) 386178
>>>>> > max locked memory       (kbytes, -l) 64
>>>>> > max memory size         (kbytes, -m) unlimited
>>>>> > open files                      (-n) 60000
>>>>> > pipe size            (512 bytes, -p) 8
>>>>> > POSIX message queues     (bytes, -q) 819200
>>>>> > real-time priority              (-r) 0
>>>>> > stack size              (kbytes, -s) 8192
>>>>> > cpu time               (seconds, -t) unlimited
>>>>> > max user processes              (-u) 10240
>>>>> > virtual memory          (kbytes, -v) unlimited
>>>>> > file locks                      (-x) unlimited
>>>>> >
>>>>> > bumped up the open files and max user processes on all the slaves.
>>>>> >
>>>>> >
>>>>> >
>>>>> > -giri
>>>>> >
>>>>> >
>>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <phunt@apache.org>
wrote:
>>>>> >>
>>>>> >> Giri any chance can you take a look at the ulimit issue on the
H#
>>>>> >> machines? All the ZK precommit builds are failing as a result.
>>>>> >>
>>>>> >> I updated the precommit build last night to output "ulimit -a"
and it
>>>>> >> says the current limit is 4096, can we bump that up or set the
default
>>>>> >> to unlimited?
>>>>> >>
>>>>> >> Thanks!
>>>>> >>
>>>>> >> Patrick
>>>>> >>
>>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <rakeshr@huawei.com>
wrote:
>>>>> >> > +1
>>>>> >> >
>>>>> >> >
>>>>> >> > Adding one more point. I could see the following error
too in the
>>>>> >> > pre-commit build.
>>>>> >> >
>>>>> >> >      [exec]     [junit] Exception in thread "CommitProcWorkThread-16"
>>>>> >> > java.lang.NoClassDefFoundError:
>>>>> >> > org/apache/zookeeper/server/ConnectionBean
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> >
>>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>> >> >      [exec]     [junit]         at
>>>>> >> > java.lang.Thread.run(Thread.java:662)
>>>>> >> >
>>>>> >> > -Rakesh
>>>>> >> >
>>>>> >> > -----Original Message-----
>>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>>>>> >> > Sent: 21 July 2014 02:19
>>>>> >> > To: dev@zookeeper.apache.org
>>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
>>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >
>>>>> >> > +1
>>>>> >> >
>>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <phunt@apache.org>
wrote:
>>>>> >> >
>>>>> >> >> Hi builds folks, is this a system wide issue or something
we should
>>>>> >> >> address ourselves? Thanks!
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >>
>>>>> >> >> ---------- Forwarded message ----------
>>>>> >> >> From: Patrick Hunt <phunt@apache.org>
>>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
>>>>> >> >> To: Giridharan Kesavan <gkesavan@hortonworks.com>
>>>>> >> >> Cc: DevZooKeeper <dev@zookeeper.apache.org>,
Andrew Bayer
>>>>> >> >> <andrew@cloudera.com>
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Hi Giri, can you check that the new hosts (H#) have
the ulimit set
>>>>> >> >> to
>>>>> >> >> what it was set to on the original hadoop# hosts? I'm
seeing new
>>>>> >> >> test
>>>>> >> >> failures with
>>>>> >> >>
>>>>> >> >>     [exec]     [junit] java.io.FileNotFoundException:
>>>>> >> >>
>>>>> >> >> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>>>>> >> >>
>>>>> >> >> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>>>>> >> >> 01
>>>>> >> >> (Too many open files)
>>>>> >> >>
>>>>> >> >> which we've not seen before. I believe this means that
we running
>>>>> >> >> out
>>>>> >> >> of file descriptors?
>>>>> >> >>
>>>>> >> >> Can you verify and address if possible?
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >>
>>>>> >> >> Patrick
>>>>> >> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > CONFIDENTIALITY NOTICE
>>>>> > NOTICE: This message is intended for the use of the individual or
entity
>>>>> > to
>>>>> > which it is addressed and may contain information that is confidential,
>>>>> > privileged and exempt from disclosure under applicable law. If the
>>>>> > reader of
>>>>> > this message is not the intended recipient, you are hereby notified
that
>>>>> > any
>>>>> > printing, copying, dissemination, distribution, disclosure or forwarding
>>>>> > of
>>>>> > this communication is strictly prohibited. If you have received
this
>>>>> > communication in error, please contact the sender immediately and
delete
>>>>> > it
>>>>> > from your system. Thank You.
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or entity
to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the reader
of
>>>> this message is not the intended recipient, you are hereby notified that
any
>>>> printing, copying, dissemination, distribution, disclosure or forwarding
of
>>>> this communication is strictly prohibited. If you have received this
>>>> communication in error, please contact the sender immediately and delete
it
>>>> from your system. Thank You.

Mime
View raw message