www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bong Chen Lock <bongche...@gmail.com>
Subject Re: ulimit changed with Apache Jenkins upgrade?
Date Wed, 03 Sep 2014 17:24:03 GMT
On 4 Sep 2014 00:21, "Patrick Hunt" <phunt@apache.org> wrote:

> I've opened a BUILDS jira for this:
> https://issues.apache.org/jira/browse/BUILDS-17
>
> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <phunt@apache.org> wrote:
> > Giri could you or one of the other folks look into this? Our tests
> > have been broken for some time.
> >
> >
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
> >
> > The same test is still failing with the same error (too many open
> > files) and afaict the ulimit is still just set to 4k. I'm printing the
> > ulimit info at the start of the job, here it is:
> >
> > core file size          (blocks, -c) 0
> > data seg size           (kbytes, -d) unlimited
> > scheduling priority             (-e) 0
> > file size               (blocks, -f) unlimited
> > pending signals                 (-i) 386177
> > max locked memory       (kbytes, -l) 64
> > max memory size         (kbytes, -m) unlimited
> > open files                      (-n) 4096
> > pipe size            (512 bytes, -p) 8
> > POSIX message queues     (bytes, -q) 819200
> > real-time priority              (-r) 0
> > stack size              (kbytes, -s) 8192
> > cpu time               (seconds, -t) unlimited
> > max user processes              (-u) 386177
> > virtual memory          (kbytes, -v) unlimited
> > file locks                      (-x) unlimited
> >
> >
> > Patrick
> >
> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <phunt@apache.org> wrote:
> >> Does someone in builds@ have the ability to address this? (restart the
> >> jenkins slaves so that the new ulimit limits will take effect)
> >>
> >> Thanks!
> >>
> >> Patrick
> >>
> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <phunt@apache.org> wrote:
> >>> Giri do you mean me? I don't have access to that afaict.
> >>>
> >>> Patrick
> >>>
> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
> >>> <gkesavan@hortonworks.com> wrote:
> >>>> jenkins slaves might need a restart.
> >>>>
> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
> page?
> >>>>
> >>>> -giri
> >>>>
> >>>>
> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <phunt@apache.org>
> wrote:
> >>>>>
> >>>>> Thanks Giri! Unfortunately though it seems to not have taken effect,
> I
> >>>>> just kicked off a precommit build and I see
> >>>>>
> >>>>> core file size          (blocks, -c) 0
> >>>>> data seg size           (kbytes, -d) unlimited
> >>>>> scheduling priority             (-e) 0
> >>>>> file size               (blocks, -f) unlimited
> >>>>> pending signals                 (-i) 386178
> >>>>> max locked memory       (kbytes, -l) 64
> >>>>> max memory size         (kbytes, -m) unlimited
> >>>>> open files                      (-n) 4096
> >>>>> pipe size            (512 bytes, -p) 8
> >>>>> POSIX message queues     (bytes, -q) 819200
> >>>>> real-time priority              (-r) 0
> >>>>> stack size              (kbytes, -s) 8192
> >>>>> cpu time               (seconds, -t) unlimited
> >>>>> max user processes              (-u) 386178
> >>>>> virtual memory          (kbytes, -v) unlimited
> >>>>> file locks                      (-x) unlimited
> >>>>>
> >>>>> more details here:
> >>>>>
> >>>>>
> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
> >>>>>
> >>>>> Patrick
> >>>>>
> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
> >>>>> <gkesavan@hortonworks.com> wrote:
> >>>>> >
> >>>>> >
> >>>>> > jenkins@asf901:~$ ulimit -a
> >>>>> > core file size          (blocks, -c) 0
> >>>>> > data seg size           (kbytes, -d) unlimited
> >>>>> > scheduling priority             (-e) 0
> >>>>> > file size               (blocks, -f) unlimited
> >>>>> > pending signals                 (-i) 386178
> >>>>> > max locked memory       (kbytes, -l) 64
> >>>>> > max memory size         (kbytes, -m) unlimited
> >>>>> > open files                      (-n) 60000
> >>>>> > pipe size            (512 bytes, -p) 8
> >>>>> > POSIX message queues     (bytes, -q) 819200
> >>>>> > real-time priority              (-r) 0
> >>>>> > stack size              (kbytes, -s) 8192
> >>>>> > cpu time               (seconds, -t) unlimited
> >>>>> > max user processes              (-u) 10240
> >>>>> > virtual memory          (kbytes, -v) unlimited
> >>>>> > file locks                      (-x) unlimited
> >>>>> >
> >>>>> > bumped up the open files and max user processes on all the
slaves.
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -giri
> >>>>> >
> >>>>> >
> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <phunt@apache.org>
> wrote:
> >>>>> >>
> >>>>> >> Giri any chance can you take a look at the ulimit issue
on the H#
> >>>>> >> machines? All the ZK precommit builds are failing as a
result.
> >>>>> >>
> >>>>> >> I updated the precommit build last night to output "ulimit
-a"
> and it
> >>>>> >> says the current limit is 4096, can we bump that up or
set the
> default
> >>>>> >> to unlimited?
> >>>>> >>
> >>>>> >> Thanks!
> >>>>> >>
> >>>>> >> Patrick
> >>>>> >>
> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <rakeshr@huawei.com>
> wrote:
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Adding one more point. I could see the following error
too in
> the
> >>>>> >> > pre-commit build.
> >>>>> >> >
> >>>>> >> >      [exec]     [junit] Exception in thread
> "CommitProcWorkThread-16"
> >>>>> >> > java.lang.NoClassDefFoundError:
> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> >
> >>>>> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>>>> >> >      [exec]     [junit]         at
> >>>>> >> > java.lang.Thread.run(Thread.java:662)
> >>>>> >> >
> >>>>> >> > -Rakesh
> >>>>> >> >
> >>>>> >> > -----Original Message-----
> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
> >>>>> >> > Sent: 21 July 2014 02:19
> >>>>> >> > To: dev@zookeeper.apache.org
> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan Kesavan
> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >
> >>>>> >> > +1
> >>>>> >> >
> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <phunt@apache.org>
> wrote:
> >>>>> >> >
> >>>>> >> >> Hi builds folks, is this a system wide issue or
something we
> should
> >>>>> >> >> address ourselves? Thanks!
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >>
> >>>>> >> >> ---------- Forwarded message ----------
> >>>>> >> >> From: Patrick Hunt <phunt@apache.org>
> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
> >>>>> >> >> Subject: ulimit changed with Apache Jenkins upgrade?
> >>>>> >> >> To: Giridharan Kesavan <gkesavan@hortonworks.com>
> >>>>> >> >> Cc: DevZooKeeper <dev@zookeeper.apache.org>,
Andrew Bayer
> >>>>> >> >> <andrew@cloudera.com>
> >>>>> >> >>
> >>>>> >> >>
> >>>>> >> >> Hi Giri, can you check that the new hosts (H#)
have the ulimit
> set
> >>>>> >> >> to
> >>>>> >> >> what it was set to on the original hadoop# hosts?
I'm seeing
> new
> >>>>> >> >> test
> >>>>> >> >> failures with
> >>>>> >> >>
> >>>>> >> >>     [exec]     [junit] java.io.FileNotFoundException:
> >>>>> >> >>
> >>>>> >> >>
> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
> >>>>> >> >>
> >>>>> >> >>
> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
> >>>>> >> >> 01
> >>>>> >> >> (Too many open files)
> >>>>> >> >>
> >>>>> >> >> which we've not seen before. I believe this means
that we
> running
> >>>>> >> >> out
> >>>>> >> >> of file descriptors?
> >>>>> >> >>
> >>>>> >> >> Can you verify and address if possible?
> >>>>> >> >>
> >>>>> >> >> Thanks,
> >>>>> >> >>
> >>>>> >> >> Patrick
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > CONFIDENTIALITY NOTICE
> >>>>> > NOTICE: This message is intended for the use of the individual
or
> entity
> >>>>> > to
> >>>>> > which it is addressed and may contain information that is
> confidential,
> >>>>> > privileged and exempt from disclosure under applicable law.
If the
> >>>>> > reader of
> >>>>> > this message is not the intended recipient, you are hereby
> notified that
> >>>>> > any
> >>>>> > printing, copying, dissemination, distribution, disclosure
or
> forwarding
> >>>>> > of
> >>>>> > this communication is strictly prohibited. If you have received
> this
> >>>>> > communication in error, please contact the sender immediately
and
> delete
> >>>>> > it
> >>>>> > from your system. Thank You.
> >>>>
> >>>>
> >>>>
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> entity to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> reader of
> >>>> this message is not the intended recipient, you are hereby notified
> that any
> >>>> printing, copying, dissemination, distribution, disclosure or
> forwarding of
> >>>> this communication is strictly prohibited. If you have received this
> >>>> communication in error, please contact the sender immediately and
> delete it
> >>>> from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message