www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bong Chen Lock <bongche...@gmail.com>
Subject Re: ulimit changed with Apache Jenkins upgrade?
Date Wed, 03 Sep 2014 17:24:27 GMT
On 4 Sep 2014 00:24, "Bong Chen Lock" <bongchen38@gmail.com> wrote:

> On 4 Sep 2014 00:21, "Patrick Hunt" <phunt@apache.org> wrote:
>
>> I've opened a BUILDS jira for this:
>> https://issues.apache.org/jira/browse/BUILDS-17
>>
>> On Sat, Aug 30, 2014 at 12:16 PM, Patrick Hunt <phunt@apache.org> wrote:
>> > Giri could you or one of the other folks look into this? Our tests
>> > have been broken for some time.
>> >
>> >
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2303/
>> >
>> > The same test is still failing with the same error (too many open
>> > files) and afaict the ulimit is still just set to 4k. I'm printing the
>> > ulimit info at the start of the job, here it is:
>> >
>> > core file size          (blocks, -c) 0
>> > data seg size           (kbytes, -d) unlimited
>> > scheduling priority             (-e) 0
>> > file size               (blocks, -f) unlimited
>> > pending signals                 (-i) 386177
>> > max locked memory       (kbytes, -l) 64
>> > max memory size         (kbytes, -m) unlimited
>> > open files                      (-n) 4096
>> > pipe size            (512 bytes, -p) 8
>> > POSIX message queues     (bytes, -q) 819200
>> > real-time priority              (-r) 0
>> > stack size              (kbytes, -s) 8192
>> > cpu time               (seconds, -t) unlimited
>> > max user processes              (-u) 386177
>> > virtual memory          (kbytes, -v) unlimited
>> > file locks                      (-x) unlimited
>> >
>> >
>> > Patrick
>> >
>> > On Wed, Jul 23, 2014 at 11:21 AM, Patrick Hunt <phunt@apache.org>
>> wrote:
>> >> Does someone in builds@ have the ability to address this? (restart the
>> >> jenkins slaves so that the new ulimit limits will take effect)
>> >>
>> >> Thanks!
>> >>
>> >> Patrick
>> >>
>> >> On Wed, Jul 23, 2014 at 9:33 AM, Patrick Hunt <phunt@apache.org>
>> wrote:
>> >>> Giri do you mean me? I don't have access to that afaict.
>> >>>
>> >>> Patrick
>> >>>
>> >>> On Wed, Jul 23, 2014 at 12:50 AM, Giridharan Kesavan
>> >>> <gkesavan@hortonworks.com> wrote:
>> >>>> jenkins slaves might need a restart.
>> >>>>
>> >>>> Could you pls re-launch the slaves from the jenkins UI configuration
>> page?
>> >>>>
>> >>>> -giri
>> >>>>
>> >>>>
>> >>>> On Tue, Jul 22, 2014 at 2:25 PM, Patrick Hunt <phunt@apache.org>
>> wrote:
>> >>>>>
>> >>>>> Thanks Giri! Unfortunately though it seems to not have taken
>> effect, I
>> >>>>> just kicked off a precommit build and I see
>> >>>>>
>> >>>>> core file size          (blocks, -c) 0
>> >>>>> data seg size           (kbytes, -d) unlimited
>> >>>>> scheduling priority             (-e) 0
>> >>>>> file size               (blocks, -f) unlimited
>> >>>>> pending signals                 (-i) 386178
>> >>>>> max locked memory       (kbytes, -l) 64
>> >>>>> max memory size         (kbytes, -m) unlimited
>> >>>>> open files                      (-n) 4096
>> >>>>> pipe size            (512 bytes, -p) 8
>> >>>>> POSIX message queues     (bytes, -q) 819200
>> >>>>> real-time priority              (-r) 0
>> >>>>> stack size              (kbytes, -s) 8192
>> >>>>> cpu time               (seconds, -t) unlimited
>> >>>>> max user processes              (-u) 386178
>> >>>>> virtual memory          (kbytes, -v) unlimited
>> >>>>> file locks                      (-x) unlimited
>> >>>>>
>> >>>>> more details here:
>> >>>>>
>> >>>>>
>> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/2215/console
>> >>>>>
>> >>>>> Patrick
>> >>>>>
>> >>>>> On Tue, Jul 22, 2014 at 1:29 PM, Giridharan Kesavan
>> >>>>> <gkesavan@hortonworks.com> wrote:
>> >>>>> >
>> >>>>> >
>> >>>>> > jenkins@asf901:~$ ulimit -a
>> >>>>> > core file size          (blocks, -c) 0
>> >>>>> > data seg size           (kbytes, -d) unlimited
>> >>>>> > scheduling priority             (-e) 0
>> >>>>> > file size               (blocks, -f) unlimited
>> >>>>> > pending signals                 (-i) 386178
>> >>>>> > max locked memory       (kbytes, -l) 64
>> >>>>> > max memory size         (kbytes, -m) unlimited
>> >>>>> > open files                      (-n) 60000
>> >>>>> > pipe size            (512 bytes, -p) 8
>> >>>>> > POSIX message queues     (bytes, -q) 819200
>> >>>>> > real-time priority              (-r) 0
>> >>>>> > stack size              (kbytes, -s) 8192
>> >>>>> > cpu time               (seconds, -t) unlimited
>> >>>>> > max user processes              (-u) 10240
>> >>>>> > virtual memory          (kbytes, -v) unlimited
>> >>>>> > file locks                      (-x) unlimited
>> >>>>> >
>> >>>>> > bumped up the open files and max user processes on all
the slaves.
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > -giri
>> >>>>> >
>> >>>>> >
>> >>>>> > On Tue, Jul 22, 2014 at 12:04 PM, Patrick Hunt <phunt@apache.org>
>> wrote:
>> >>>>> >>
>> >>>>> >> Giri any chance can you take a look at the ulimit issue
on the H#
>> >>>>> >> machines? All the ZK precommit builds are failing as
a result.
>> >>>>> >>
>> >>>>> >> I updated the precommit build last night to output
"ulimit -a"
>> and it
>> >>>>> >> says the current limit is 4096, can we bump that up
or set the
>> default
>> >>>>> >> to unlimited?
>> >>>>> >>
>> >>>>> >> Thanks!
>> >>>>> >>
>> >>>>> >> Patrick
>> >>>>> >>
>> >>>>> >> On Sun, Jul 20, 2014 at 10:32 PM, Rakesh R <rakeshr@huawei.com>
>> wrote:
>> >>>>> >> > +1
>> >>>>> >> >
>> >>>>> >> >
>> >>>>> >> > Adding one more point. I could see the following
error too in
>> the
>> >>>>> >> > pre-commit build.
>> >>>>> >> >
>> >>>>> >> >      [exec]     [junit] Exception in thread
>> "CommitProcWorkThread-16"
>> >>>>> >> > java.lang.NoClassDefFoundError:
>> >>>>> >> > org/apache/zookeeper/server/ConnectionBean
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:159)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:594)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:198)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.quorum.CommitProcessor$CommitWorkRequest.doWork(CommitProcessor.java:295)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:161)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> >
>> >>>>> >> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> >>>>> >> >      [exec]     [junit]         at
>> >>>>> >> > java.lang.Thread.run(Thread.java:662)
>> >>>>> >> >
>> >>>>> >> > -Rakesh
>> >>>>> >> >
>> >>>>> >> > -----Original Message-----
>> >>>>> >> > From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>> >>>>> >> > Sent: 21 July 2014 02:19
>> >>>>> >> > To: dev@zookeeper.apache.org
>> >>>>> >> > Cc: Andrew Bayer; builds@apache.org; Giridharan
Kesavan
>> >>>>> >> > Subject: Re: ulimit changed with Apache Jenkins
upgrade?
>> >>>>> >> >
>> >>>>> >> > +1
>> >>>>> >> >
>> >>>>> >> > On 18 Jul 2014, at 18:59, Patrick Hunt <phunt@apache.org>
>> wrote:
>> >>>>> >> >
>> >>>>> >> >> Hi builds folks, is this a system wide issue
or something we
>> should
>> >>>>> >> >> address ourselves? Thanks!
>> >>>>> >> >>
>> >>>>> >> >> Patrick
>> >>>>> >> >>
>> >>>>> >> >> ---------- Forwarded message ----------
>> >>>>> >> >> From: Patrick Hunt <phunt@apache.org>
>> >>>>> >> >> Date: Fri, Jul 18, 2014 at 10:38 AM
>> >>>>> >> >> Subject: ulimit changed with Apache Jenkins
upgrade?
>> >>>>> >> >> To: Giridharan Kesavan <gkesavan@hortonworks.com>
>> >>>>> >> >> Cc: DevZooKeeper <dev@zookeeper.apache.org>,
Andrew Bayer
>> >>>>> >> >> <andrew@cloudera.com>
>> >>>>> >> >>
>> >>>>> >> >>
>> >>>>> >> >> Hi Giri, can you check that the new hosts
(H#) have the
>> ulimit set
>> >>>>> >> >> to
>> >>>>> >> >> what it was set to on the original hadoop#
hosts? I'm seeing
>> new
>> >>>>> >> >> test
>> >>>>> >> >> failures with
>> >>>>> >> >>
>> >>>>> >> >>     [exec]     [junit] java.io.FileNotFoundException:
>> >>>>> >> >>
>> >>>>> >> >>
>> /home/jenkins/jenkins-slave/workspace/PreCommit-ZOOKEEPER-Build/trunk/
>> >>>>> >> >>
>> >>>>> >> >>
>> build/test/tmp/test7610638215300246179.junit.dir/version-2/log.1000000
>> >>>>> >> >> 01
>> >>>>> >> >> (Too many open files)
>> >>>>> >> >>
>> >>>>> >> >> which we've not seen before. I believe this
means that we
>> running
>> >>>>> >> >> out
>> >>>>> >> >> of file descriptors?
>> >>>>> >> >>
>> >>>>> >> >> Can you verify and address if possible?
>> >>>>> >> >>
>> >>>>> >> >> Thanks,
>> >>>>> >> >>
>> >>>>> >> >> Patrick
>> >>>>> >> >
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > CONFIDENTIALITY NOTICE
>> >>>>> > NOTICE: This message is intended for the use of the individual
or
>> entity
>> >>>>> > to
>> >>>>> > which it is addressed and may contain information that
is
>> confidential,
>> >>>>> > privileged and exempt from disclosure under applicable
law. If the
>> >>>>> > reader of
>> >>>>> > this message is not the intended recipient, you are hereby
>> notified that
>> >>>>> > any
>> >>>>> > printing, copying, dissemination, distribution, disclosure
or
>> forwarding
>> >>>>> > of
>> >>>>> > this communication is strictly prohibited. If you have
received
>> this
>> >>>>> > communication in error, please contact the sender immediately
and
>> delete
>> >>>>> > it
>> >>>>> > from your system. Thank You.
>> >>>>
>> >>>>
>> >>>>
>> >>>> CONFIDENTIALITY NOTICE
>> >>>> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >>>> which it is addressed and may contain information that is
>> confidential,
>> >>>> privileged and exempt from disclosure under applicable law. If the
>> reader of
>> >>>> this message is not the intended recipient, you are hereby notified
>> that any
>> >>>> printing, copying, dissemination, distribution, disclosure or
>> forwarding of
>> >>>> this communication is strictly prohibited. If you have received
this
>> >>>> communication in error, please contact the sender immediately and
>> delete it
>> >>>> from your system. Thank You.
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message