hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: max number of open files when testing HBase on the Jenkins server
Date Sun, 27 Nov 2011 18:13:24 GMT
Looks like hunting down FD leak would be a good candidate for Tuesday's
Hackathon :)

On Sun, Nov 27, 2011 at 9:56 AM, Todd Lipcon <todd@cloudera.com> wrote:

> On Sun, Nov 27, 2011 at 9:29 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > We have many issues to tackle, build failure being one of them.
> >
> > Trying to reduce number of descriptors to 32k could be a goal for the
> > future. For the time being, its priority is not high.
>
> My feeling is that this is a bug - the build failure is doing exactly
> what it's supposed to: alerting us of a bug. If the builds are using
> this many FDs, it's likely a leak -- just like the "out of memory"
> error last week was actually do to an infinite loop spewing exceptions
> in a test.
>
> Avoiding these errors by raising limits just hides the bug so we find
> it later in production. Unless we have a good explanation that shows
> that it's "just a test issue" we shouldn't ignore it or hide it by
> raising ulimit.
>
> -Todd
>
> >
> > On Sat, Nov 26, 2011 at 5:01 PM, lars hofhansl <lhofhansl@yahoo.com>
> wrote:
> >
> >> I agree. The tests should not need more than 32k open files.
> >>
> >>
> >>
> >> ----- Original Message -----
> >> From: Todd Lipcon <todd@cloudera.com>
> >> To: dev@hbase.apache.org
> >> Cc:
> >> Sent: Saturday, November 26, 2011 10:42 AM
> >> Subject: Re: max number of open files when testing HBase on the Jenkins
> >> server
> >>
> >> I disagree that we should raise the limit. If we're using up 32k
> >> descriptors in unit tests, it's probably a real leak we should be
> >> concerned about. Maybe we can add a temporary patch upstream which
> >> does:
> >> catch (IOException ioe) {
> >>   if (ioe.getMessage().contains("Too many open files")) {
> >>     Runtime.getRuntime().exec("lsof -u " + System.getProperty("
> user.name
> >> "));
> >>   }
> >> }
> >> or something like that?
> >>
> >> -Todd
> >>
> >> On Thu, Nov 24, 2011 at 6:27 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >> > Mikhail:
> >> > If you look at
> >> > https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you
> >> would
> >> > see:
> >> >
> >> > max memory size         (kbytes, -m) unlimited
> >> > open files                      (-n) 32768
> >> >
> >> > It is not clear why 32768 was not enough for TestAdmin.
> >> >
> >> > Normally we don't regard test failure resulted from 'Too many open
> files'
> >> > as real test failure.
> >> >
> >> > I agree the effective limit should be raised.
> >> >
> >> > Cheers
> >> >
> >> > On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin <
> >> > bautin.mailing.lists@gmail.com> wrote:
> >> >
> >> >> Hello,
> >> >>
> >> >> I saw this error when testing my patch on Jenkins:
> >> >>
> >> >> Caused by: java.io.IOException: Too many open files
> >> >>        at sun.nio.ch.IOUtil.initPipe(Native Method)
> >> >>        at
> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >> >>        at
> >> >>
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >> >>        at java.nio.channels.Selector.open(Selector.java:209)
> >> >>        at
> >> >>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
> >> >>        at
> sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown
> >> >> Source)
> >> >>        at
> >> >>
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >>        at
> >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> >>        at java.lang.Class.newInstance0(Class.java:355)
> >> >>        at java.lang.Class.newInstance(Class.java:308)
> >> >>        at
> >> >>
> org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
> >> >>        ... 55 more
> >> >>
> >> >> (Full log:
> >> >>
> >>
> https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/
> >> >> )
> >> >>
> >> >> I am not sure who maintains the Jenkins server, but it would be nice
> to
> >> >> increase the maximum number of open files for HBase unit tests.
> >> >>
> >> >> Thanks,
> >> >> --Mikhail
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message