hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <lhofha...@yahoo.com>
Subject Re: max number of open files when testing HBase on the Jenkins server
Date Sun, 27 Nov 2011 01:01:01 GMT
I agree. The tests should not need more than 32k open files.



----- Original Message -----
From: Todd Lipcon <todd@cloudera.com>
To: dev@hbase.apache.org
Cc: 
Sent: Saturday, November 26, 2011 10:42 AM
Subject: Re: max number of open files when testing HBase on the Jenkins server

I disagree that we should raise the limit. If we're using up 32k
descriptors in unit tests, it's probably a real leak we should be
concerned about. Maybe we can add a temporary patch upstream which
does:
catch (IOException ioe) {
  if (ioe.getMessage().contains("Too many open files")) {
    Runtime.getRuntime().exec("lsof -u " + System.getProperty("user.name"));
  }
}
or something like that?

-Todd

On Thu, Nov 24, 2011 at 6:27 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> Mikhail:
> If you look at
> https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you would
> see:
>
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 32768
>
> It is not clear why 32768 was not enough for TestAdmin.
>
> Normally we don't regard test failure resulted from 'Too many open files'
> as real test failure.
>
> I agree the effective limit should be raised.
>
> Cheers
>
> On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin <
> bautin.mailing.lists@gmail.com> wrote:
>
>> Hello,
>>
>> I saw this error when testing my patch on Jenkins:
>>
>> Caused by: java.io.IOException: Too many open files
>>        at sun.nio.ch.IOUtil.initPipe(Native Method)
>>        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>        at
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>        at java.nio.channels.Selector.open(Selector.java:209)
>>        at
>> org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
>>        at sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown
>> Source)
>>        at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>        at java.lang.Class.newInstance0(Class.java:355)
>>        at java.lang.Class.newInstance(Class.java:308)
>>        at
>> org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
>>        ... 55 more
>>
>> (Full log:
>> https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/
>> )
>>
>> I am not sure who maintains the Jenkins server, but it would be nice to
>> increase the maximum number of open files for HBase unit tests.
>>
>> Thanks,
>> --Mikhail
>>
>



-- 
Todd Lipcon
Software Engineer, Cloudera


Mime
View raw message