hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4965) Monitor the open file descriptors and the threads counters during the unit tests
Date Sat, 10 Dec 2011 17:29:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166912#comment-13166912
] 

stack commented on HBASE-4965:
------------------------------

This is interesting.  When I changed the hadoop-qa build to add build of long tests too, I
inserted another output of ulimit that runs after checkout and patch application and this
time fds limits is 1024 as you found above nkeywal.  Let me talk to Giri, the hadoopqa master:

{code}
/home/jenkins/tools/maven/latest/bin/mvn clean test -DHbasePatchProcess
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2048
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
{code}
                
> Monitor the open file descriptors and the threads counters during the unit tests
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-4965
>                 URL: https://issues.apache.org/jira/browse/HBASE-4965
>             Project: HBase
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.94.0
>         Environment: all
>            Reporter: nkeywal
>            Assignee: nkeywal
>            Priority: Minor
>         Attachments: 4965-v2.txt, 4965_all.patch, 4965_all.patch, 4965_all.patch, ResourceChecker.java,
ResourceCheckerJUnitRule.java
>
>
> We're seeing a lot of issues with hadoop-qa related to threads or file descriptors.
> Monitoring these counters would ease the analysis.
> Note as well that
>  - if we want to execute the tests in the same jvm (because the test is small or because
we want to share the cluster) we can't afford to leak too many resources
>  - if the tests leak, it's more difficult to detect a leak in the software itself.
> I attach piece of code that I used. It requires two lines in a unit test class to:
> - before every test, count the threads and the open file descriptor
> - after every test, compare with the previous value.
> I ran it on some tests; we have for example:
> - client.TestMultiParallel#testBatchWithManyColsInOneRowGetAndPut: 232 threads (was 231),
390 file descriptors (was 390). => TestMultiParallel uses 232 threads!
> - client.TestMultipleTimestamps#testWithColumnDeletes: 152 threads (was 151), 283 file
descriptors (was 282).
> - client.TestAdmin#testCheckHBaseAvailableClosesConnection: 477 threads (was 294), 815
file descriptors (was 461)
> - client.TestMetaMigrationRemovingHTD#testMetaMigration: 149 threads (was 148), 310 file
descriptors (was 307).
> It's not always leaks, we can expect some pooling effects. But still...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message