hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allan Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18203) Intelligently manage a pool of open references to store files
Date Sat, 10 Jun 2017 15:37:19 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045581#comment-16045581

Allan Yang commented on HBASE-18203:

It is a very useful feature. We often encounter "open too many files" exception if storefile
number is too many on a single RS. the RS process holds too many handlers(tcp connections)
and grows unboundedly with storefile numbers.

> Intelligently manage a pool of open references to store files
> -------------------------------------------------------------
>                 Key: HBASE-18203
>                 URL: https://issues.apache.org/jira/browse/HBASE-18203
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>    Affects Versions: 2.0.0
>            Reporter: Andrew Purtell
> When bringing a region online we open every store file and keep the file open, to avoid
further round trips to the HDFS namenode during reads. Naively keeping open every store file
we encounter is a bad idea. There should be an upper bound. We should close and reopen files
as needed once we are above the upper bound. We should choose candidates to close on a LRU
basis. Otherwise we can (and some users have in production) overrun high (~64k) open file
handle limits on the server if the aggregate number of store files is too large. 
> Note the 'open files' here refers to open/active references to files at the HDFS level.
How this maps to active file descriptors at the OS level depends on concurrency of access
(block transfers, short circuit reads). The more open files we have at the HDFS level the
higher number of OS level file handles we can expect to consume.

This message was sent by Atlassian JIRA

View raw message