hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kannan Muthukkaruppan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-3693) isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase
Date Wed, 23 Mar 2011 17:38:05 GMT
isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase
------------------------------------------------------------------------------

                 Key: HBASE-3693
                 URL: https://issues.apache.org/jira/browse/HBASE-3693
             Project: HBase
          Issue Type: Improvement
            Reporter: Kannan Muthukkaruppan
            Assignee: Liyin Tang


We noticed that are lots of listStatus calls on the ColumnFamily directories within each regions,
coming from this codepath:

{code}
compactionSelection()
 --> isMajorCompaction 
    --> getLowestTimestamp()
       -->  FileStatus[] stats = fs.listStatus(p);
{code}

So on every compactionSelection() we're taking this hit. While not immediately an issue, just
from log inspection, this accounts for quite a large number of RPCs to namenode at the moment
and seems like an unnecessary load to be sending to the namenode.

Seems like it would be easy to cache the timestamp for each opened/created StoreFile, in memory,
in the region server, and avoid going to DFS each time for this information.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message