hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3693) isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase
Date Fri, 25 Mar 2011 04:05:05 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011088#comment-13011088

stack commented on HBASE-3693:

I like your patch Liyin.  Can I commit it? (I'm nervous about committing your stuff -- I seem
to be jumping the gun).

I'd fix this misspelling: stroeFileNum 

Also, why a setModificationTimeStamp?  Seems odd being able to set this?  It comes from the

I'd like to get this into 0.90.2.

> isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase
> ------------------------------------------------------------------------------
>                 Key: HBASE-3693
>                 URL: https://issues.apache.org/jira/browse/HBASE-3693
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Liyin Tang
>         Attachments: Hbase-3693[r1085306].patch
> We noticed that there are lots of listStatus calls on the ColumnFamily directories within
each region, coming from this codepath:
> {code}
> compactionSelection()
>  --> isMajorCompaction 
>     --> getLowestTimestamp()
>        -->  FileStatus[] stats = fs.listStatus(p);
> {code}
> So on every compactionSelection() we're taking this hit. While not immediately an issue,
just from log inspection, this accounts for quite a large number of RPCs to namenode at the
moment and seems like an unnecessary load to be sending to the namenode.
> Seems like it would be easy to cache the timestamp for each opened/created StoreFile,
in memory, in the region server, and avoid going to DFS each time for this information.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message