hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3693) isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase
Date Fri, 25 Mar 2011 04:05:05 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011088#comment-13011088
] 

stack commented on HBASE-3693:
------------------------------

I like your patch Liyin.  Can I commit it? (I'm nervous about committing your stuff -- I seem
to be jumping the gun).

I'd fix this misspelling: stroeFileNum 

Also, why a setModificationTimeStamp?  Seems odd being able to set this?  It comes from the
fs?

I'd like to get this into 0.90.2.

> isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase
> ------------------------------------------------------------------------------
>
>                 Key: HBASE-3693
>                 URL: https://issues.apache.org/jira/browse/HBASE-3693
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Liyin Tang
>         Attachments: Hbase-3693[r1085306].patch
>
>
> We noticed that there are lots of listStatus calls on the ColumnFamily directories within
each region, coming from this codepath:
> {code}
> compactionSelection()
>  --> isMajorCompaction 
>     --> getLowestTimestamp()
>        -->  FileStatus[] stats = fs.listStatus(p);
> {code}
> So on every compactionSelection() we're taking this hit. While not immediately an issue,
just from log inspection, this accounts for quite a large number of RPCs to namenode at the
moment and seems like an unnecessary load to be sending to the namenode.
> Seems like it would be easy to cache the timestamp for each opened/created StoreFile,
in memory, in the region server, and avoid going to DFS each time for this information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message