Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 25 Mar 2011 21:05:05 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: 
 <1766622180.12341.1301087105930.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1600369857.6638.1300901885940.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HBASE-3693) isMajorCompaction() check triggers
 lots of listStatus DFS RPC calls from HBase
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-3693.
--------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

Committed to TRUNK (not to branch since only an 'improvement').  Nice patch Liyin.  Thanks.

> isMajorCompaction() check triggers lots of listStatus DFS RPC calls from HBase
> ------------------------------------------------------------------------------
>
>                 Key: HBASE-3693
>                 URL: https://issues.apache.org/jira/browse/HBASE-3693
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Kannan Muthukkaruppan
>            Assignee: Liyin Tang
>         Attachments: Hbase-3693[r1085248]_2.patch, Hbase-3693[r1085306].patch
>
>
> We noticed that there are lots of listStatus calls on the ColumnFamily directories within each region, coming from this codepath:
> {code}
> compactionSelection()
>  --> isMajorCompaction 
>     --> getLowestTimestamp()
>        -->  FileStatus[] stats = fs.listStatus(p);
> {code}
> So on every compactionSelection() we're taking this hit. While not immediately an issue, just from log inspection, this accounts for quite a large number of RPCs to namenode at the moment and seems like an unnecessary load to be sending to the namenode.
> Seems like it would be easy to cache the timestamp for each opened/created StoreFile, in memory, in the region server, and avoid going to DFS each time for this information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira