hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhihong Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6529) With HFile v2, the region server will always perform an extra copy of source files
Date Fri, 10 Aug 2012 03:37:10 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432586#comment-13432586
] 

Zhihong Ted Yu commented on HBASE-6529:
---------------------------------------

I should have performed the following search much earlier:
{code}
hbase-server zhihyu$ find src/main/ -name '*.java' -exec grep -i 'fs.*.equals(' {} \; -print
        while (!(argumentOnTopOfStack.equals(ParseConstants.LPAREN_BUFFER))) {
    if (argumentOnTopOfStack.equals(ParseConstants.OR_BUFFER)) {
    } else if (argumentOnTopOfStack.equals(ParseConstants.AND_BUFFER)) {
    } else if (argumentOnTopOfStack.equals(ParseConstants.SKIP_BUFFER)) {
    } else if (argumentOnTopOfStack.equals(ParseConstants.WHILE_BUFFER)) {
    } else if (argumentOnTopOfStack.equals(ParseConstants.LPAREN_BUFFER)) {
src/main//java/org/apache/hadoop/hbase/filter/ParseFilter.java
      final FileSystem fs = (DEFAULT_PATH.equals(parts))
src/main//java/org/apache/hadoop/hbase/mapreduce/hadoopbackport/TotalOrderPartitioner.java
    if (!srcFs.equals(fs)) {
src/main//java/org/apache/hadoop/hbase/regionserver/Store.java
{code}
So Store.java is the only place where HFileSystem is compared against FileSystem.

The patch is valid.

My earlier comment was for encapsulating the call to getBackingFs() so that people can use
the utility in other places.

                
> With HFile v2, the region server will always perform an extra copy of source files
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-6529
>                 URL: https://issues.apache.org/jira/browse/HBASE-6529
>             Project: HBase
>          Issue Type: Bug
>          Components: performance, regionserver
>    Affects Versions: 0.94.0, 0.96.0
>            Reporter: Jason Dai
>         Attachments: hbase-6529.diff
>
>
> With HFile v2 implementation in HBase 0.94 & 0.96, the region server will use HFileSystem
as its {color:blue}fs{color}. When it performs bulk load in Store.bulkLoadHFile(), it checks
if its {color:blue}fs{color} is the same as {color:blue}srcFs{color}, which however will be
DistributedFileSystem. Consequently, it will always perform an extra copy of source files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message