Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 18 Aug 2014 20:21:18 +0000 (UTC)
From: "Jerry He (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12734841.1408392503234.25178.1408393278771@arcas>
In-Reply-To: <JIRA.12734841.1408392503234@arcas>
References: <JIRA.12734841.1408392503234@arcas>
Subject: [jira] [Commented] (HBASE-11772) Bulk load mvcc and seqId issues
 with native hfiles
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101214#comment-14101214 ] 

Jerry He commented on HBASE-11772:
----------------------------------

Here is the proposed fix:

1) Better detection of bulk loaded files.  We can use the loaded file name with '_SeqId_' since we already use it as marker to get the load time seqId.
2)  Regard bulk loaded files always have mvcc 0.  Don't call StoreFileScanner.skipKVsNewerThanReadpoint() during scan if it is bulk loaded file no matter whether or not it has mvcc in the file.
3) The problem 3 will probably be fixed by 1).

> Bulk load mvcc and seqId issues with native hfiles
> --------------------------------------------------
>
>                 Key: HBASE-11772
>                 URL: https://issues.apache.org/jira/browse/HBASE-11772
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.98.5
>            Reporter: Jerry He
>
> There are mvcc and seqId issues when bulk load native hfiles -- meaning hfiles that are direct file copy-out from hbase, not from HFileOutputFormat job.
> There are differences between these two types of hfiles.
> Native hfiles have possible non-zero MAX_MEMSTORE_TS_KEY value and non-zero mvcc values in cells. 
> Native hfiles also have MAX_SEQ_ID_KEY.
> Native hfiles do not have BULKLOAD_TIME_KEY.
> Here are a couple of problems I observed when bulk load native hfiles.
> 1.  Cells in newly bulk loaded hfiles can be invisible to scan.
> It is easy to re-create.
> Bulk load a native hfile that has a larger mvcc value in cells, e.g 10
> If the current readpoint when initiating a scan is less than 10, the cells in the new hfile are skipped, thus become invisible.
> We don't reset the readpoint of a region after bulk load.
> 2. The current StoreFile.isBulkLoadResult() is implemented as:
> {code}
> return metadataMap.containsKey(BULKLOAD_TIME_KEY)
> {code}
> which does not detect bulkloaded native hfiles.
> 3. Another observed problem is possible data loss during log recovery. 
> It is similar to HBASE-10958 reported by [~jdcryans]. Borrow the re-create steps from HBASE-10958.
> 1) Create an empty table
> 2) Put one row in it (let's say it gets seqid 1)
> 3) Bulk load one native hfile with large seqId ( e.g. 100). The native hfile can be obtained by copying out from existing table.
> 4) Kill the region server that holds the table's region.
> Scan the table once the region is made available again. The first row, at seqid 1, will be missing since the HFile with seqid 100 makes us believe that everything that came before it was flushed. 
> The problem 3 is probably related to 2. We will be ok if we get the appended seqId during bulk load instead of 100 from inside the file.


--
This message was sent by Atlassian JIRA
(v6.2#6252)