hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Appy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18059) The scanner order for memstore scanners are wrong
Date Mon, 05 Jun 2017 16:16:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037151#comment-16037151

Appy commented on HBASE-18059:

Looking at things more closely after [~tianjingyun]'s comment. HBASE-15236 is year old, so
had to dig deep again.
If the following invariant always holds, then you're right. [~stack] can you  please confirm,
since my knowledge of seqId isn't much.
"Seq id of cells in memstore will always be great than that of any store file or bulk loaded
Detailed analysis:

KVHeap uses [KVScannerComparator|https://github.com/apache/hbase/blob/e65d8653e566bbbae03578a1f9ad858cabcb48bc/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java#L191].
It first compares the cells from two scanners using a CellComparator. Only when the comparison
is equal, it uses scanner.getScannerOrder().

Thinking about the case when two scanners in a KeyValueHeap might return the same key, KVScannerComparator
will resolve ties as follows:
- memstore scanners vs storefile(SF) scanner:
 -- if cell in SF has seqId: Its seqId should be less than that of memstore's cell. Memstore
scanner will win. 
 -- if cell in SF does NOT have seqId: SF'cell [defaults to seqId=0|https://github.com/apache/hbase/blob/e65d8653e566bbbae03578a1f9ad858cabcb48bc/hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java#L142],
memstore scanner will win. (Note that seqId of cells in hfiles are removed on major compaction
if older than certain time, default is 5 days)
- memstore vs bulk loaded file(BLF)
 -- memstore cell's will have higher seqId, so memstore scanner will win.
- SF vs SF -> Either cells' seqId will be used to tie break, or getScannerOrder() (determined
by store files' seq ids).
- SF vs BLF
 -- if both cells have seqId -> one with higher seqId wins.
  -- BLF's cell has seqId and SF's cell doesn't -> BLF scanner wins since SF's cell's seqId
defaults to 0. (This will happen when bulk load file is new but SF is result of major compaction
where cell's seqId were erased).
 -- SF's cell has seqId but BLF's cell doesn't -> can't happen. BLF always have seqId.
- BLF vs BLF
 -- Cell's seqid will be used to  tie break.

> The scanner order for memstore scanners are wrong
> -------------------------------------------------
>                 Key: HBASE-18059
>                 URL: https://issues.apache.org/jira/browse/HBASE-18059
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver, scan, Scanners
>    Affects Versions: 2.0.0
>            Reporter: Duo Zhang
>            Assignee: Jingyun Tian
>            Priority: Critical
>             Fix For: 2.0.0
> This is comments for KeyValueScanner.getScannerOrder
> {code:title=KeyValueScanner.java}
>   /**
>    * Get the order of this KeyValueScanner. This is only relevant for StoreFileScanners
>    * MemStoreScanners (other scanners simply return 0). This is required for comparing
>    * files to find out which one has the latest data. StoreFileScanners are ordered from
>    * (oldest) to newest in increasing order. MemStoreScanner gets LONG.max since it always
>    * contains freshest data.
>    */
>   long getScannerOrder();
> {code}
> As now we may have multiple memstore scanners, I think the right way to select scanner
order for memstore scanner is to ordered from Long.MAX_VALUE in decreasing order.
> But in CompactingMemStore and DefaultMemStore, the scanner order for memstore scanner
is also start from 0, which will be messed up with StoreFileScanners.

This message was sent by Atlassian JIRA

View raw message