hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
Date Fri, 11 Dec 2015 09:05:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052479#comment-15052479
] 

Anoop Sam John commented on HBASE-14895:
----------------------------------------

one issue found wrt maintain the scanners in a new List in StoreScanner (These are memstore
scanner and store file scanners at the begin of the scan)
Over the scan time, some of the scanners would have reached its end and we would have eventually
closed them off. And the PriorityQ in KVHeap would have removed them as well.  But we are
still holding those refs in this new List in Store Scanner.
Now after a flush we will use this new List to recreate new Heap. We will remove old MemstoreScaner
and add new MemstoreScanner as well as scanner to the new flushed file. And use all of them
to create new KVHeap. (This might include already closed StoreFileScanners as well).
In KVHeap constructor 
{code}
KeyValueHeap(List<? extends KeyValueScanner> scanners,
      KVScannerComparator comparator) throws IOException {
    this.comparator = comparator;
    if (!scanners.isEmpty()) {
      this.heap = new PriorityQueue<KeyValueScanner>(scanners.size(),
          this.comparator);
      for (KeyValueScanner scanner : scanners) {
        if (scanner.peek() != null) {
          this.heap.add(scanner);
        } else {
          this.scannersForDelayedClose.add(scanner);
        }
      }
      this.current = pollRealKV();
    }
  }
  {code}
So the closed scanners wil have peek Cell as null we wont add them to heap. Good. But we will
add them to delayed close and after one shipped call we will try close them again
StoreFileScanner#close()
{code}
public void close() {
    cur = null;
    this.hfs.close();
    if (this.reader != null) {
      this.reader.decrementRefCount();
    }
  }
  {code}
No impact because of hfs close. But the decr read ref count will be a problem and can cause
issue.

In a simple way we can we can have closed flag in StoreFileScanner which can be used to check
when close call on second time or close call nullifying the reader and hfs on StoreFileScanner.

Or else if we dont need to maintain a new List of scanners in StoreScanner and make use of
the current scanners in the KVHeap for recreating the new heap, that would have looked better.
No need to maintain same scanners in 2 data structures.


> Seek only to the newly flushed file on scanner reset on flush
> -------------------------------------------------------------
>
>                 Key: HBASE-14895
>                 URL: https://issues.apache.org/jira/browse/HBASE-14895
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 2.0.0
>
>         Attachments: HBASE-14895.patch, HBASE-14895_1.patch, HBASE-14895_1.patch, HBASE-14895_2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message