hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amitanand Aiyer (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-4485) Eliminate window of missing Data
Date Mon, 26 Sep 2011 22:47:12 GMT

     [ https://issues.apache.org/jira/browse/HBASE-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Amitanand Aiyer updated HBASE-4485:
-----------------------------------

    Description: 
After incorporating v11 of the 2856 fix, we discovered that we are still having some ACID
violations.

This time, however, the problem is not about including "newer" updates; but, about missing
older updates
that should be including. 

Here is what seems to be happening.

There is a race condition in the StoreScanner.getScanners()

  private List<KeyValueScanner> getScanners(Scan scan,
      final NavigableSet<byte[]> columns) throws IOException {
    // First the store file scanners
    List<StoreFileScanner> sfScanners = StoreFileScanner
      .getScannersForStoreFiles(store.getStorefiles(), cacheBlocks,
                                isGet, false);
    List<KeyValueScanner> scanners =
      new ArrayList<KeyValueScanner>(sfScanners.size()+1);

    // include only those scan files which pass all filters
    for (StoreFileScanner sfs : sfScanners) {
      if (sfs.shouldSeek(scan, columns)) {
        scanners.add(sfs);
      }
    }

    // Then the memstore scanners
    if (this.store.memstore.shouldSeek(scan)) {
      scanners.addAll(this.store.memstore.getScanners());
    }
    return scanners;
  }


If for example there is a call to Store.updateStorefiles() that happens between
the store.getStorefiles() and this.store.memstore.getScanners(); then
it is possible that there was a new HFile created, that is not seen by the
StoreScanner, and the data is not present in the Memstore.snapshot either.


  was:
After incorporating v11 of the 2856 fix, we discovered that we are still having some ACID
violations.

This time, however, the problem is not about including "newer" updates; but, about missing
older updates
that should be including. 

Here is what seems to be happing.


0 - Scanner starts scanning.

0 - MemStore.snapshot is called.

    Scanner has access to kvHeap and snapshot

1-  Flush takes place. 
     1.1 KV's in the snapshot are written to the disk.
     1.2 HFile is ready. 

2   Store.updateStoreFiles() deletes the old snapshot.
    
     2.1 updateReaders will not be called until the end of the columnFamily seek.

3  For a brief window of time, scanner does not have access to certain KeyValues.
   a) Scanner has no longer access to the snapshot because it is flushed to the
disk. 
   b) It does not yet have access to the HFile because the updateReaders was
not called yet.
        

    
> Eliminate window of missing Data
> --------------------------------
>
>                 Key: HBASE-4485
>                 URL: https://issues.apache.org/jira/browse/HBASE-4485
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Amitanand Aiyer
>            Assignee: Amitanand Aiyer
>             Fix For: 0.94.0
>
>         Attachments: 4485-v1.diff
>
>
> After incorporating v11 of the 2856 fix, we discovered that we are still having some
ACID violations.
> This time, however, the problem is not about including "newer" updates; but, about missing
older updates
> that should be including. 
> Here is what seems to be happening.
> There is a race condition in the StoreScanner.getScanners()
>   private List<KeyValueScanner> getScanners(Scan scan,
>       final NavigableSet<byte[]> columns) throws IOException {
>     // First the store file scanners
>     List<StoreFileScanner> sfScanners = StoreFileScanner
>       .getScannersForStoreFiles(store.getStorefiles(), cacheBlocks,
>                                 isGet, false);
>     List<KeyValueScanner> scanners =
>       new ArrayList<KeyValueScanner>(sfScanners.size()+1);
>     // include only those scan files which pass all filters
>     for (StoreFileScanner sfs : sfScanners) {
>       if (sfs.shouldSeek(scan, columns)) {
>         scanners.add(sfs);
>       }
>     }
>     // Then the memstore scanners
>     if (this.store.memstore.shouldSeek(scan)) {
>       scanners.addAll(this.store.memstore.getScanners());
>     }
>     return scanners;
>   }
> If for example there is a call to Store.updateStorefiles() that happens between
> the store.getStorefiles() and this.store.memstore.getScanners(); then
> it is possible that there was a new HFile created, that is not seen by the
> StoreScanner, and the data is not present in the Memstore.snapshot either.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message