hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after HBASE-532
Date Wed, 23 Apr 2008 04:15:24 GMT

     [ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-588:
------------------------

    Attachment: 588-v3.patch

{code}
Add a ChangedReadersObserver interface.  HStore notifies registered
observers when list of HStoreFile Readers changes -- at flush time
and at compaction time.  Scanners are only current observers.

M  src/test/org/apache/hadoop/hbase/MultiRegionTable.java
    (getCacheFlushListern): Renamed as getFlushRequester
M  src/java/org/apache/hadoop/hbase/regionserver/HStore.java
    Added synchronizations on this.storefiles.
    (activeScanners, newScannerLock): Removed.  We no longer try to
    block out scanners when compacting (Turns out same functionality
    for blocking scanners is up in HRegion used around closing/split).
    (changedReadersObservers): Added.
    (updateReaders): New method that manages the insertion of new
    reader on flush.  Also calls new notifyChangedReadersObservers.
    (notifyChagnedReadersObservers, addChangedReadersObserver,
      deleteChangedReadersObservers): Added.
    (completeCompaction): Previous deleting old store files, we'd
    remove from this.storefiles and delete all in one step; now we
    do the remove first, notify all observers of readers, and then
    do the delete so observers have a chance to clean up any old
    references to files about to be deleted.  Removed all the lockout
    of new scanner creation and wait on old scanners to come int.
    (updateActiveScanners): Removed.
    (getStorefiles): Accessor.  Added.
M  src/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
    Added implementation of new ChangedReadersObserver interface.
    Added a lock that we hold when 'nexting' and when changing the
    set of readers out from under the scanner.
    Changed the constructor moving bulk into new openReaders method
    that we reuse when list of Readers changes.
    (next): Moved head of this method into new getNextViableRow (used
    to be called chosenRow and chosenTimestamp).  New method returns
    simple datastructure of row and timestamp (ViableRow).
    (close): On close, remove ourselves as ChangedReadersObserver (we
    added ourselves in the constructor).
    (updateReaders): Changed the set of Readers out from under the
    Scanner.
A  src/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
    Added.
M  src/java/org/apache/hadoop/hbase/regionserver/Flusher.java
    Changed name of the interface we implement from CacheFlushListener to
    FlushRequester.
D  src/java/org/apache/hadoop/hbase/regionserver/CacheFlushListener.java
    Renamed as FlushRequester.
M  src/java/org/apache/hadoop/hbase/regionserver/HStoreScanner.java
    Remove update of activeScanners.
A  src/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
    Added.  Rename of CacheFlushListener.
M  src/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
    Renamed method getCacheFlushListener as getFlushRequester.
M src/java/org/apache/hadoop/hbase/regionserver/HAbstractScanner.java
    Formatting.
M  src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
    CacheFlushListener was renamed as FlushListener.
{code}

> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
>                 Key: HBASE-588
>                 URL: https://issues.apache.org/jira/browse/HBASE-588
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Priority: Blocker
>             Fix For: 0.1.2
>
>         Attachments: 588-v2.patch, 588-v3.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot.  Snapshot used to copy
current live memcache into a 'snapshot' TreeMap inside in Memcache.  This snapshot TreeMap
was an accumulation of all snapshots since last flush.   Whenever we took out a scanner, we'd
do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding
Scanner had complete copy).  Memcache snapshots were cleared when a flush started.   Flushing
could take near no time to up to tens of seconds during which an scanners taken out meantime
would not see the edits in the snapshot currently being flushed and gets or getFull would
also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file
had made it out to disk.  This fixed gets and getFull and any scanners taken out during flushing.
 But there is still a hole.  Any outstanding scanners will be going against the state of Store
Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot
to each Scanner (Pass the snapshot if not empty).  The notion is that the Scanner would hold
on to the Scanner reference should it be cleared by flushing.  Upside is that scanner wouldn't
have to be concerned with the new flush that has been put out to disk.  Downsides are that
Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't
done it yet.  And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event.  Listeners such as outstanding scanners
would notice event and open the new Reader.  Would have to skip forward in the new Reader
to catch up with the current set but shouldn't be bad.  Same mechanism could be used to let
compactions be moved into place while scanners were outstanding closing down all existing
readers skipping to the current 'next' location in the new compacted store file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message