Return-Path: Delivered-To: apmail-hadoop-hbase-dev-archive@locus.apache.org Received: (qmail 12285 invoked from network); 22 Apr 2008 23:08:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 22 Apr 2008 23:08:44 -0000 Received: (qmail 51350 invoked by uid 500); 22 Apr 2008 23:08:45 -0000 Delivered-To: apmail-hadoop-hbase-dev-archive@hadoop.apache.org Received: (qmail 51313 invoked by uid 500); 22 Apr 2008 23:08:45 -0000 Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-dev@hadoop.apache.org Received: (qmail 51271 invoked by uid 99); 22 Apr 2008 23:08:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Apr 2008 16:08:45 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Apr 2008 23:08:09 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 85CB0234C0FC for ; Tue, 22 Apr 2008 16:05:21 -0700 (PDT) Message-ID: <1094389764.1208905521546.JavaMail.jira@brutus> Date: Tue, 22 Apr 2008 16:05:21 -0700 (PDT) From: "stack (JIRA)" To: hbase-dev@hadoop.apache.org Subject: [jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after HBASE-532 In-Reply-To: <1263058875.1208454081572.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-588: ------------------------ Attachment: 588.patch First attempt at fix for this issue. Doesn't yet pass all tests so needs more work. > Still a 'hole' in scanners, even after HBASE-532 > ------------------------------------------------ > > Key: HBASE-588 > URL: https://issues.apache.org/jira/browse/HBASE-588 > Project: Hadoop HBase > Issue Type: Bug > Reporter: stack > Priority: Blocker > Fix For: 0.1.2 > > Attachments: 588-v2.patch, 588.patch > > > Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them. > HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file. > Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure. > Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.