Return-Path: 
 <hbase-issues-return-73-apmail-hadoop-hbase-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-hbase-issues-archive@minotaur.apache.org
Received: (qmail 18873 invoked from network); 10 Mar 2010 05:06:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Mar 2010 05:06:21 -0000
Received: (qmail 58979 invoked by uid 500); 10 Mar 2010 05:05:52 -0000
Delivered-To: apmail-hadoop-hbase-issues-archive@hadoop.apache.org
Received: (qmail 58951 invoked by uid 500); 10 Mar 2010 05:05:52 -0000
Mailing-List: contact hbase-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hbase-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hbase-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hbase-issues@hadoop.apache.org>
List-Id: <hbase-issues.hadoop.apache.org>
Reply-To: hbase-issues@hadoop.apache.org
Delivered-To: mailing list hbase-issues@hadoop.apache.org
Received: (qmail 58937 invoked by uid 99); 10 Mar 2010 05:05:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Mar 2010 05:05:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Mar 2010 05:05:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 228A8234C48C
	for <hbase-issues@hadoop.apache.org>; Wed, 10 Mar 2010 05:05:27 +0000 (UTC)
Message-ID: <1677133127.170211268197527126.JavaMail.jira@brutus.apache.org>
Date: Wed, 10 Mar 2010 05:05:27 +0000 (UTC)
From: "Kannan Muthukkaruppan (JIRA)" <jira@apache.org>
To: hbase-issues@hadoop.apache.org
Subject: [jira] Commented: (HBASE-2283) row level atomicity
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HBASE-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843436#action_12843436 ] 

Kannan Muthukkaruppan commented on HBASE-2283:
----------------------------------------------

Thanks for your input.

Have the code changes to support things upward compatibly.  The serialized format of a KeyValue starts with a "int" length. Overloading the length now for versioning. If the length is the special value -1, then will interpret the rest of the data in new format. Else, interpret the data in old format. 

A HLog entry could now either be:

<HLogKey>:<KeyValue>
or,
<HLogKey>:<-1 <# of edits, <KeyValue1>, <KeyValue2>, ...>>

I think we have the overall fix pretty much code complete. Will put up the patch after some basic testing (hopefully by tomorrow). And then will continue more detailed testing in parallel.





> row level atomicity 
> --------------------
>
>                 Key: HBASE-2283
>                 URL: https://issues.apache.org/jira/browse/HBASE-2283
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: Kannan Muthukkaruppan
>            Priority: Blocker
>             Fix For: 0.20.4, 0.21.0
>
>
> The flow during a HRegionServer.put() seems to be the following. [For now, let's just consider single row Put containing edits to multiple column families/columns.]
> HRegionServer.put() does a:
>         HRegion.put();
>        syncWal()  (the HDFS sync call).  /* this is assuming we have HDFS-200 */
> HRegion.put() does a:
>   for each column family 
>   {
>       HLog.append(all edits to the colum family);
>       write all edits to Memstore;
>   }
> HLog.append() does a :
>   foreach edit in a single column family {
>     doWrite()
>   }
> doWrite() does a:
>    this.writer.append().
> There seems to be two related issues here that could result in inconsistencies.
> Issue #1: A put() does a bunch of HLog.append() calls. These in turn do a bunch of "write" calls on the underlying DFS stream.  If we crash after having written out some append's to DFS, recovery will run and apply a partial transaction to memstore.  
> Issue #2: The updates to memstore  should happen after the sync rather than before. Otherwise, there is the danger that the write to DFS (sync) fails for some reason & we return an error to the client, but we have already taken edits to the memstore. So subsequent reads will serve uncommitted data.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.