Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Message-ID: <22222437.108271275412299265.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 13:11:39 -0400 (EDT)
From: "stack (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Created: (HBASE-2644) Investigate sync 'voodoo' splitting
 WALs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Investigate sync 'voodoo' splitting WALs
----------------------------------------

                 Key: HBASE-2644
                 URL: https://issues.apache.org/jira/browse/HBASE-2644
             Project: HBase
          Issue Type: Bug
            Reporter: stack
            Priority: Blocker
             Fix For: 0.21.0


The sequence file sync seems to actually help for some weird reason when recovering parts of edits (odd -- it looks like it just adds a marker to the file).  Investigate.  This comes of review of hbase-2437.   Below is copied from http://review.hbase.org/r/74/

{code}
no point to call .sync() here, it just wastes a bunch of IO to write "sync markers" which we don't make any real use of.
Cosmin Lehene 6 days, 23 hours ago (May 25th, 2010, 9:07 a.m.)
sync() used to call syncFs(). It looks like HBASE-2544 changed things a bit, but it doesn't only add the SequenceFile sync marker.

I added this after I've seen inconsistent results when running splitLog on bigger hlogs. Try copying a log from the cluster locally and run splitLog from the command line a few times without flushing it after each append. I used to get inconsistent results between runs and calling sync fixed it.

There's this "//TODO: test the split of a large (lots of regions > 500 file). In my tests it seems without hflush"  in the TestHLogSplit. 

We could do some testing to figure out why would log entries be lost when running locally.

What would be a better way to flush the writer?
Todd Lipcon 5 days, 19 hours ago (May 26th, 2010, 1:31 p.m.)
This seems really voodoo.. if anything we're probably masking a real bug by doing this. Can you write a unit test which shows this problem (even if it takes 30 minutes to run, would be good to have in our arsenal)
Cosmin Lehene 2 days, 18 hours ago (May 29th, 2010, 2:13 p.m.)
I can't reproduce it on hdfs-0.20. I can't compile hdfs-0.21 (again) for some reason. I'll give it another try some other time. 

Added the test. Also tried with a real 60MB log file. 
I'm not sure if we should leave the test active.
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.