hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-2644) Investigate sync 'voodoo' splitting WALs
Date Tue, 01 Jun 2010 17:11:39 GMT
Investigate sync 'voodoo' splitting WALs
----------------------------------------

                 Key: HBASE-2644
                 URL: https://issues.apache.org/jira/browse/HBASE-2644
             Project: HBase
          Issue Type: Bug
            Reporter: stack
            Priority: Blocker
             Fix For: 0.21.0


The sequence file sync seems to actually help for some weird reason when recovering parts
of edits (odd -- it looks like it just adds a marker to the file).  Investigate.  This comes
of review of hbase-2437.   Below is copied from http://review.hbase.org/r/74/

{code}
no point to call .sync() here, it just wastes a bunch of IO to write "sync markers" which
we don't make any real use of.
Cosmin Lehene 6 days, 23 hours ago (May 25th, 2010, 9:07 a.m.)
sync() used to call syncFs(). It looks like HBASE-2544 changed things a bit, but it doesn't
only add the SequenceFile sync marker.

I added this after I've seen inconsistent results when running splitLog on bigger hlogs. Try
copying a log from the cluster locally and run splitLog from the command line a few times
without flushing it after each append. I used to get inconsistent results between runs and
calling sync fixed it.

There's this "//TODO: test the split of a large (lots of regions > 500 file). In my tests
it seems without hflush"  in the TestHLogSplit. 

We could do some testing to figure out why would log entries be lost when running locally.

What would be a better way to flush the writer?
Todd Lipcon 5 days, 19 hours ago (May 26th, 2010, 1:31 p.m.)
This seems really voodoo.. if anything we're probably masking a real bug by doing this. Can
you write a unit test which shows this problem (even if it takes 30 minutes to run, would
be good to have in our arsenal)
Cosmin Lehene 2 days, 18 hours ago (May 29th, 2010, 2:13 p.m.)
I can't reproduce it on hdfs-0.20. I can't compile hdfs-0.21 (again) for some reason. I'll
give it another try some other time. 

Added the test. Also tried with a real 60MB log file. 
I'm not sure if we should leave the test active.
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message