hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: hflush not working for me?
Date Fri, 09 Oct 2009 21:32:02 GMT
On Fri, Oct 9, 2009 at 1:27 PM, Tsz Wo (Nicholas), Sze <
s29752-hadoopuser@yahoo.com> wrote:

> Hi St.Ack,
> > ... soft lease to 1 second ...
> You are right that you don't have to change soft lease.  It is for append
> but not related to hflash.

I should not have to set it then?  I can remove this 70 second pause in
middle of my test?

> > Do I have to do open as another user?
> This should not be necessary.
> Could you send me/post your test?
Sure, as long as you don't hold this ugly code against me ever after.

I checked in the code so you could try it:

Its the first test, testSync.

It starts out by copying whats down in the hdfs testReadWhileWriting.  That
bit works fine.

Then comes the ugly stuff.

HLog is our write-ahead log wrapper.  Internally it writes out to a
SequenceFile.Writer.  The SequenceFile.Writer has been doctored using
reflection so the out datamember is non-private.  A call to HLog.sync runs
the SequenceFile.Writer.sync -- which DOES NOT call sync on the backing
output stream -- and then it calls sync on the now accessible out stream
(Sorry its so ugly -- I'm trying to hack stuff up fast so all of hbase gets
access to this new facility).  If I trace in the debugger, I can see that
the sync on the out data member goes down into hflush.  Queued up edits are
flushed.  It seems like it should be working.

Do I have to do some doctoring of the reader? (It doesn't seem so given that
the code at the head of this test works).

Thanks for taking a look Nicholas.

To run the test, you can do "ant clean jar test -Dtestcase=TestHLog".

(Let me know if you want an eclipse .project + .classpath so you can get it
up in an ide to run debugger).


> Nicholas Sze
> *From:* stack <stack@duboce.net>
> *To:* hdfs-user@hadoop.apache.org
> *Sent:* Fri, October 9, 2009 1:13:37 PM
> *Subject:* hflush not working for me?
> I'm putting together some unit tests up in our application that exercise
> hflush.  I'm using minidfscluster and a jar made by building head of the
> 0.21 branch of hdfs (from about a minute ago).
> Code opens a file, writes a bunch of edits, invokes hflush (by calling sync
> on DFSDataOutputStream instance) and then, without closing the Writer, opens
> a Reader on same file.  This Reader does not see any edits not to mind edits
> up to the sync invocation.
> I can trace the code and see how on hflush it sends the queued packets of
> edits.
> I studied TestReadWhileWriting.  I've set setBoolean("dfs.support.append",
> true) before minidfscluster spins up.  I can't set soft lease to 1 second
> because not in same package so I just wait out the default minute.  It
> doesn't seem to make a difference.
> Do I have to do open as another user?
> Thanks for any pointers,
> St.Ack

View raw message