incubator-blur-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: Transaction Log in HDFS
Date Sat, 24 Dec 2016 15:55:03 GMT
On Wed, Dec 21, 2016 at 7:26 AM, Ravikumar Govindarajan <
ravikumar.govindarajan@gmail.com> wrote:

> On Tue, Dec 20, 2016 at 10:33 PM, Aaron McCurry <amccurry@gmail.com>
> wrote:
>
> > Perhaps if you really need a WAL that you could look at the sync/fencing
> > code in the KV store to implement a WAL.
> >
>
> Thanks for this. I can straightaway base my WAL impl on this..
>
> Btw, I see a special reflective method HdfsUtil.getLength() & KV
> InputStream does not read beyond this length. Any reasons for that?
>

The reason for it is to get the current posted length from the namenode.
Newer hdfs implements may not need this but with the versions that Blur was
tested against it was required to get an updated length.  Basically the
namenode will post the length of the file when the block barrier is crossed
or when the file is closed.  However the namenode metadata will not be
reflected upon a fsync and to ensure the correct operation the most up to
date length is needed.  Hence the reflection call.  Let me know if you have
questions.

Aaron


>
> --
> Ravi
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message