incubator-blur-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: Transaction Log in HDFS
Date Tue, 20 Dec 2016 17:03:03 GMT
On Tue, Dec 20, 2016 at 5:59 AM, Ravikumar Govindarajan <
ravikumar.govindarajan@gmail.com> wrote:

> Thanks a lot Aaron.
>
> One quick clarification... Is it possible for you to provide an off-heap
> option for the _pointers Map, since we already have Slab allocation and
> related functionality in place?
>

I'm sure it's possible however the main use case didn't use a lot of memory
so that sort of feature was never implemented.


>
> I guess the challenge is to use Unsafe buffers optimally, for varying
> lengths of values.
>

Yeah that would be a problem if variable blocks are stored.

Perhaps if you really need a WAL that you could look at the sync/fencing
code in the KV store to implement a WAL.  Also HBase uses (or did at one
point)  a standard sequence file for a WAL.


>
> --
> Ravi
>
> On Thu, Dec 15, 2016 at 7:17 PM, Aaron McCurry <amccurry@gmail.com> wrote:
>
> > Take a look at:
> >
> > https://github.com/apache/incubator-blur/blob/master/blur-kvs
> >
> > https://github.com/apache/incubator-blur/blob/master/
> > blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java
> >
> > https://github.com/apache/incubator-blur/blob/master/
> > blur-kvs/src/test/java/org/apache/blur/kvs/HdfsKeyValueStoreTest.java
> >
> > The HdfsKeyValueStore took the place of the WAL in Blur.  It's an
> inmemory
> > cache that needs to be purged (and written to disk) from time to time.
> >
> > Let me know if you have questions.
> >
> > Aaron
> >
> > On Thu, Dec 15, 2016 at 7:22 AM, Ravikumar Govindarajan <
> > ravikumar.govindarajan@gmail.com> wrote:
> >
> > > I remember seeing Blur writing transaction logs (WAL) in HDFS but is
> > > removed as of today...
> > >
> > > Was re-implementing WAL-log for a particular use-case in Blur. Would be
> > > great if community experienced any issues during WAL-writes in HDFS &
> > could
> > > share them here...
> > >
> > > I was looking at corner cases where WAL files could go corrupt. I guess
> > > there should be many scenarios but just off the top of my head
> > >
> > >    - Program crashing on OOM/Seg-fault, with open WAL streams to HDFS
> > >    - Name-Node, Data-Node differing on length of written block-data
> > >    - Data-Node(s) failing to ack sync calls correctly
> > >
> > > Any insights is much appreciated
> > >
> > > --
> > > Ravi
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message