incubator-blur-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravikumar Govindarajan <ravikumar.govindara...@gmail.com>
Subject Re: Transaction Log in HDFS
Date Tue, 20 Dec 2016 10:59:54 GMT
Thanks a lot Aaron.

One quick clarification... Is it possible for you to provide an off-heap
option for the _pointers Map, since we already have Slab allocation and
related functionality in place?

I guess the challenge is to use Unsafe buffers optimally, for varying
lengths of values.

--
Ravi

On Thu, Dec 15, 2016 at 7:17 PM, Aaron McCurry <amccurry@gmail.com> wrote:

> Take a look at:
>
> https://github.com/apache/incubator-blur/blob/master/blur-kvs
>
> https://github.com/apache/incubator-blur/blob/master/
> blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java
>
> https://github.com/apache/incubator-blur/blob/master/
> blur-kvs/src/test/java/org/apache/blur/kvs/HdfsKeyValueStoreTest.java
>
> The HdfsKeyValueStore took the place of the WAL in Blur.  It's an inmemory
> cache that needs to be purged (and written to disk) from time to time.
>
> Let me know if you have questions.
>
> Aaron
>
> On Thu, Dec 15, 2016 at 7:22 AM, Ravikumar Govindarajan <
> ravikumar.govindarajan@gmail.com> wrote:
>
> > I remember seeing Blur writing transaction logs (WAL) in HDFS but is
> > removed as of today...
> >
> > Was re-implementing WAL-log for a particular use-case in Blur. Would be
> > great if community experienced any issues during WAL-writes in HDFS &
> could
> > share them here...
> >
> > I was looking at corner cases where WAL files could go corrupt. I guess
> > there should be many scenarios but just off the top of my head
> >
> >    - Program crashing on OOM/Seg-fault, with open WAL streams to HDFS
> >    - Name-Node, Data-Node differing on length of written block-data
> >    - Data-Node(s) failing to ack sync calls correctly
> >
> > Any insights is much appreciated
> >
> > --
> > Ravi
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message