incubator-blur-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: Transaction Log in HDFS
Date Thu, 15 Dec 2016 13:47:03 GMT
Take a look at:

https://github.com/apache/incubator-blur/blob/master/blur-kvs

https://github.com/apache/incubator-blur/blob/master/blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java

https://github.com/apache/incubator-blur/blob/master/blur-kvs/src/test/java/org/apache/blur/kvs/HdfsKeyValueStoreTest.java

The HdfsKeyValueStore took the place of the WAL in Blur.  It's an inmemory
cache that needs to be purged (and written to disk) from time to time.

Let me know if you have questions.

Aaron

On Thu, Dec 15, 2016 at 7:22 AM, Ravikumar Govindarajan <
ravikumar.govindarajan@gmail.com> wrote:

> I remember seeing Blur writing transaction logs (WAL) in HDFS but is
> removed as of today...
>
> Was re-implementing WAL-log for a particular use-case in Blur. Would be
> great if community experienced any issues during WAL-writes in HDFS & could
> share them here...
>
> I was looking at corner cases where WAL files could go corrupt. I guess
> there should be many scenarios but just off the top of my head
>
>    - Program crashing on OOM/Seg-fault, with open WAL streams to HDFS
>    - Name-Node, Data-Node differing on length of written block-data
>    - Data-Node(s) failing to ack sync calls correctly
>
> Any insights is much appreciated
>
> --
> Ravi
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message