incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron McCurry (JIRA)" <>
Subject [jira] [Commented] (BLUR-234) Create a softlink like capability in the HDFSDirectory
Date Wed, 16 Oct 2013 10:12:45 GMT


Aaron McCurry commented on BLUR-234:

Great!  Thanks!

There is a partially built one in the blur-store already.  Needs more work though.


> Create a softlink like capability in the HDFSDirectory
> ------------------------------------------------------
>                 Key: BLUR-234
>                 URL:
>             Project: Apache Blur
>          Issue Type: Sub-task
>          Components: Blur
>    Affects Versions: 0.3.0
>            Reporter: Aaron McCurry
>             Fix For: 0.3.0
> The problem we are trying to solve here is minimizing file copying.  During a merge of
an external index produced by MR into a shard index normally the index files are copied. 
In a lot of cases the new external index(es) are very large.  This can cause some serious
performance problems because all the new data would be copied into shard index.  Normally
this can happens across the cluster at the same time so it will likely turn into an IO storm.
> The current implementation in the IndexImporter that deals with this problem does so
by overriding method in the HDFSDirectory that moves the files in HDFS instead of copying.
 This makes those merges very fast, but it's risky because if the shard index writer doesn't
commit the changes the files are not moved back to their original location.  Instead they
are deleted, loss of data.
> So I'm preposing that we create a softlink system that allows for links to the be created
instead of being moved.  That way if the commit fails the links are removed and the original
data files are in the their original location.

This message was sent by Atlassian JIRA

View raw message