incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikrant Navalgund (JIRA)" <>
Subject [jira] [Commented] (BLUR-234) Create a softlink like capability in the HDFSDirectory
Date Fri, 25 Oct 2013 13:20:31 GMT


Vikrant Navalgund commented on BLUR-234:

Hi Aaron,
Can you please help me in understanding the context. I have gone through the IndexImporter
timer task. 
These are some of my questions.
    "To provide for a mechanism to not copy around the index files in HDFS"
    Right now the index from /table/shard-0000x/xxxx.commit are being added by the indexwriter
in the destination.
    After the indexing the /table/shard-0000x/xxxx.commit is being deleted. 
    I also understand that we need some kind of a transaction mechanism to make sure the failed
operations are re-tried.
    But how does a HDFSSoftLinkDirectory help us? 
    What is a HDFSSoftlinkDirectory semantics?
        Does this provide the mechanism for the transaction like operation, by deleting the
links once the operation is done?
        Does the linkpath have flat files like filename0001.tmp that gets deleted once the
indexing is over?   
        Do we have a linkpath for each corresponding ../xxxx.commit path ?
   The main ticket and the sub-task(this ticket) are kind of confusing me. What am I missing


> Create a softlink like capability in the HDFSDirectory
> ------------------------------------------------------
>                 Key: BLUR-234
>                 URL:
>             Project: Apache Blur
>          Issue Type: Sub-task
>          Components: Blur
>    Affects Versions: 0.3.0
>            Reporter: Aaron McCurry
>             Fix For: 0.3.0
> The problem we are trying to solve here is minimizing file copying.  During a merge of
an external index produced by MR into a shard index normally the index files are copied. 
In a lot of cases the new external index(es) are very large.  This can cause some serious
performance problems because all the new data would be copied into shard index.  Normally
this can happens across the cluster at the same time so it will likely turn into an IO storm.
> The current implementation in the IndexImporter that deals with this problem does so
by overriding method in the HDFSDirectory that moves the files in HDFS instead of copying.
 This makes those merges very fast, but it's risky because if the shard index writer doesn't
commit the changes the files are not moved back to their original location.  Instead they
are deleted, loss of data.
> So I'm preposing that we create a softlink system that allows for links to the be created
instead of being moved.  That way if the commit fails the links are removed and the original
data files are in the their original location.

This message was sent by Atlassian JIRA

View raw message