incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron McCurry (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BLUR-290) NRT Updates using RAMDirectory & Swap
Date Wed, 13 Nov 2013 00:17:20 GMT

    [ https://issues.apache.org/jira/browse/BLUR-290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820714#comment-13820714
] 

Aaron McCurry commented on BLUR-290:
------------------------------------

This is the very thing I have struggled to find an answer to over the last 3+ years of developing
the system.  :-(

I'm really not sure how to solve this problem without doing more work on indexing and keeping
the read/search side the same.

Let me throw out a slightly different idea.  If we make the assumption that indexing and searching
performance are the 2 things we want to optimize the system to achieve.  And storage is less
of a concern.

What if we index each Row into a mini-index, an index of just that Row.  Then we merge that
mini-index into the main (deleting any existing Row) and store that mini-index as a single
binary field on the Row.  That way instead of re-indexing the documents a second time when
we modify, we read in the mini-index.  Make the update, and rewrite the Row+mini index again.

What do you think?

> NRT Updates using RAMDirectory & Swap
> -------------------------------------
>
>                 Key: BLUR-290
>                 URL: https://issues.apache.org/jira/browse/BLUR-290
>             Project: Apache Blur
>          Issue Type: New Feature
>    Affects Versions: experimental-dev
>            Reporter: Ravikumar
>         Attachments: BlurFlushingIndexWriter.java, BlurIndexTracker.java, BlurRealTimeIndex.java,
BlurRealTimeIndexWriter.java, BlurRealTimeManager.java, BlurRealTimeManagerReopenThread.java,
RealTimeTransactionRecorder.java, SlabAllocator.java, SlabRAMDirectory.java, SlabRAMFile.java,
SlabRAMInputStream.java, SlabRAMOutputStream.java, SortingMultiReader.java
>
>
> We have been discussing about handling humungous rows in Blur (BLUR-220). Explore the
idea of using RAMDirectory at the front, backed by persistent-index.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message