incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravikumar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (BLUR-290) NRT Updates using RAMDirectory & Swap
Date Tue, 05 Nov 2013 12:20:20 GMT

     [ https://issues.apache.org/jira/browse/BLUR-290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravikumar updated BLUR-290:
---------------------------

    Attachment: BlurIndexTracker.java
                BlurRealTimeIndex.java
                RealTimeTransactionRecorder.java
                BlurRealTimeManagerReopenThread.java
                SlabRAMOutputStream.java
                SlabRAMInputStream.java
                SlabAllocator.java
                SlabRAMFile.java
                SlabRAMDirectory.java
                SortingMultiReader.java
                BlurFlushingIndexWriter.java
                BlurRealTimeManager.java
                BlurRealTimeIndexWriter.java

The basic points, I have dwelled upon is

1. RealTime index holds a current IW. It keeps accumulating docs there, until the "Commit
Thread" interrupts and swaps it.
2. Once a swap happens, a new transaction-log is opened
3. Previous data gets sort-flushed to disk and committed. At this point, the old-transaction
log is also deleted.
3. IndexSearcher is actually a MultiReader holding one disk-reader and many RAM-readers.
4. NRT re-opens are co-ordinated during "Step 3" to avoid duplicate records being searched
etc...

Sources will not compile. I am attaching only few, to get initial reactions

[~amccurry]  I have just tried to handle the write-path alone. Search/Read path can be iterated
later if this patch seems worthwhile.

> NRT Updates using RAMDirectory & Swap
> -------------------------------------
>
>                 Key: BLUR-290
>                 URL: https://issues.apache.org/jira/browse/BLUR-290
>             Project: Apache Blur
>          Issue Type: New Feature
>    Affects Versions: experimental-dev
>            Reporter: Ravikumar
>         Attachments: BlurFlushingIndexWriter.java, BlurIndexTracker.java, BlurRealTimeIndex.java,
BlurRealTimeIndexWriter.java, BlurRealTimeManager.java, BlurRealTimeManagerReopenThread.java,
RealTimeTransactionRecorder.java, SlabAllocator.java, SlabRAMDirectory.java, SlabRAMFile.java,
SlabRAMInputStream.java, SlabRAMOutputStream.java, SortingMultiReader.java
>
>
> We have been discussing about handling humungous rows in Blur (BLUR-220). Explore the
idea of using RAMDirectory at the front, backed by persistent-index.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message