incubator-blur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron McCurry (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BLUR-290) NRT Updates using RAMDirectory & Swap
Date Mon, 25 Nov 2013 01:16:35 GMT

    [ https://issues.apache.org/jira/browse/BLUR-290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13831127#comment-13831127
] 

Aaron McCurry commented on BLUR-290:
------------------------------------

- I was just trying a proof-of-concept stuff. There are many nocommits and loose ends, so
haven't really gone to the perf aspect. But if you can eval, will be great. I have fixed some
bugs and attached few test-cases, in case you want to try it.

I will try to give it a go this week.

- +1 for introducing table-types. A less adventurous table-type can even write unsorted segments
to disk and take a hit during searches, while doing the balancing act during merge.

I think that it will be a property that we read from the table properties on open.  That way
if someone else comes up with their own they don't have to modify anymore code than necessary.

- It would be great if we can detect that NRT re-opens are going to suffer under load and
hence adjust accordingly. Shouldn't it be a policy that is applicable to all table-types instead
of it being a table-type itself?

Hmm, not sure.  I have started creating a simple blur index that will not show any uncommitted
data.  Thus the visibility of the data is going to be very slow.  I have also starting working
on an alternate version that allows for the inbound data to be written to a hdfs backed queue
for indexing.  That way the indexer can take larger and larger chunks of data if it gets behind
the inbound data.  So perhaps we could combine all of these?  Maybe not.  Not sure at this
point.  I did notice that I will need to modify the BlurIndex api to allow for partial Row
updates (Record appending or something).

- PS: Sorry for the huge number of files. Is there place to check-in, may be an experimental-branch?

At this point there's no where I can give you access.  We should probably have infrastructure
mirror apache git to github.  That way it would be easier for you to branch and post pull
requests etc.

Aaron

> NRT Updates using RAMDirectory & Swap
> -------------------------------------
>
>                 Key: BLUR-290
>                 URL: https://issues.apache.org/jira/browse/BLUR-290
>             Project: Apache Blur
>          Issue Type: New Feature
>    Affects Versions: experimental-dev
>            Reporter: Ravikumar
>         Attachments: BlurFieldsConsumer.java, BlurFieldsConsumer.java, BlurFieldsConsumer.java,
BlurFlushingIndexWriter.java, BlurIndexTracker.java, BlurPostingsConsumer.java, BlurPostingsConsumer.java,
BlurPostingsFormat.java, BlurPostingsFormat.java, BlurRealTimeIndex.java, BlurRealTimeIndex.java,
BlurRealTimeIndexTest.java, BlurRealTimeIndexWriter.java, BlurRealTimeManager.java, BlurRealTimeManagerReopenThread.java,
BlurRowCodec.java, BlurRowCodec.java, BlurSegmentInfoFormat.java, BlurSegmentInfoWriter.java,
BlurTermsConsumer.java, BlurTermsConsumer.java, CompressingRowIndexReader.java, CompressingRowIndexWriter.java,
CompressingRowReader.java, CompressingRowReader.java, CompressingRowReader.java, CompressingRowWriter.java,
CompressingRowWriter.java, CompressingRowWriter.java, GrowableByteArrayDataOutput.java, PrimeDocCache.java,
RealTimeTransactionRecorder.java, RealTimeTransactionRecorder.java, RowCache.java, RowDocsCollector.java,
RowDocsCollector.java, RowReaderCache.java, RowReaderCache.java, SlabAllocator.java, SlabRAMDirectory.java,
SlabRAMFile.java, SlabRAMInputStream.java, SlabRAMOutputStream.java, SortingMultiReader.java,
SortingMultiReader.java, TestCompressingRowWriter.java, TestCompressingRowWriter.java
>
>
> We have been discussing about handling humungous rows in Blur (BLUR-220). Explore the
idea of using RAMDirectory at the front, backed by persistent-index.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message