lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanne Grinovero <s.grinov...@sourcesense.com>
Subject Re: A new Lucene Directory available
Date Sun, 15 Nov 2009 12:14:59 GMT
Hi Lukas,
Our reference during early design was Lucene 2.4.1, but we look
forward for compatibility and new tricks.
Current trunk is compatible towards Lucene's trunk, but I won't close
ISPN-275 until it's confirmed against a released Lucene 3.0.0 :
hopefully this will come before Infinispan 4 release.

Regards,
Sanne

On Sun, Nov 15, 2009 at 8:50 AM, Lukáš Vlček <lukas.vlcek@gmail.com> wrote:
> Hi,
>
> this sounds very interesting. Do you know which versions of Lucene are
> supported?
> Do you know if it would work with upcoming Lucene 3.0.x?
> https://jira.jboss.org/jira/browse/ISPN-275
>
> Regards,
> Lukas
>
> http://blog.lukas-vlcek.com/
>
>
> On Sun, Nov 15, 2009 at 5:33 AM, Sanne Grinovero
> <s.grinovero@sourcesense.com> wrote:
>>
>> Hi John,
>> I didn't run a long running reliable benchmark, so at the moment I
>> can't really speak of numbers.
>> Suggestions and help on performance testing are welcome: I guess it
>> will shine in some situations, not necessarily all, so really choosing
>> a correct ratio of concurrent writers/searches, number of nodes in the
>> cluster and resources per node will never be fair enough to compare
>> this Directory with others.
>>
>> On paper the premises are good: it's all in-memory, until it fits: it
>> will distribute data across nodes and overflow to disk is supported
>> (called passivation). A permanent store can be configured, so you
>> could set it to periodically flush incrementally to slower storages
>> like a database, a filesystem, a cloud storage service. This makes it
>> possible to avoid losing state even when all nodes are shut down.
>> A RAMDirectory is AFAIK not recommended as you could hit memory limits
>> and because it's basically a synchronized HashMap; Infinispan
>> implements ConcurrentHashMap and doesn't need synchronization.
>> Even if the data is replicated across nodes each node has it's own
>> local cache, so when caches are warm and all segments fit in memory it
>> should be, theoretically, the fastest Directory ever. The more it will
>> read from disk, the more it will behave similarly to a FSDirectory
>> with some buffers.
>>
>> As per Lucene's design, writes can happen only at one node at a time:
>> one IndexWriter can own the lock, but IndexReaders and Searchers are
>> not blocked, so when using this Directory it should behave exactly as
>> if you had multiple processes sharing a local NIOFSdirectory:
>> basically the situation is that you can't scale on writers, but you
>> can scale near-linearly with readers adding in more power from more
>> machines.
>>
>> Besides performance, the reasons to implement this was to be able to
>> easily add or remove processing power to a service (clouds), make it
>> easier to share indexes across nodes, and last but not least to remove
>> single points of failure: all data is distributed and there is no such
>> notion of Master: services will continue running fine when killing any
>> node.
>>
>> I hope this peeks your interest, sorry if I couldn't provide numbers.
>>
>> Regards,
>> Sanne
>>
>> On Sat, Nov 14, 2009 at 11:15 PM, John Wang <john.wang@gmail.com> wrote:
>> > HI Sanne:
>> >
>> >     Very interesting!
>> >
>> >     What kinda performance should we expect with this, comparing to
>> > regular
>> > FSDIrectory on local HD.
>> > Thanks
>> > -John
>> >
>> > On Sat, Nov 14, 2009 at 11:44 AM, Sanne Grinovero
>> > <s.grinovero@sourcesense.com> wrote:
>> >>
>> >> Hello all,
>> >> I'm a Lucene user and fan, I wanted to tell you that we just released
>> >> a first technology preview of a distributed in memory Directory for
>> >> Lucene.
>> >>
>> >> The release announcement:
>> >>
>> >>
>> >> http://infinispan.blogspot.com/2009/11/second-release-candidate-for-400.html
>> >>
>> >> From there you'll find links to the Wiki, to the sources, to the issue
>> >> tracker. A minimal demo is included with the sources.
>> >>
>> >> This was developed together with Google Summer of Code student Lukasz
>> >> Moren and much support from the Infinispan and Hibernate Search teams,
>> >> as we are storing the index segments on Infinispan and using it's
>> >> atomic distributed locks to implement a Lucene LockFactory.
>> >>
>> >> Initial idea was to contribute it directly to Lucene, but as
>> >> Infinispan is a LGPL dependency we had to distribute it with
>> >> Infinispan (as the other way around would have introduced some legal
>> >> issues); still we hope you appreciate the effort and are interested in
>> >> giving it a try.
>> >> All kind of feedback is welcome, especially on benchmarking
>> >> methodologies as I yet have to do some serious performance tests.
>> >>
>> >> Main code, build with Maven2:
>> >> svn co
>> >>
>> >> http://anonsvn.jboss.org/repos/infinispan/tags/4.0.0.CR2/lucene-directory/
>> >> infinispan-directory
>> >>
>> >> Demo, see the Readme:
>> >> svn co
>> >>
>> >> http://anonsvn.jboss.org/repos/infinispan/tags/4.0.0.CR2/demos/lucene-directory/
>> >> lucene-demo
>> >>
>> >> Best Regards,
>> >> Sanne
>> >>
>> >> --
>> >> Sanne Grinovero
>> >> Sourcesense - making sense of Open  Source: http://www.sourcesense.com
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> >>
>> >
>> >
>>
>>
>>
>> --
>> Sanne Grinovero
>> Sourcesense - making sense of Open  Source: http://www.sourcesense.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>



-- 
Sanne Grinovero
Sourcesense - making sense of Open  Source: http://www.sourcesense.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message