lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amin Mohammed-Coleman <ami...@gmail.com>
Subject Re: Using lucene in a clustered app server
Date Mon, 15 Jun 2009 08:04:19 GMT
Hi

I'm looking at Hadoop and Katta and I was wondering if some may be able
clarify the following:

1) Is Katta replacing the Hadoop Lucene contribution
2) Are people still using Hadoop Lucene to perform indexing


Cheers
Amin

On Sat, Jun 13, 2009 at 7:46 AM, Amin Mohammed-Coleman <aminmc@gmail.com>wrote:

> Hi
> Thanks for getting back to me.  I have briefly looked at Hadoop.  It looks
> interesting but I haven't thought about using it for indexing.  But I think
> it might be on the plan.  I am doing the initial investigation and
> technology frameworks.  I would be very interested to get an idea of how you
> are designing your cluster indexing.  I have been doing a documentation
> which I would be happy to show you.
>
>
> Cheers
> Amin
>
>
> On Thu, Jun 11, 2009 at 5:29 PM, Tarandeep Singh <tarandeep@gmail.com>wrote:
>
>> Hi Amin,
>>
>> I am also working on similar implementation. But right now I am working on
>> building Lucene Index in parallel using Hadoop.
>>
>> How are you building your Lucene index (or indexes) ? Have you tried
>> Hadoop?
>> If yes, we can share some design issues.
>>
>> Also, for serving Lucene indexes over cluster, you might be interested in
>> Katta project-
>> katta.sourceforge.net
>>
>> -Tarandeep
>>
>> On Thu, Jun 11, 2009 at 1:04 AM, Amin Mohammed-Coleman <aminmc@gmail.com
>> >wrote:
>>
>> > Hi
>> >
>> > we are thinking of implementing a solution like this.  All nodes on the
>> > cluster has a local index from which searches are performed.  Once an
>> > update
>> > on the index needs to happen the work is sent to a queue where there is
>> a
>> > master who is responsible for updating the central index.  The master is
>> > then responsible for propagating/notifying other nodes in the cluster
>> that
>> > their local index is not in sync.  The local index worker can perform an
>> > update against the master index.
>> >
>> > We haven't implemented this yet but its the approach we'll take inorder
>> to
>> > deal with clustering.  If anyone has any ideas which could improve this
>> or
>> > a
>> > better approach your thoughts will be appreciated.
>> >
>> > Cheers
>> > Amin
>> >
>> > On Wed, Jun 10, 2009 at 10:15 AM, Ian Lea <ian.lea@gmail.com> wrote:
>> >
>> > > I'd recommend using your favourite queueing service to pass all
>> > > updates to a central process, the one and only process that updates
>> > > the index.  If you don't already have a favourite queueing service,
>> > >
>> >
>> http://en.wikipedia.org/wiki/Java_Message_Service#Provider_implementations
>> > > lists several JMS implementations.
>> > >
>> > >
>> > > --
>> > > Ian.
>> > >
>> > > On Tue, Jun 9, 2009 at 9:02 PM, Newman, Billy<Billy.Newman@itt.com>
>> > wrote:
>> > > > I am trying to figure out the best way to add to a lucene index
>> across
>> > a
>> > > clustered app server.  I cannot grab an IndexWriter for each node in
>> the
>> > > cluster, because I would run into lock file problems.  I am not sure
>> if I
>> > > can share one IndexWriter across the cluster because what happens when
>> > two
>> > > or more nodes update the same thing.  Anyone have any insight?
>> > > >
>> > > > Thanks.
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message