lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amin Mohammed-Coleman <ami...@gmail.com>
Subject Re: Using lucene in a clustered app server
Date Sat, 13 Jun 2009 06:46:53 GMT
Hi
Thanks for getting back to me.  I have briefly looked at Hadoop.  It looks
interesting but I haven't thought about using it for indexing.  But I think
it might be on the plan.  I am doing the initial investigation and
technology frameworks.  I would be very interested to get an idea of how you
are designing your cluster indexing.  I have been doing a documentation
which I would be happy to show you.


Cheers
Amin

On Thu, Jun 11, 2009 at 5:29 PM, Tarandeep Singh <tarandeep@gmail.com>wrote:

> Hi Amin,
>
> I am also working on similar implementation. But right now I am working on
> building Lucene Index in parallel using Hadoop.
>
> How are you building your Lucene index (or indexes) ? Have you tried
> Hadoop?
> If yes, we can share some design issues.
>
> Also, for serving Lucene indexes over cluster, you might be interested in
> Katta project-
> katta.sourceforge.net
>
> -Tarandeep
>
> On Thu, Jun 11, 2009 at 1:04 AM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
> > Hi
> >
> > we are thinking of implementing a solution like this.  All nodes on the
> > cluster has a local index from which searches are performed.  Once an
> > update
> > on the index needs to happen the work is sent to a queue where there is a
> > master who is responsible for updating the central index.  The master is
> > then responsible for propagating/notifying other nodes in the cluster
> that
> > their local index is not in sync.  The local index worker can perform an
> > update against the master index.
> >
> > We haven't implemented this yet but its the approach we'll take inorder
> to
> > deal with clustering.  If anyone has any ideas which could improve this
> or
> > a
> > better approach your thoughts will be appreciated.
> >
> > Cheers
> > Amin
> >
> > On Wed, Jun 10, 2009 at 10:15 AM, Ian Lea <ian.lea@gmail.com> wrote:
> >
> > > I'd recommend using your favourite queueing service to pass all
> > > updates to a central process, the one and only process that updates
> > > the index.  If you don't already have a favourite queueing service,
> > >
> >
> http://en.wikipedia.org/wiki/Java_Message_Service#Provider_implementations
> > > lists several JMS implementations.
> > >
> > >
> > > --
> > > Ian.
> > >
> > > On Tue, Jun 9, 2009 at 9:02 PM, Newman, Billy<Billy.Newman@itt.com>
> > wrote:
> > > > I am trying to figure out the best way to add to a lucene index
> across
> > a
> > > clustered app server.  I cannot grab an IndexWriter for each node in
> the
> > > cluster, because I would run into lock file problems.  I am not sure if
> I
> > > can share one IndexWriter across the cluster because what happens when
> > two
> > > or more nodes update the same thing.  Anyone have any insight?
> > > >
> > > > Thanks.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message