lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tarandeep Singh <tarand...@gmail.com>
Subject Re: Using lucene in a clustered app server
Date Mon, 15 Jun 2009 09:32:47 GMT
On Mon, Jun 15, 2009 at 1:04 AM, Amin Mohammed-Coleman <aminmc@gmail.com>wrote:

> Hi
>
> I'm looking at Hadoop and Katta and I was wondering if some may be able
> clarify the following:
>
> 1) Is Katta replacing the Hadoop Lucene contribution


You mean the index package in Hadoop's contrib folder?
So far what I have understood is Katta is geared more towards Index shards
management and serving. Katta is not bothered how you created your index
shards. Although it provides tools to create index using Hadoop.


> 2) Are people still using Hadoop Lucene to perform indexing


Not sure of that but it also depends on your needs. Best would be to index a
subset of your documents and see if it meets yoru requirements, else either
modify the code or write your own code. I am creating my own map reduce jobs
to index documents.

-Tarandeep


>
>
>
> Cheers
> Amin
>
> On Sat, Jun 13, 2009 at 7:46 AM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
> > Hi
> > Thanks for getting back to me.  I have briefly looked at Hadoop.  It
> looks
> > interesting but I haven't thought about using it for indexing.  But I
> think
> > it might be on the plan.  I am doing the initial investigation and
> > technology frameworks.  I would be very interested to get an idea of how
> you
> > are designing your cluster indexing.  I have been doing a documentation
> > which I would be happy to show you.
> >
> >
> > Cheers
> > Amin
> >
> >
> > On Thu, Jun 11, 2009 at 5:29 PM, Tarandeep Singh <tarandeep@gmail.com
> >wrote:
> >
> >> Hi Amin,
> >>
> >> I am also working on similar implementation. But right now I am working
> on
> >> building Lucene Index in parallel using Hadoop.
> >>
> >> How are you building your Lucene index (or indexes) ? Have you tried
> >> Hadoop?
> >> If yes, we can share some design issues.
> >>
> >> Also, for serving Lucene indexes over cluster, you might be interested
> in
> >> Katta project-
> >> katta.sourceforge.net
> >>
> >> -Tarandeep
> >>
> >> On Thu, Jun 11, 2009 at 1:04 AM, Amin Mohammed-Coleman <
> aminmc@gmail.com
> >> >wrote:
> >>
> >> > Hi
> >> >
> >> > we are thinking of implementing a solution like this.  All nodes on
> the
> >> > cluster has a local index from which searches are performed.  Once an
> >> > update
> >> > on the index needs to happen the work is sent to a queue where there
> is
> >> a
> >> > master who is responsible for updating the central index.  The master
> is
> >> > then responsible for propagating/notifying other nodes in the cluster
> >> that
> >> > their local index is not in sync.  The local index worker can perform
> an
> >> > update against the master index.
> >> >
> >> > We haven't implemented this yet but its the approach we'll take
> inorder
> >> to
> >> > deal with clustering.  If anyone has any ideas which could improve
> this
> >> or
> >> > a
> >> > better approach your thoughts will be appreciated.
> >> >
> >> > Cheers
> >> > Amin
> >> >
> >> > On Wed, Jun 10, 2009 at 10:15 AM, Ian Lea <ian.lea@gmail.com> wrote:
> >> >
> >> > > I'd recommend using your favourite queueing service to pass all
> >> > > updates to a central process, the one and only process that updates
> >> > > the index.  If you don't already have a favourite queueing service,
> >> > >
> >> >
> >>
> http://en.wikipedia.org/wiki/Java_Message_Service#Provider_implementations
> >> > > lists several JMS implementations.
> >> > >
> >> > >
> >> > > --
> >> > > Ian.
> >> > >
> >> > > On Tue, Jun 9, 2009 at 9:02 PM, Newman, Billy<Billy.Newman@itt.com>
> >> > wrote:
> >> > > > I am trying to figure out the best way to add to a lucene index
> >> across
> >> > a
> >> > > clustered app server.  I cannot grab an IndexWriter for each node
in
> >> the
> >> > > cluster, because I would run into lock file problems.  I am not sure
> >> if I
> >> > > can share one IndexWriter across the cluster because what happens
> when
> >> > two
> >> > > or more nodes update the same thing.  Anyone have any insight?
> >> > > >
> >> > > > Thanks.
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message