hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Trivial Update of "DistributedLucene" by MarkButler
Date Wed, 19 Dec 2007 11:42:11 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by MarkButler:
http://wiki.apache.org/lucene-hadoop/DistributedLucene

------------------------------------------------------------------------------
  
  One potental problem is a document overwrite implemented as a delete then an add. More than
one client doing this for the same document could result in 0 or 2 documents, instead of 1.
 I guess clients will just need to be relatively coordinated in their activities. Either the
two clients must coordinate, to make sure that they're not updating the same document at the
same time, or use a strategy where updates are routed to the slave that contained the old
version of the document. That would require a broadcast query to figure out which slave that
is.
  
+ Good point. Either the two clients must coordinate, to make sure that they're not updating
the same document at the same time, or use a strategy where updates are routed to the slave
that contained the old version of the document. That would require a broadcast query to figure
out which slave that is.
+ 
  4. How do additions work?
  
- It's unfortunate the master needs to be involved on every document add? That should not
normally be the case. Clients can cache the set of writable index locations and directly submit
new documents without involving the master.
+ The master should not be involved in adds. Clients can cache the set of writable index locations
and directly submit new documents without involving the master.
+ 
+ The master should be out of the loop as much as possible. One approach is that clients randomly
assign documents to indexes and send the updates directly to the indexing node. Alternately,
clients might index locally, then ship the updates to a node packaged as an index. That was
the intent of the addIndex method.
  
  5. How do commits work?
  
  It seems like the master might want to be involved in commits too, or maybe we just rely
on the slave to master heartbeat to kick of immediately after a commit so that index replication
can be initiated? I like the latter approach. New versions are only published as frequently
as clients poll the master for updated IndexLocations. Clients keep a cache of both readable
and updatable index locations that are periodically refreshed.
  
- other comments
+ 6. Broadcasts versus IPC
  
- If deletes were broadcast, and documents could go to any partition, that would be one way
around it (with the downside of a less powerful master that could implement certain distribution
policies). Another way to lessen the master-in-the-middle cost is to make sure one can aggregate
small requests.
+ Currently Hadoop does not support broadcasts, and there are problems getting broadcasts
to work across clusters. Do we need to use broadcasts or can we use the same approach as HDFS
and Hbase?
+ 
+ 7. Finding updateable indexes
  
  Looking at 
  {{{
@@ -112, +118 @@

  }}}
  I'd assumed that the updateable version of an index does not move around very often. Perhaps
a lease mechanism is required. For example, a call to getUpdateableIndex might be valid for
ten minutes.
  
- We might consider a delete() on the master interface too. That way it could
-     3) hide the delete policy (broadcast or directl-to-server-that-has-doc)
-     2) potentially do some batching of deletes
-     1) simply do the delete locally if there is a single index partition
-     and this is a combination master/searcher
- 
- I'm reticent to put any frequently-made call on the master. I'd prefer to keep the master
only involved at an executive level, with all per-document and per-query traffic going directly
from client to slave.
- 
- 
- 
- I was not imagining a real-time system, where the next query after a document is added would
always include that document. Is that a requirement? That's harder.
- 
- At this point I'm mostly trying to see if this functionality would meet the needs of Solr,
Nutch and others.
- 
- Must we include a notion of document identity and/or document version in the mechanism?
Would that facillitate updates and coherency?
- 
- In Nutch a typical case is that you have a bunch of URLs with content that may-or-may-not
have been previously indexed. The approach I'm currently leaning towards is that we'd broadcast
the deletions of all of these to all slaves, then add index them to randomly assigned indexes.
In Nutch multiple clients would naturally be coordinated, since each url is represented only
once in each update cycle.
- 
  === Reply from Doug ===
- 
- Yonik Seeley wrote:
- 
-     On 10/18/06, Doug Cutting <[EMAIL PROTECTED]> wrote:
- 
-         We assume that, within an index, a file with a given name is written
-         only once.
- 
-     Is this necessary, and will we need the lockless patch (that avoids
-     renaming or rewriting *any* files), or is Lucene's current index
-     behavior sufficient?
- 
- It's not strictly required, but it would make index synchronization a lot simpler. Yes,
I was assuming the lockless patch would be committed to Lucene before this project gets very
far. Something more than that would be required in order to keep old versions, but this could
be as simple as a Directory subclass that refuses to remove files for a time.
- 
-     The search side seems straightforward enough, but I haven't totally
-     figured out how the update side should work.
- 
- The master should be out of the loop as much as possible. One approach is that clients randomly
assign documents to indexes and send the updates directly to the indexing node. Alternately,
clients might index locally, then ship the updates to a node packaged as an index. That was
the intent of the addIndex method.
- 
-     One potental problem is a document overwrite implemented as a delete
-     then an add.
-     More than one client doing this for the same document could result in
-     0 or 2 documents, instead of 1.  I guess clients will just need to be
-     relatively coordinated in their activities.
- 
- Good point. Either the two clients must coordinate, to make sure that they're not updating
the same document at the same time, or use a strategy where updates are routed to the slave
that contained the old version of the document. That would require a broadcast query to figure
out which slave that is.
- 
-     It's unfortunate the master needs to be involved on every document add.
  
  That should not normally be the case. Clients can cache the set of writable index locations
and directly submit new documents without involving the master.
  
@@ -280, +240 @@

  It doesn't need to be in the interfaces I don't think, so it depends
  on the scope of the index server implementations.
  
- -Yonik
- 
  
  === Mark's comments ===
  

Mime
View raw message