lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Rutherglen (JIRA)" <>
Subject [jira] Updated: (LUCENE-1336) Distributed Lucene using Hadoop RPC based RMI with dynamic classloading
Date Tue, 15 Jul 2008 18:19:31 GMT


Jason Rutherglen updated LUCENE-1336:

    Attachment: lucene-1336.patch


Depends on commons-io-1.3.2.jar, hadoop-0.17.1-core.jar

Note about the class loading.  Core classes from Lucene are excluded from dynamic loading
as they are assumed to exist on the server.  These are defined in LuceneClasses.

Test requires executing TestRMIServer before TestRMIClient

Test case creates Query subclass that returns a random string from Query.toString(String field).
 A method is called on TestRMIServer which returns the random string
to the client.  A second Query subclass of the same name as the first is compiled with a different
random string.  The same call is made and the returned strings are different.  This means
the method call worked and the new Query subclass was dynamically loaded.  

Todo: Add test case using Searchable, create IndexWriterService, improve remote exception

> Distributed Lucene using Hadoop RPC based RMI with dynamic classloading
> -----------------------------------------------------------------------
>                 Key: LUCENE-1336
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: contrib/*
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>            Priority: Minor
>         Attachments: lucene-1336.patch
> Hadoop RPC based RMI system for use with Lucene Searchable.  Keeps the application logic
on the client side with removing the need to deploy application logic to the Lucene servers.
 Removes the need to provision new code to potentially hundreds of servers for every application
logic change.  
> The use case is any deployment requiring Lucene on many servers.  This system provides
the added advantage of allowing custom Query and Filter classes (or other classes) to be defined
on for example a development machine and executed on the server without deploying the custom
classes to the servers first.  This can save a lot of time and effort in provisioning, restarting
processes.  In the future this patch will include an IndexWriterService interface which will
enable document indexing.  This will allow subclasses of Analyzer to be dynamically loaded
onto a server as documents are added by the client.
> Hadoop RPC is more scalable than Sun's RMI implementation because it uses non blocking
sockets.  Hadoop RPC is also far easier to understand and customize if needed as it is embodied
in 2 main class files org.apache.hadoop.ipc.Client and org.apache.hadoop.ipc.Server.  
> Features include automatic dynamic classloading.  The dynamic classloading enables newly
compiled client classes inheriting core objects such as Query or Filter to be used to query
the server without first deploying the code to the server.  
> Using RMI dynamic classloading is not used in practice because it is hard to setup, requiring
placing the new code in jar files on a web server on the client.  Then requires custom system
properties to be setup as well as Java security manager configuration.  
> The dynamic classloading in Hadoop RMI for Lucene uses RMI to load the classes.  Custom
serialization and deserialization manages the classes and the class versions on the server
and client side.  New class files are automatically detected and loaded using ClassLoader.getResourceAsStream
and so this system does not require creating a JAR file.  The use of the same networking system
used for the remote method invocation is used for the loading classes over the network.  This
removes the necessity of a separate web server dedicated to the task and makes deployment
a few lines of code.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message