lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noble Paul നോബിള്‍ नोब्ळ् <noble.p...@gmail.com>
Subject Re: Can I run Lucene in google app engine?
Date Mon, 13 Apr 2009 14:11:50 GMT
yeah we never know which jvm has the latest data

you can have a separate app for writes

say solr_master.appspot.com

instead of POSTing data to the master it may be able to pull data (say
using DataImportHandler) and index it and when it has to commit it can
serialize the data into a Blob and save it. This ensures that all the
writes happen on 1 single host

The slaves can have a cron job which polls for the latest data and
fetch the index and update itself

The catch here is that we must have a small index were everything can
be done in 30 secs (that is the longest a task can run).

--Noble



On Mon, Apr 13, 2009 at 3:55 PM, Erdinc Yilmazel
<erdincyilmazel@gmail.com> wrote:
> Do you think writing a Directory based on google datastore is going to
> be useful? Indexing may be done using the cron jobs. I am not sure how
> the RAMDirectory will fit because the requests can be scattered to
> very different jvm instances in app engine environment.
>
> Erdinc
>
> On Mon, Apr 13, 2009 at 11:11 AM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
>> Yes, I believe so (we have a unit test asserting this).
>>
>> But, there's no guarantee of cross-version compatibility of the serialized form.
>>
>> Mike
>>
>> 2009/4/13 Noble Paul നോബിള്‍  नोब्ळ् <noble.paul@gmail.com>:
>>> that is great.
>>>
>>> Is it possible to serialize and deserialize a RAMDirectory?
>>>
>>> --Noble
>>>
>>>
>>>
>>>
>>> On Mon, Apr 13, 2009 at 3:29 PM, Michael McCandless
>>> <lucene@mikemccandless.com> wrote:
>>>> Lucene's ConcurrentMergeScheduler spawns threads, but you can switch
>>>> that to SerialMergeScheduler which does not use threads.
>>>>
>>>> Some optional classes (FilterManager, ParallelMultiSearcher,
>>>> TimeLimitedCollector) spawn threads, but presumably you could avoid
>>>> them.
>>>>
>>>> Otherwise Lucene doesn't create its own threads.
>>>>
>>>> Mike
>>>>
>>>> On Mon, Apr 13, 2009 at 5:20 AM, Erdinc Yilmazel
>>>> <erdincyilmazel@gmail.com> wrote:
>>>>> I looked at the lucene source code out of curiosity to see if it
>>>>> spawns threads in indexing/searching and saw that it does. Are you
>>>>> sure it is going to work even if the index is read only?
>>>>>
>>>>> Erdinc
>>>>>
>>>>> 2009/4/13 Noble Paul നോബിള്‍  नोब्ळ् <noble.paul@gmail.com>:
>>>>>> On Mon, Apr 13, 2009 at 1:10 PM, Chris Lu <chris.lu@gmail.com>
wrote:
>>>>>>> Surely it's possible, but it has too much limitations to prevent
a scalable
>>>>>>> Luceen usage. However, it depends on your requirement.
>>>>>>>
>>>>>>> 1) You can not write index on disk, but you can read files. So
theoretically
>>>>>>> if the index is read-only and small, you can package it with
the war file.
>>>>>> I can probably write the whole index as  Blob and read it and serve
>>>>>> the index from RAM. This index can be loaded once in the start time
>>>>>> and i can setup a cron task to see if it got updated
>>>>>>>
>>>>>>> 2) If you need to update the index, you will have to store the
index with
>>>>>>> Google's data store, just like store an index into databases.
Sure it'll
>>>>>>> work. But performance would suffer because of transferring the
whole index
>>>>>>> into memory, then really start searching. On the other hand,
this could be a
>>>>>>> good solution for small index with per-user data.
>>>>>> as you said this is a solution if the index size is small
>>>>>>
>>>>>>>
>>>>>>> 3) For large changing indexes, you need to find other solutions
to maintain
>>>>>>> lucene index.
>>>>>>>
>>>>>>> My personal opinion is, finding a $20/month VPS hosting is far
easier than
>>>>>>> changing the way to code.
>>>>>>
>>>>>> The point is ,does Lucens use threads internally which the sandbox
disallows
>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Chris Lu
>>>>>>> -------------------------
>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>> site: http://www.dbsight.net
>>>>>>> demo: http://search.dbsight.com
>>>>>>> Lucene Database Search in 3 minutes:
>>>>>>> http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>> DBSight customer, a shopping comparison site, (anonymous per
request) got
>>>>>>> 2.6 Million Euro funding!
>>>>>>>
>>>>>>>
>>>>>>> Noble Paul ??????? ?????? wrote:
>>>>>>>>
>>>>>>>> Is it possible to run Lucene in google app engine? has anyone
tried it?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Noble Paul
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>



-- 
--Noble Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message