Michael, thank you.

Query filter only solves half of my problem. Unfortunately I do need  
to have a proper score for some fields.

I ended up extending Term class (I removed sealed attribute which is a  
bad thing). The new myTerm class has one boolean member, omitIdf.  
Then, when I compile my queries, I use myTerm with omitIdf set to  
true, for some fields. Then I extended Similarity cladd and I cast  
Term passes into Idf method to myTerm and only calculate Idf if  
omitIdf is true. Seems to work.

I don't like the solution but that's the best I could do today.

Any thoughts?

Regards,
Artem


On 10/12/2009, at 15:51, Michael Garski <mgarski@myspace-inc.com> wrote:

> Artem,
>
> Do you need any scoring information at all on that field?  How about  
> using a QueryFilter for those fields?
>
> Michael
>
>
> -----Original Message-----
> From: Artem Chereisky [mailto:a.chereisky@gmail.com]
> Sent: Wed 12/9/2009 4:53 PM
> To: lucene-net-user@incubator.apache.org; lucene-net-developer@incubator.apache.org
> Subject: idf on per-field basis
>
> Hi,
>
> I came across a situation when my scores are adversely affected by  
> the IDF
> component. Let me explain.
>
> My index documents contain a number of fields, for some, TF and IDF  
> are
> important and need to be taken into account, for others niether TF  
> nor IDF
> should apply. I dealt with TF by omiting norms during indexing but I  
> can't
> find a way to calculate IDF for certain fields only.
>
> The formula for IDF is defined in Similarity. I have my own  
> implementation
> of Similarity where I can set it to 1 or use the default  
> implementation.
> mySearcher.SetSimilarity is where I tell Lucene which similarity  
> instance to
> use, but that's global, so it applies to all fields in the index.
>
> So, here's my question. Is there a way to calculate IDF on per-field  
> basis?
>
> Regards,
> Art
>
>

I am going to use Lucene for a new public community portal. I would love to
be the one, who tests 2.9.1 for production, but I think, before that Michael
should test at least the numeric fields feature as he said.

Markus

-----Ursprüngliche Nachricht-----
Von: George Aroush [mailto:george@aroush.net] 
Gesendet: Donnerstag, 10. Dezember 2009 04:19
An: lucene-net-user@incubator.apache.org
Betreff: RE: Safe to use Release 2.9.1 Beta for production?

I really want to see folks using 2.9.1, if possible, in production too.
It's the only way we can be sure of its quality.  And yes, all NUnit test
are passing but I want to see how it's doing outside NUnit too.

-- George

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Wednesday, December 09, 2009 3:42 AM
To: lucene-net-user@incubator.apache.org
Subject: AW: Safe to use Release 2.9.1 Beta for production?

I actually need especially the numeric range feature, because I've got a
fulltext search within a specified distance, so I need to filter the results
by spatial data.

Is the Lucene Spatial contrib the right way to go? It's been ported by Roger
Chapman.

Markus


-----Ursprüngliche Nachricht-----
Von: Michael Garski [mailto:mgarski@myspace-inc.com] 
Gesendet: Mittwoch, 9. Dezember 2009 02:26
An: lucene-net-user@incubator.apache.org
Betreff: RE: Safe to use Release 2.9.1 Beta for production?

Two of the features I've been looking forward to and am now testing are
numeric fields, which provide significant performance improvements on
numeric range queries, and the underlying change to per-segment searching
and caching.  The latter sounds rather innocuous at first glance, but when
coupled with a custom merge policy can improve heap utilization for large
indexes.

You can certainly build your application against 2.4 and then migrate to 2.9
at a later time if you are under time constraints to deliver something
quickly.

Michael

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Tuesday, December 08, 2009 1:24 PM
To: lucene-net-user@incubator.apache.org
Subject: AW: Safe to use Release 2.9.1 Beta for production?

Thanks for your advice.

Are there any mature features I might miss when using version 2.4.0?

Markus

-----Ursprüngliche Nachricht-----
Von: xzxz@mail.ru [mailto:xzxz@mail.ru] 
Gesendet: Dienstag, 8. Dezember 2009 18:23
An: lucene-net-user@incubator.apache.org
Betreff: Re: Safe to use Release 2.9.1 Beta for production?

I would not recommend use 2.9.1 in the production right now (it may be 
ok, but may not). It is safer to use 2.4.0.

---
Andrei

Markus Wolters wrote:
> Hi @all,
>
>  
>
> I'm new to Lucene(.NET) and want to include it into my current ASP.NET
> project I'm working on to support fuzzy full-text searches.
>
>  
>
> I am unsure which version to use. Is it safe already to use the /trunk
> version, which is 2.9.1 Beta right now? Or should I better use the tag
> 2.4.0? I need special querys  including spatial data, so I thought the
> latest versions would be a good starting point...
>
>  
>
> Anyways, keep up the good work!
>
>  
>
> I did some comparison to Sphinx and Xapian, but I think at least for me,
> Lucene is the right way to go.
>
>  
>
> Cheers,
>
> Markus
>
>  
>
>
>   

Hi George,

actually, there IS a port by Robert Chapman unless I'm mistaken:

http://issues.apache.org/jira/browse/LUCENENET-199

I had not yet the time to check it.

Markus

-----Ursprüngliche Nachricht-----
Von: George Aroush [mailto:george@aroush.net] 
Gesendet: Donnerstag, 10. Dezember 2009 04:16
An: lucene-net-user@incubator.apache.org
Betreff: RE: Spatial search with Lucene.Net

There isn't any port of Spatial.Net under contrib.  If someone has ported it, and want to
contribute it, that would be great.
 
-- George

-----Original Message-----
From: Michael Garski [mailto:mgarski@myspace-inc.com] 
Sent: Wednesday, December 09, 2009 5:12 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Spatial search with Lucene.Net

I'm not familiar with any of the spatial search libraries in the contrib section, however
they may need to be updated to use 2.9's numeric fields.

Michael 

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Wednesday, December 09, 2009 12:46 AM
To: lucene-net-user@incubator.apache.org
Subject: Spatial search with Lucene.Net

What is the best way to incorporate spatial search data into lucene?

The Spatial.Net contrib, which is been ported by Robert Chapman or this code
if found here:

http://sujitpal.blogspot.com/2008/02/spatial-search-with-lucene.html

Markus

Hi Michael

Thanks, I finally found my problem. The final query text executed by Lucene
QueryParser was accidentally modified by another function. using wildcard
query is okay and no problem so far.

Thanks.

Floyd



2009/12/10 Michael Garski <mgarski@myspace-inc.com>

> Be sure to use the same analyzer at search time that you do at index
> time.  The StandardAnalyzer will keep the numbers and allow you to
> search for the text "2009123" in the field "title" with "title:2009123"
> or "title:2009*".  I'm not sure why you are having difficulty using the
> wildcard query.
>
> Wildcard queries do not perform the best if you have a large number of
> terms that could match the wildcard.  If that is the case you will want
> to alter the analysis of your text.
>
> Michael
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Sunday, December 06, 2009 7:44 PM
> To: lucene-net-user@incubator.apache.org
>  Subject: Re: How to search number in text field?
>
> Hi Michael
>
> The query is constructed using QueryText.
> Partial of my code as floowing
>
> string queryText = "title:2009123";
> QueryParser parser = new QueryParser(SpecialFields.Title, Analyzer);
>            parser.SetLowercaseExpandedTerms(false);
>            Lucene.Net.Search.Query query
> =parser.Parse(queryText);
> searcher.Search(query); //searcher is an IndexSearcher instance
>
> Floyd
>
>
>
> 2009/12/5 Michael Garski <mgarski@myspace-inc.com>
>
> > Floyd,
> >
> > How are you constructing the query?
> >
> > Michael
> >
> > -----Original Message-----
> > From: Floyd Wu [mailto:floyd.wu@gmail.com]
> > Sent: Thursday, December 03, 2009 10:33 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: How to search number in text field?
> >
> > Hi all,
> > When using StandardAnalyzer indexing documents, Lucene.Net did not
> > search
> > for numbers in text fields with numbers.
> > For example, I have built a index which title is 2009123.
> > Fire the querystring as "title:2009" returns no records and even if
> > using
> > title:2009*
> > There does exist record with title value=2009123;
> > How can I search this record when my client really want to use 2009 as
> > keyword for searching?
> >
> > Thanks
> >
> > Floyd
> >
> >
>
>

Artem,

Do you need any scoring information at all on that field?  How about using a QueryFilter for
those fields?

Michael


-----Original Message-----
From: Artem Chereisky [mailto:a.chereisky@gmail.com]
Sent: Wed 12/9/2009 4:53 PM
To: lucene-net-user@incubator.apache.org; lucene-net-developer@incubator.apache.org
Subject: idf on per-field basis
 
Hi,

I came across a situation when my scores are adversely affected by the IDF
component. Let me explain.

My index documents contain a number of fields, for some, TF and IDF are
important and need to be taken into account, for others niether TF nor IDF
should apply. I dealt with TF by omiting norms during indexing but I can't
find a way to calculate IDF for certain fields only.

The formula for IDF is defined in Similarity. I have my own implementation
of Similarity where I can set it to 1 or use the default implementation.
mySearcher.SetSimilarity is where I tell Lucene which similarity instance to
use, but that's global, so it applies to all fields in the index.

So, here's my question. Is there a way to calculate IDF on per-field basis?

Regards,
Art

 

I really want to see folks using 2.9.1, if possible, in production too.
It's the only way we can be sure of its quality.  And yes, all NUnit test
are passing but I want to see how it's doing outside NUnit too.

-- George

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Wednesday, December 09, 2009 3:42 AM
To: lucene-net-user@incubator.apache.org
Subject: AW: Safe to use Release 2.9.1 Beta for production?

I actually need especially the numeric range feature, because I've got a
fulltext search within a specified distance, so I need to filter the results
by spatial data.

Is the Lucene Spatial contrib the right way to go? It's been ported by Roger
Chapman.

Markus


-----Ursprüngliche Nachricht-----
Von: Michael Garski [mailto:mgarski@myspace-inc.com] 
Gesendet: Mittwoch, 9. Dezember 2009 02:26
An: lucene-net-user@incubator.apache.org
Betreff: RE: Safe to use Release 2.9.1 Beta for production?

Two of the features I've been looking forward to and am now testing are
numeric fields, which provide significant performance improvements on
numeric range queries, and the underlying change to per-segment searching
and caching.  The latter sounds rather innocuous at first glance, but when
coupled with a custom merge policy can improve heap utilization for large
indexes.

You can certainly build your application against 2.4 and then migrate to 2.9
at a later time if you are under time constraints to deliver something
quickly.

Michael

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Tuesday, December 08, 2009 1:24 PM
To: lucene-net-user@incubator.apache.org
Subject: AW: Safe to use Release 2.9.1 Beta for production?

Thanks for your advice.

Are there any mature features I might miss when using version 2.4.0?

Markus

-----Ursprüngliche Nachricht-----
Von: xzxz@mail.ru [mailto:xzxz@mail.ru] 
Gesendet: Dienstag, 8. Dezember 2009 18:23
An: lucene-net-user@incubator.apache.org
Betreff: Re: Safe to use Release 2.9.1 Beta for production?

I would not recommend use 2.9.1 in the production right now (it may be 
ok, but may not). It is safer to use 2.4.0.

---
Andrei

Markus Wolters wrote:
> Hi @all,
>
>  
>
> I'm new to Lucene(.NET) and want to include it into my current ASP.NET
> project I'm working on to support fuzzy full-text searches.
>
>  
>
> I am unsure which version to use. Is it safe already to use the /trunk
> version, which is 2.9.1 Beta right now? Or should I better use the tag
> 2.4.0? I need special querys  including spatial data, so I thought the
> latest versions would be a good starting point...
>
>  
>
> Anyways, keep up the good work!
>
>  
>
> I did some comparison to Sphinx and Xapian, but I think at least for me,
> Lucene is the right way to go.
>
>  
>
> Cheers,
>
> Markus
>
>  
>
>
>   

There isn't any port of Spatial.Net under contrib.  If someone has ported it, and want to contribute
it, that would be great.
 
-- George

-----Original Message-----
From: Michael Garski [mailto:mgarski@myspace-inc.com] 
Sent: Wednesday, December 09, 2009 5:12 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Spatial search with Lucene.Net

I'm not familiar with any of the spatial search libraries in the contrib section, however
they may need to be updated to use 2.9's numeric fields.

Michael 

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Wednesday, December 09, 2009 12:46 AM
To: lucene-net-user@incubator.apache.org
Subject: Spatial search with Lucene.Net

What is the best way to incorporate spatial search data into lucene?

The Spatial.Net contrib, which is been ported by Robert Chapman or this code
if found here:

http://sujitpal.blogspot.com/2008/02/spatial-search-with-lucene.html

Markus

Hi,

I came across a situation when my scores are adversely affected by the IDF
component. Let me explain.

My index documents contain a number of fields, for some, TF and IDF are
important and need to be taken into account, for others niether TF nor IDF
should apply. I dealt with TF by omiting norms during indexing but I can't
find a way to calculate IDF for certain fields only.

The formula for IDF is defined in Similarity. I have my own implementation
of Similarity where I can set it to 1 or use the default implementation.
mySearcher.SetSimilarity is where I tell Lucene which similarity instance to
use, but that's global, so it applies to all fields in the index.

So, here's my question. Is there a way to calculate IDF on per-field basis?

Regards,
Art

Be sure to use the same analyzer at search time that you do at index
time.  The StandardAnalyzer will keep the numbers and allow you to
search for the text "2009123" in the field "title" with "title:2009123"
or "title:2009*".  I'm not sure why you are having difficulty using the
wildcard query.  

Wildcard queries do not perform the best if you have a large number of
terms that could match the wildcard.  If that is the case you will want
to alter the analysis of your text.

Michael

-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com] 
Sent: Sunday, December 06, 2009 7:44 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: How to search number in text field?

Hi Michael

The query is constructed using QueryText.
Partial of my code as floowing

string queryText = "title:2009123";
QueryParser parser = new QueryParser(SpecialFields.Title, Analyzer);
            parser.SetLowercaseExpandedTerms(false);
            Lucene.Net.Search.Query query
=parser.Parse(queryText);
searcher.Search(query); //searcher is an IndexSearcher instance

Floyd



2009/12/5 Michael Garski <mgarski@myspace-inc.com>

> Floyd,
>
> How are you constructing the query?
>
> Michael
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, December 03, 2009 10:33 PM
> To: lucene-net-user@incubator.apache.org
> Subject: How to search number in text field?
>
> Hi all,
> When using StandardAnalyzer indexing documents, Lucene.Net did not
> search
> for numbers in text fields with numbers.
> For example, I have built a index which title is 2009123.
> Fire the querystring as "title:2009" returns no records and even if
> using
> title:2009*
> There does exist record with title value=2009123;
> How can I search this record when my client really want to use 2009 as
> keyword for searching?
>
> Thanks
>
> Floyd
>
>

I'm not familiar with any of the spatial search libraries in the contrib section, however they
may need to be updated to use 2.9's numeric fields.

Michael 

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Wednesday, December 09, 2009 12:46 AM
To: lucene-net-user@incubator.apache.org
Subject: Spatial search with Lucene.Net

What is the best way to incorporate spatial search data into lucene?

The Spatial.Net contrib, which is been ported by Robert Chapman or this code
if found here:

http://sujitpal.blogspot.com/2008/02/spatial-search-with-lucene.html

Markus

Thanks a lot! This works perfect! I added a line

System.Environment.SetEnvironmentVariable("OS", "linux")

Do you know how to file a bug to the developers?!

Thanks & Regards,

Johannes



Johannes von Drachenfels
Phone: +49-7231-9223800
Mobile: +49-171-6710815
 
Drachenfels GmbH
Bleichstrasse 56
75173 Pforzheim
Germany
 
Geschäftsführer: Johannes von Drachenfels
Sitz der Gesellschaft: Pforzheim
Handelsregister: Registergericht Mannheim: HRB 504389
 
Notice: This transmittal and/or attachments may be privileged or confidential. If you are
not the intended recipient, you are hereby notified that you have received this transmittal
in error; any review, dissemination, or copying is strictly prohibited. If you received this
transmittal in error, please notify us immediately by reply and immediately delete this message
and all its attachments. Thank you. 


-----Ursprüngliche Nachricht-----
Von: news [mailto:news@ger.gmane.org] Im Auftrag von Robert Jordan
Gesendet: Mittwoch, 9. Dezember 2009 15:17
An: lucene-net-user@incubator.apache.org
Betreff: Re: System.TypeInitializationException on linux (vbnc)

On 09.12.2009 14:57, Johannes Drachenfels wrote:
> Hi,
>
>
>
> sorry for re-posting - I am new to this list...
>
>
>
> I am switching from lucene.net 2.0.0.4 to lucene.net 2.9.1.1 and I have
> problems in a very early state.
>
>
>
> With version 2.0.0.4 it works fine!
>
> With version 2.9.1.1 on Microsoft it works fine
>
> With version 2.9.1.1 on linux I always get the following errors:
>
>
>
> ################################
>
> System.TypeInitializationException: An exception was thrown by the type
> initializer for Lucene.Net.Store.FSDirectory --->
> System.TypeInitializationException: An exception was thrown by the type
> initializer for Lucene.Net.Util.Constants --->
> System.NullReferenceException: Object reference not set to an instance
> of an object
>
>    at Lucene.Net.Util.Constants..cctor () [0x00000]
>
>    --- End of inner exception stack trace ---
>
>    at Lucene.Net.Store.FSDirectory..cctor () [0x00000]
>
>    --- End of inner exception stack trace ---
>
>    at ConsoleApplication1.Module1.Main () [0x00000]


This is probably caused by this line in Lucene.Net.Util.Constants.cs:

public static readonly System.String OS_NAME = 
System.Environment.GetEnvironmentVariable("OS");

Try to assign some value to the env var "OS":

	OS=foo mono yourapp.exe

Robert

On 09.12.2009 14:57, Johannes Drachenfels wrote:
> Hi,
>
>
>
> sorry for re-posting - I am new to this list...
>
>
>
> I am switching from lucene.net 2.0.0.4 to lucene.net 2.9.1.1 and I have
> problems in a very early state.
>
>
>
> With version 2.0.0.4 it works fine!
>
> With version 2.9.1.1 on Microsoft it works fine
>
> With version 2.9.1.1 on linux I always get the following errors:
>
>
>
> ################################
>
> System.TypeInitializationException: An exception was thrown by the type
> initializer for Lucene.Net.Store.FSDirectory --->
> System.TypeInitializationException: An exception was thrown by the type
> initializer for Lucene.Net.Util.Constants --->
> System.NullReferenceException: Object reference not set to an instance
> of an object
>
>    at Lucene.Net.Util.Constants..cctor () [0x00000]
>
>    --- End of inner exception stack trace ---
>
>    at Lucene.Net.Store.FSDirectory..cctor () [0x00000]
>
>    --- End of inner exception stack trace ---
>
>    at ConsoleApplication1.Module1.Main () [0x00000]


This is probably caused by this line in Lucene.Net.Util.Constants.cs:

public static readonly System.String OS_NAME = 
System.Environment.GetEnvironmentVariable("OS");

Try to assign some value to the env var "OS":

	OS=foo mono yourapp.exe

Robert

Hi,

 

sorry for re-posting - I am new to this list...

 

I am switching from lucene.net 2.0.0.4 to lucene.net 2.9.1.1 and I have
problems in a very early state. 

 

With version 2.0.0.4 it works fine!

With version 2.9.1.1 on Microsoft it works fine

With version 2.9.1.1 on linux I always get the following errors:

 

################################

System.TypeInitializationException: An exception was thrown by the type
initializer for Lucene.Net.Store.FSDirectory --->
System.TypeInitializationException: An exception was thrown by the type
initializer for Lucene.Net.Util.Constants --->
System.NullReferenceException: Object reference not set to an instance
of an object

  at Lucene.Net.Util.Constants..cctor () [0x00000]

  --- End of inner exception stack trace ---

  at Lucene.Net.Store.FSDirectory..cctor () [0x00000]

  --- End of inner exception stack trace ---

  at ConsoleApplication1.Module1.Main () [0x00000]

################################

System.TypeInitializationException: An exception was thrown by the type
initializer for Lucene.Net.Index.IndexWriter --->
System.TypeInitializationException: An exception was thrown by the type
initializer for Lucene.Net.Index.DocumentsWriter --->
System.TypeInitializationException: An exception was thrown by the type
initializer for Lucene.Net.Util.Constants --->
System.NullReferenceException: Object reference not set to an instance
of an object

  at Lucene.Net.Util.Constants..cctor () [0x00000]

  --- End of inner exception stack trace ---

  at Lucene.Net.Store.FSDirectory..cctor () [0x00000]

  --- End of inner exception stack trace ---

  at Lucene.Net.Index.IndexWriter..cctor () [0x00000]

  --- End of inner exception stack trace ---

  at ConsoleApplication1.Module1.Main () [0x00000]

 

>From debugging I found the reason for the errors:

 

            dir =
Lucene.Net.Store.FSDirectory.GetDirectory(indexFileLocation, True)

 

and

 

            indexWriter = New Lucene.Net.Index.IndexWriter(dirR,
analyzer, True)

 

but I do not have any idea what's happen here.... I have attached the
source of my test application! 

 

Thanks for help & Regards,

 

Johannes

 

What is the best way to incorporate spatial search data into lucene?

The Spatial.Net contrib, which is been ported by Robert Chapman or this code
if found here:

http://sujitpal.blogspot.com/2008/02/spatial-search-with-lucene.html

Markus

I actually need especially the numeric range feature, because I've got a
fulltext search within a specified distance, so I need to filter the results
by spatial data.

Is the Lucene Spatial contrib the right way to go? It's been ported by Roger
Chapman.

Markus


-----Ursprüngliche Nachricht-----
Von: Michael Garski [mailto:mgarski@myspace-inc.com] 
Gesendet: Mittwoch, 9. Dezember 2009 02:26
An: lucene-net-user@incubator.apache.org
Betreff: RE: Safe to use Release 2.9.1 Beta for production?

Two of the features I've been looking forward to and am now testing are
numeric fields, which provide significant performance improvements on
numeric range queries, and the underlying change to per-segment searching
and caching.  The latter sounds rather innocuous at first glance, but when
coupled with a custom merge policy can improve heap utilization for large
indexes.

You can certainly build your application against 2.4 and then migrate to 2.9
at a later time if you are under time constraints to deliver something
quickly.

Michael

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Tuesday, December 08, 2009 1:24 PM
To: lucene-net-user@incubator.apache.org
Subject: AW: Safe to use Release 2.9.1 Beta for production?

Thanks for your advice.

Are there any mature features I might miss when using version 2.4.0?

Markus

-----Ursprüngliche Nachricht-----
Von: xzxz@mail.ru [mailto:xzxz@mail.ru] 
Gesendet: Dienstag, 8. Dezember 2009 18:23
An: lucene-net-user@incubator.apache.org
Betreff: Re: Safe to use Release 2.9.1 Beta for production?

I would not recommend use 2.9.1 in the production right now (it may be 
ok, but may not). It is safer to use 2.4.0.

---
Andrei

Markus Wolters wrote:
> Hi @all,
>
>  
>
> I'm new to Lucene(.NET) and want to include it into my current ASP.NET
> project I'm working on to support fuzzy full-text searches.
>
>  
>
> I am unsure which version to use. Is it safe already to use the /trunk
> version, which is 2.9.1 Beta right now? Or should I better use the tag
> 2.4.0? I need special querys  including spatial data, so I thought the
> latest versions would be a good starting point...
>
>  
>
> Anyways, keep up the good work!
>
>  
>
> I did some comparison to Sphinx and Xapian, but I think at least for me,
> Lucene is the right way to go.
>
>  
>
> Cheers,
>
> Markus
>
>  
>
>
>   

Two of the features I've been looking forward to and am now testing are numeric fields, which
provide significant performance improvements on numeric range queries, and the underlying
change to per-segment searching and caching.  The latter sounds rather innocuous at first
glance, but when coupled with a custom merge policy can improve heap utilization for large
indexes.

You can certainly build your application against 2.4 and then migrate to 2.9 at a later time
if you are under time constraints to deliver something quickly.

Michael

-----Original Message-----
From: Markus Wolters [mailto:markus@naxma.com] 
Sent: Tuesday, December 08, 2009 1:24 PM
To: lucene-net-user@incubator.apache.org
Subject: AW: Safe to use Release 2.9.1 Beta for production?

Thanks for your advice.

Are there any mature features I might miss when using version 2.4.0?

Markus

-----Ursprüngliche Nachricht-----
Von: xzxz@mail.ru [mailto:xzxz@mail.ru] 
Gesendet: Dienstag, 8. Dezember 2009 18:23
An: lucene-net-user@incubator.apache.org
Betreff: Re: Safe to use Release 2.9.1 Beta for production?

I would not recommend use 2.9.1 in the production right now (it may be 
ok, but may not). It is safer to use 2.4.0.

---
Andrei

Markus Wolters wrote:
> Hi @all,
>
>  
>
> I'm new to Lucene(.NET) and want to include it into my current ASP.NET
> project I'm working on to support fuzzy full-text searches.
>
>  
>
> I am unsure which version to use. Is it safe already to use the /trunk
> version, which is 2.9.1 Beta right now? Or should I better use the tag
> 2.4.0? I need special querys  including spatial data, so I thought the
> latest versions would be a good starting point...
>
>  
>
> Anyways, keep up the good work!
>
>  
>
> I did some comparison to Sphinx and Xapian, but I think at least for me,
> Lucene is the right way to go.
>
>  
>
> Cheers,
>
> Markus
>
>  
>
>
>   

Thanks for your advice.

Are there any mature features I might miss when using version 2.4.0?

Markus

-----Ursprüngliche Nachricht-----
Von: xzxz@mail.ru [mailto:xzxz@mail.ru] 
Gesendet: Dienstag, 8. Dezember 2009 18:23
An: lucene-net-user@incubator.apache.org
Betreff: Re: Safe to use Release 2.9.1 Beta for production?

I would not recommend use 2.9.1 in the production right now (it may be 
ok, but may not). It is safer to use 2.4.0.

---
Andrei

Markus Wolters wrote:
> Hi @all,
>
>  
>
> I'm new to Lucene(.NET) and want to include it into my current ASP.NET
> project I'm working on to support fuzzy full-text searches.
>
>  
>
> I am unsure which version to use. Is it safe already to use the /trunk
> version, which is 2.9.1 Beta right now? Or should I better use the tag
> 2.4.0? I need special querys  including spatial data, so I thought the
> latest versions would be a good starting point...
>
>  
>
> Anyways, keep up the good work!
>
>  
>
> I did some comparison to Sphinx and Xapian, but I think at least for me,
> Lucene is the right way to go.
>
>  
>
> Cheers,
>
> Markus
>
>  
>
>
>   

I would not recommend use 2.9.1 in the production right now (it may be 
ok, but may not). It is safer to use 2.4.0.

---
Andrei

Markus Wolters wrote:
> Hi @all,
>
>  
>
> I'm new to Lucene(.NET) and want to include it into my current ASP.NET
> project I'm working on to support fuzzy full-text searches.
>
>  
>
> I am unsure which version to use. Is it safe already to use the /trunk
> version, which is 2.9.1 Beta right now? Or should I better use the tag
> 2.4.0? I need special querys  including spatial data, so I thought the
> latest versions would be a good starting point...
>
>  
>
> Anyways, keep up the good work!
>
>  
>
> I did some comparison to Sphinx and Xapian, but I think at least for me,
> Lucene is the right way to go.
>
>  
>
> Cheers,
>
> Markus
>
>  
>
>
>   

Hi @all,

 

I'm new to Lucene(.NET) and want to include it into my current ASP.NET
project I'm working on to support fuzzy full-text searches.

 

I am unsure which version to use. Is it safe already to use the /trunk
version, which is 2.9.1 Beta right now? Or should I better use the tag
2.4.0? I need special querys  including spatial data, so I thought the
latest versions would be a good starting point...

 

Anyways, keep up the good work!

 

I did some comparison to Sphinx and Xapian, but I think at least for me,
Lucene is the right way to go.

 

Cheers,

Markus

 

Hi Michael

The query is constructed using QueryText.
Partial of my code as floowing

string queryText = "title:2009123";
QueryParser parser = new QueryParser(SpecialFields.Title, Analyzer);
            parser.SetLowercaseExpandedTerms(false);
            Lucene.Net.Search.Query query
=parser.Parse(queryText);
searcher.Search(query); //searcher is an IndexSearcher instance

Floyd



2009/12/5 Michael Garski <mgarski@myspace-inc.com>

> Floyd,
>
> How are you constructing the query?
>
> Michael
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Thursday, December 03, 2009 10:33 PM
> To: lucene-net-user@incubator.apache.org
> Subject: How to search number in text field?
>
> Hi all,
> When using StandardAnalyzer indexing documents, Lucene.Net did not
> search
> for numbers in text fields with numbers.
> For example, I have built a index which title is 2009123.
> Fire the querystring as "title:2009" returns no records and even if
> using
> title:2009*
> There does exist record with title value=2009123;
> How can I search this record when my client really want to use 2009 as
> keyword for searching?
>
> Thanks
>
> Floyd
>
>

Floyd,

How are you constructing the query?

Michael

-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com] 
Sent: Thursday, December 03, 2009 10:33 PM
To: lucene-net-user@incubator.apache.org
Subject: How to search number in text field?

Hi all,
When using StandardAnalyzer indexing documents, Lucene.Net did not
search
for numbers in text fields with numbers.
For example, I have built a index which title is 2009123.
Fire the querystring as "title:2009" returns no records and even if
using
title:2009*
There does exist record with title value=2009123;
How can I search this record when my client really want to use 2009 as
keyword for searching?

Thanks

Floyd

Hi all,
When using StandardAnalyzer indexing documents, Lucene.Net did not search
for numbers in text fields with numbers.
For example, I have built a index which title is 2009123.
Fire the querystring as "title:2009" returns no records and even if using
title:2009*
There does exist record with title value=2009123;
How can I search this record when my client really want to use 2009 as
keyword for searching?

Thanks

Floyd

Hi Michael, thx for your assistance! 

I haven't tried the Lucene 2.4 though I have download it from the svn, since I prefer a .dll
file. I haven't tried build it from visual studio, I'm affraid building it by hand will take
some more time for me. I've finally tricks the program, by indexing 30000 doc, close the writer,
then index the next 30000 docs, and it seems to work, though I still not understand why it
goes that way. I think it's because my single document have too much field and each field's
content is rather long. I tried to index one field only, and it works even for 100000 docs,
so it must have something to do with the content of the doc..

And for the catch statement, thanks for the link, it give me some insight how to use it properly!

Thanks again Michael, for your suggestions!




________________________________
From: Michael Garski <mgarski@myspace-inc.com>
To: lucene-net-user@incubator.apache.org
Sent: Fri, November 27, 2009 4:05:59 AM
Subject: RE: [ask] index out of range exceptions.

Marcelino - 

Give it a try with Lucene 2.4 - there is a tag in the SVN repository for it.  If it still
occurs, a stack trace with line numbers inside of Lucene would be helpful.  You may want to
remove your catch statement completely as you are just re-throwing the exception anyways and
could be suppressing the source.

http://blogs.msdn.com/jmstall/archive/2007/02/07/catch-rethrow.aspx

Michael


-----Original Message-----
From: Marcelino Ponty [mailto:marcelino_ponty@yahoo.com]
Sent: Thu 11/26/2009 12:51 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: [ask] index out of range exceptions.

Hi Michael!

Thanks for your fast response!

In the stack trace:
at Lucene.Net.Index.DocumentsWriter.Abort(AbortException ae) at Lucene.Net.Index.DocumentsWriter.UpdateDocument(Document
doc, Analyzer analyzer, Term delTerm) at Lucene.Net.Index.DocumentsWriter.AddDocument(Document
doc, Analyzer analyzer) at Lucene.Net.Index.IndexWriter.AddDocument(Document doc, Analyzer
analyzer) at Lucene.Net.Index.IndexWriter.AddDocument(Document doc) at Atmalib.Lucene.Indexing.ExecuteIndexing(Directory
dir) in C:\Documents and Settings\Administrator\My Documents\Visual Studio 2005\Projects\CreateIndex\CreateIndex\App_Code\Indexing.cs:line
95

I use Lucene.NET version 2.3.1.2. 

And my program is absolutely simple: 

//main program
        protected static void ExecuteIndexing(Directory dir)
        {
        IndexWriter writer = new IndexWriter(dir, new StandardAnalyzer(), true);
            try
            {
                string query = "select top 40000 * from v_simple_artikel";
//use public class method to query database
                SqlDataReader row = Atmalib.DataAccess.GetDataReaderFromQuery(query); 

//read query results and add document
                while (row.Read())
                {
                   Document doc = new Document();
               doc.Add(new Field("kode_koleksi", row["kode_koleksi"].ToString(), Field.Store.YES,
Field.Index.NO, Field.TermVector.NO));
               doc.Add(new Field("kode_artikel", row["kode_artikel"].ToString(), Field.Store.YES,
Field.Index.NO, Field.TermVector.NO));

//....
//adding many other fields
//...

Field.Index.TOKENIZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
               doc.Add(new Field("kata_kunci_jurnal", row["kata_kunci_jurnal"].ToString(),
Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
                   writer.AddDocument(doc);
                }
                writer.Optimize();
            }
            catch (Exception exp)
            {
                throw exp;
            }

            finally
            {
                writer.Close();
            }
        }
//end of main program

I'll be very grateful if you can give me assistance. Thank you!!



________________________________
From: Michael Garski <mgarski@myspace-inc.com>
To: lucene-net-user@incubator.apache.org
Sent: Fri, November 27, 2009 2:51:19 AM
Subject: Re: [ask] index out of range exceptions.

Hi Marcelino,

Can you provide tha stack trace from the exception and a code snippet/description of what
you are doing when it is thrown along with the version of Lucene.net you are using?

Michael

On Nov 26, 2009, at 11:48 AM, "Marcelino Ponty" <marcelino_ponty@yahoo.com> wrote:

> Hi all!
> 
> I'm a new user of Lucene.NET, but I have experience once in using Ferret which is written
in Ruby language. I've come to a problem and hope any of you can help me.
> 
> I'm going to index 50,000 doc, but it failed and give
> 
> System.IndexOutOfRangeExceptions: Index was outside the bound of the array.
> 
> I test with 30,000 doc and it succeed. I think this has something to do with basic predefined
parameters while setting up the indexing process which I should have set, but I don't know
which parameter it is and where should I set it. Do you guys have any idea? I think the answer
should be simple. In Ferret, I didn't meet this problem.
> 
> Thanks for any assistance!
> Regards,
> Marcelino Ponty
> (Phone +62819 - 3223 54 84)
> "Ad Maiorem Dei Gloriam"
> 
> 


      

Marcelino - 

Give it a try with Lucene 2.4 - there is a tag in the SVN repository for it.  If it still
occurs, a stack trace with line numbers inside of Lucene would be helpful.  You may want to
remove your catch statement completely as you are just re-throwing the exception anyways and
could be suppressing the source.

http://blogs.msdn.com/jmstall/archive/2007/02/07/catch-rethrow.aspx

Michael


-----Original Message-----
From: Marcelino Ponty [mailto:marcelino_ponty@yahoo.com]
Sent: Thu 11/26/2009 12:51 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: [ask] index out of range exceptions.
 
Hi Michael!

Thanks for your fast response!

In the stack trace:
at Lucene.Net.Index.DocumentsWriter.Abort(AbortException ae) at Lucene.Net.Index.DocumentsWriter.UpdateDocument(Document
doc, Analyzer analyzer, Term delTerm) at Lucene.Net.Index.DocumentsWriter.AddDocument(Document
doc, Analyzer analyzer) at Lucene.Net.Index.IndexWriter.AddDocument(Document doc, Analyzer
analyzer) at Lucene.Net.Index.IndexWriter.AddDocument(Document doc) at Atmalib.Lucene.Indexing.ExecuteIndexing(Directory
dir) in C:\Documents and Settings\Administrator\My Documents\Visual Studio 2005\Projects\CreateIndex\CreateIndex\App_Code\Indexing.cs:line
95

I use Lucene.NET version 2.3.1.2. 

And my program is absolutely simple: 

//main program
        protected static void ExecuteIndexing(Directory dir)
        {
        IndexWriter writer = new IndexWriter(dir, new StandardAnalyzer(), true);
            try
            {
                string query = "select top 40000 * from v_simple_artikel";
//use public class method to query database
                SqlDataReader row = Atmalib.DataAccess.GetDataReaderFromQuery(query); 

//read query results and add document
                while (row.Read())
                {
                   Document doc = new Document();
               doc.Add(new Field("kode_koleksi", row["kode_koleksi"].ToString(), Field.Store.YES,
Field.Index.NO, Field.TermVector.NO));
               doc.Add(new Field("kode_artikel", row["kode_artikel"].ToString(), Field.Store.YES,
Field.Index.NO, Field.TermVector.NO));

//....
//adding many other fields
//...

Field.Index.TOKENIZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
               doc.Add(new Field("kata_kunci_jurnal", row["kata_kunci_jurnal"].ToString(),
Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
                   writer.AddDocument(doc);
                }
                writer.Optimize();
            }
            catch (Exception exp)
            {
                throw exp;
            }

            finally
            {
                writer.Close();
            }
        }
//end of main program

I'll be very grateful if you can give me assistance. Thank you!!



________________________________
From: Michael Garski <mgarski@myspace-inc.com>
To: lucene-net-user@incubator.apache.org
Sent: Fri, November 27, 2009 2:51:19 AM
Subject: Re: [ask] index out of range exceptions.

Hi Marcelino,

Can you provide tha stack trace from the exception and a code snippet/description of what
you are doing when it is thrown along with the version of Lucene.net you are using?

Michael

On Nov 26, 2009, at 11:48 AM, "Marcelino Ponty" <marcelino_ponty@yahoo.com> wrote:

> Hi all!
> 
> I'm a new user of Lucene.NET, but I have experience once in using Ferret which is written
in Ruby language. I've come to a problem and hope any of you can help me.
> 
> I'm going to index 50,000 doc, but it failed and give
> 
> System.IndexOutOfRangeExceptions: Index was outside the bound of the array.
> 
> I test with 30,000 doc and it succeed. I think this has something to do with basic predefined
parameters while setting up the indexing process which I should have set, but I don't know
which parameter it is and where should I set it. Do you guys have any idea? I think the answer
should be simple. In Ferret, I didn't meet this problem.
> 
> Thanks for any assistance!
> Regards,
> Marcelino Ponty
> (Phone +62819 - 3223 54 84)
> "Ad Maiorem Dei Gloriam"
> 
> 


      

 

Hi Michael!

Thanks for your fast response!

In the stack trace:
at Lucene.Net.Index.DocumentsWriter.Abort(AbortException ae) at Lucene.Net.Index.DocumentsWriter.UpdateDocument(Document
doc, Analyzer analyzer, Term delTerm) at Lucene.Net.Index.DocumentsWriter.AddDocument(Document
doc, Analyzer analyzer) at Lucene.Net.Index.IndexWriter.AddDocument(Document doc, Analyzer
analyzer) at Lucene.Net.Index.IndexWriter.AddDocument(Document doc) at Atmalib.Lucene.Indexing.ExecuteIndexing(Directory
dir) in C:\Documents and Settings\Administrator\My Documents\Visual Studio 2005\Projects\CreateIndex\CreateIndex\App_Code\Indexing.cs:line
95

I use Lucene.NET version 2.3.1.2. 

And my program is absolutely simple: 

//main program
        protected static void ExecuteIndexing(Directory dir)
        {
        IndexWriter writer = new IndexWriter(dir, new StandardAnalyzer(), true);
            try
            {
                string query = "select top 40000 * from v_simple_artikel";
//use public class method to query database
                SqlDataReader row = Atmalib.DataAccess.GetDataReaderFromQuery(query); 

//read query results and add document
                while (row.Read())
                {
                   Document doc = new Document();
               doc.Add(new Field("kode_koleksi", row["kode_koleksi"].ToString(), Field.Store.YES,
Field.Index.NO, Field.TermVector.NO));
               doc.Add(new Field("kode_artikel", row["kode_artikel"].ToString(), Field.Store.YES,
Field.Index.NO, Field.TermVector.NO));

//....
//adding many other fields
//...

Field.Index.TOKENIZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
               doc.Add(new Field("kata_kunci_jurnal", row["kata_kunci_jurnal"].ToString(),
Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
                   writer.AddDocument(doc);
                }
                writer.Optimize();
            }
            catch (Exception exp)
            {
                throw exp;
            }

            finally
            {
                writer.Close();
            }
        }
//end of main program

I'll be very grateful if you can give me assistance. Thank you!!



________________________________
From: Michael Garski <mgarski@myspace-inc.com>
To: lucene-net-user@incubator.apache.org
Sent: Fri, November 27, 2009 2:51:19 AM
Subject: Re: [ask] index out of range exceptions.

Hi Marcelino,

Can you provide tha stack trace from the exception and a code snippet/description of what
you are doing when it is thrown along with the version of Lucene.net you are using?

Michael

On Nov 26, 2009, at 11:48 AM, "Marcelino Ponty" <marcelino_ponty@yahoo.com> wrote:

> Hi all!
> 
> I'm a new user of Lucene.NET, but I have experience once in using Ferret which is written
in Ruby language. I've come to a problem and hope any of you can help me.
> 
> I'm going to index 50,000 doc, but it failed and give
> 
> System.IndexOutOfRangeExceptions: Index was outside the bound of the array.
> 
> I test with 30,000 doc and it succeed. I think this has something to do with basic predefined
parameters while setting up the indexing process which I should have set, but I don't know
which parameter it is and where should I set it. Do you guys have any idea? I think the answer
should be simple. In Ferret, I didn't meet this problem.
> 
> Thanks for any assistance!
> Regards,
> Marcelino Ponty
> (Phone +62819 - 3223 54 84)
> "Ad Maiorem Dei Gloriam"
> 
> 


      

Hi Marcelino,

Can you provide tha stack trace from the exception and a code snippet/ 
description of what you are doing when it is thrown along with the  
version of Lucene.net you are using?

Michael

On Nov 26, 2009, at 11:48 AM, "Marcelino Ponty" <marcelino_ponty@yahoo.com 
 > wrote:

> Hi all!
>
> I'm a new user of Lucene.NET, but I have experience once in using  
> Ferret which is written in Ruby language. I've come to a problem and  
> hope any of you can help me.
>
> I'm going to index 50,000 doc, but it failed and give
>
> System.IndexOutOfRangeExceptions: Index was outside the bound of the  
> array.
>
> I test with 30,000 doc and it succeed. I think this has something to  
> do with basic predefined parameters while setting up the indexing  
> process which I should have set, but I don't know which parameter it  
> is and where should I set it. Do you guys have any idea? I think the  
> answer should be simple. In Ferret, I didn't meet this problem.
>
> Thanks for any assistance!
> Regards,
> Marcelino Ponty
> (Phone +62819 - 3223 54 84)
> "Ad Maiorem Dei Gloriam"
>
>

Hi all!

I'm a new user of Lucene.NET, but I have experience once in using Ferret which is written
in Ruby language. I've come to a problem and hope any of you can help me.

I'm going to index 50,000 doc, but it failed and give 

System.IndexOutOfRangeExceptions: Index was outside the bound of the array.

I test with 30,000 doc and it succeed. I think this has something to do with basic predefined
parameters while setting up the indexing process which I should have set, but I don't know
which parameter it is and where should I set it. Do you guys have any idea? I think the answer
should be simple. In Ferret, I didn't meet this problem.

Thanks for any assistance!
Regards,
Marcelino Ponty
(Phone +62819 - 3223 54 84)
"Ad Maiorem Dei Gloriam"


      

Eran,

The transactional functionality can rollback changes to an index should
something happen during a commit.  Refer to the methods PrepareCommit &
Rollback.  You would have to implement your own logic to re-process any
changes that were rolled back.

Michael

-----Original Message-----
From: Eran Sevi [mailto:eransevi@gmail.com] 
Sent: Tuesday, November 17, 2009 9:30 AM
To: lucene-net-user@incubator.apache.org
Subject: Re: IndexWriter is slow when reader is open

Thanks Michael for the detailed explanation.It's much more clearer now.

By "transactional capabilities" do you mean that if in the middle of a
commit something happens, it is guaranteed that either all the data
added
from the last commit is in index or all the data is discarded?

We have a steady stream of documents for indexing coming in
(unfortunately
only one at a time, but at a rate of up to 50 per second) and I hoped I
could guarantee that when the add method returns, the document is
secured on
disk. We keep a status for each document in our DB and want to discard
the
original data.

We'll just have to hang on to the original data until each commit has
finished and in case of a crash or error reindex the original data.

Eran.

On Tue, Nov 17, 2009 at 5:59 PM, Michael Garski
<mgarski@myspace-inc.com>wrote:

> Eran,
>
> Make no mistake, the poor performance you are experiencing is due to
> calling commit on every document addition and not due to internal
'coding by
> exception'.  There are transactional capabilities of Lucene that will
ensure
> that your documents are added and persisted to disk.  Check out the
> IndexWriter documentation for more information.
>
> The only 'connection' between the reader and the writer are the files
on
> disk.  The writer writes them once, they are not updated, and the
reader
> holds a reference to the file to ensure it is not deleted out from
> underneath it as it still needs to read from it to perform searches.
>
> During a commit, all of your changes are written to disk and any
necessary
> segment merges take place, which leaves the older segments that were
merged
> together as 'orphans' that are no longer referenced by the segments
file and
> are cleaned up during the final stage of the commit process after all
of the
> new segments have been written.  An attempt is made to then clean up
the
> older segments that are no longer necessary, which will fail as your
reader
> still has them open.  It fails gracefully in that the file names are
> persisted internally to attempt to delete again later, hopefully after
the
> reader has been reopened and a reference to the orphaned files is no
longer
> being held.
>
> I suggest you step through the commit process in a debugger or use a
> profiler to demonstrate this issue.
>
> Michael
>
>
>
> -----Original Message-----
> From: Eran Sevi [mailto:eransevi@gmail.com]
> Sent: Tue 11/17/2009 4:55 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: IndexWriter is slow when reader is open
>
> Michael,
> Thanks for the answer.
>
> I thought the reader was less connected to the writer. Basically what
your
> saying is that as long as at least one reader is open, exceptions are
> thrown
> when trying to commit changes (or more accurately, when trying to
merge
> segments) ?
> Can you point me to the place in the source code where that happens?
>
> What happens to the new documents that were added? are they still
saved in
> another segments?
>
> It's very important to us to make sure every document is persistent in
the
> index so working in batches could be a problem.
> But if there's a way to save each added document to disk without
merging
> the
> segment with older segments, this can solve our problem. And since the
> reader can't see the new segments anyway until it's reopened, I don't
see a
> problem continuing writing documents to new segments without
performing a
> merge. I'll try to change the merge policy/scheduler and see what
happens.
>
> Anyway, coding by exception is quite bad practice. Since we're
following
> the
> java versions I guess it'll take time to be able to change that.
>
> Eran.
>
> On Mon, Nov 16, 2009 at 8:56 PM, Michael Garski
<mgarski@myspace-inc.com
> >wrote:
>
> > Eran,
> >
> > The root cause of the issue is due to calling commit after every
document
> > addition while having a reader open.  Calls to commit should be
batched
> up -
> > we frequently use batches of 100 or 1000 between commits.
> >
> > This is by design within Lucene.  Adding documents will cause
segments to
> > merge and the writer will then delete the older segments that have
been
> > merged together to create a new one, however with an open reader the
> writer
> > will not be able to delete the older segment due to a file lock held
by
> the
> > reader.  On the call to delete the file an exception is thrown and
> swallowed
> > internally and the name of the file that the delete was attempted
upon is
> > added to a list of files that can be deleted on another call.
> >
> > I suggest you refrain from calling commit so often, as that is why
you
> are
> > experiencing performance issues.
> >
> > Michael
> >
> >
> > -----Original Message-----
> > From: Eran Sevi [mailto:eransevi@gmail.com]
> > Sent: Mon 11/16/2009 5:07 AM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: IndexWriter is slow when reader is open
> >
> > I've tried to use it with read-only mode and it looks like it's even
> worse
> > right now.
> >
> > I must admit that we're abusing the indexing a bit by commiting
after
> each
> > document addition, but still when there's no reader open, each
document
> is
> > indexed in about 30-50ms and when there's a read-only reader open
then
> each
> > document is indexed in about 150-500ms.
> > Why should an open reader affect the commit process so deeply?
> >
> > I wonder if no one encountered this phenomena before.
> >
> >
> > On Sat, Nov 14, 2009 at 8:27 PM, Matt Honeycutt
<mbhoneycutt@gmail.com
> > >wrote:
> >
> > > 2.4 does indeed support read-only mode. I don't know how much it
will
> > > help, but I would definitely try it.
> > >
> > > On 11/14/09, Eran Sevi <eransevi@gmail.com> wrote:
> > > > I'm still using version 2.4 so I think there's still no read
only
> mode.
> > > > Is there no other way to prevent this slow down in previous
versions?
> > > >
> > > > Eran.
> > > >
> > > > On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski
> > > > <mgarski@myspace-inc.com>wrote:
> > > >
> > > >> Eran,
> > > >>
> > > >> What version of Lucene are you using?  Are you opening the
> IndexReader
> > > >> in read-only mode?
> > > >>
> > > >> Michael
> > > >>
> > > >> -----Original Message-----
> > > >> From: Eran Sevi [mailto:eransevi@gmail.com]
> > > >> Sent: Thursday, November 12, 2009 9:06 AM
> > > >> To: lucene-net-user@incubator.apache.org
> > > >> Subject: IndexWriter is slow when reader is open
> > > >>
> > > >> Hi,
> > > >> I'm using Lucene.Net 2.4 and I just noticed that when I index
> > documents
> > > >> while there's at least one IndexReader open on that index (even
> > without
> > > >> doing anything), the indexing speed is slower by a factor of 3
to 5.
> > > >> When
> > > >> closing the reader, the indexing speed goes back to normal.
> > > >> I'm not doing any deletes, only adds.
> > > >>
> > > >>  My index is going to be updated regularly and there's going to
be a
> > > >> reader/searcher in use almost all the time so this might be a
big
> > > >> problem
> > > >> for me.
> > > >>
> > > >> Does anyone have a clue if this is normal behavior? why does it
> happen
> > > >> and
> > > >> how can I avoid such a big loss in performance?
> > > >>
> > > >>
> > > >> Thanks,
> > > >> Eran.
> > > >>
> > > >>
> > > >
> > >
> >
> >
> >
>
>
>

Thanks Michael for the detailed explanation.It's much more clearer now.

By "transactional capabilities" do you mean that if in the middle of a
commit something happens, it is guaranteed that either all the data added
from the last commit is in index or all the data is discarded?

We have a steady stream of documents for indexing coming in (unfortunately
only one at a time, but at a rate of up to 50 per second) and I hoped I
could guarantee that when the add method returns, the document is secured on
disk. We keep a status for each document in our DB and want to discard the
original data.

We'll just have to hang on to the original data until each commit has
finished and in case of a crash or error reindex the original data.

Eran.

On Tue, Nov 17, 2009 at 5:59 PM, Michael Garski <mgarski@myspace-inc.com>wrote:

> Eran,
>
> Make no mistake, the poor performance you are experiencing is due to
> calling commit on every document addition and not due to internal 'coding by
> exception'.  There are transactional capabilities of Lucene that will ensure
> that your documents are added and persisted to disk.  Check out the
> IndexWriter documentation for more information.
>
> The only 'connection' between the reader and the writer are the files on
> disk.  The writer writes them once, they are not updated, and the reader
> holds a reference to the file to ensure it is not deleted out from
> underneath it as it still needs to read from it to perform searches.
>
> During a commit, all of your changes are written to disk and any necessary
> segment merges take place, which leaves the older segments that were merged
> together as 'orphans' that are no longer referenced by the segments file and
> are cleaned up during the final stage of the commit process after all of the
> new segments have been written.  An attempt is made to then clean up the
> older segments that are no longer necessary, which will fail as your reader
> still has them open.  It fails gracefully in that the file names are
> persisted internally to attempt to delete again later, hopefully after the
> reader has been reopened and a reference to the orphaned files is no longer
> being held.
>
> I suggest you step through the commit process in a debugger or use a
> profiler to demonstrate this issue.
>
> Michael
>
>
>
> -----Original Message-----
> From: Eran Sevi [mailto:eransevi@gmail.com]
> Sent: Tue 11/17/2009 4:55 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: IndexWriter is slow when reader is open
>
> Michael,
> Thanks for the answer.
>
> I thought the reader was less connected to the writer. Basically what your
> saying is that as long as at least one reader is open, exceptions are
> thrown
> when trying to commit changes (or more accurately, when trying to merge
> segments) ?
> Can you point me to the place in the source code where that happens?
>
> What happens to the new documents that were added? are they still saved in
> another segments?
>
> It's very important to us to make sure every document is persistent in the
> index so working in batches could be a problem.
> But if there's a way to save each added document to disk without merging
> the
> segment with older segments, this can solve our problem. And since the
> reader can't see the new segments anyway until it's reopened, I don't see a
> problem continuing writing documents to new segments without performing a
> merge. I'll try to change the merge policy/scheduler and see what happens.
>
> Anyway, coding by exception is quite bad practice. Since we're following
> the
> java versions I guess it'll take time to be able to change that.
>
> Eran.
>
> On Mon, Nov 16, 2009 at 8:56 PM, Michael Garski <mgarski@myspace-inc.com
> >wrote:
>
> > Eran,
> >
> > The root cause of the issue is due to calling commit after every document
> > addition while having a reader open.  Calls to commit should be batched
> up -
> > we frequently use batches of 100 or 1000 between commits.
> >
> > This is by design within Lucene.  Adding documents will cause segments to
> > merge and the writer will then delete the older segments that have been
> > merged together to create a new one, however with an open reader the
> writer
> > will not be able to delete the older segment due to a file lock held by
> the
> > reader.  On the call to delete the file an exception is thrown and
> swallowed
> > internally and the name of the file that the delete was attempted upon is
> > added to a list of files that can be deleted on another call.
> >
> > I suggest you refrain from calling commit so often, as that is why you
> are
> > experiencing performance issues.
> >
> > Michael
> >
> >
> > -----Original Message-----
> > From: Eran Sevi [mailto:eransevi@gmail.com]
> > Sent: Mon 11/16/2009 5:07 AM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: IndexWriter is slow when reader is open
> >
> > I've tried to use it with read-only mode and it looks like it's even
> worse
> > right now.
> >
> > I must admit that we're abusing the indexing a bit by commiting after
> each
> > document addition, but still when there's no reader open, each document
> is
> > indexed in about 30-50ms and when there's a read-only reader open then
> each
> > document is indexed in about 150-500ms.
> > Why should an open reader affect the commit process so deeply?
> >
> > I wonder if no one encountered this phenomena before.
> >
> >
> > On Sat, Nov 14, 2009 at 8:27 PM, Matt Honeycutt <mbhoneycutt@gmail.com
> > >wrote:
> >
> > > 2.4 does indeed support read-only mode. I don't know how much it will
> > > help, but I would definitely try it.
> > >
> > > On 11/14/09, Eran Sevi <eransevi@gmail.com> wrote:
> > > > I'm still using version 2.4 so I think there's still no read only
> mode.
> > > > Is there no other way to prevent this slow down in previous versions?
> > > >
> > > > Eran.
> > > >
> > > > On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski
> > > > <mgarski@myspace-inc.com>wrote:
> > > >
> > > >> Eran,
> > > >>
> > > >> What version of Lucene are you using?  Are you opening the
> IndexReader
> > > >> in read-only mode?
> > > >>
> > > >> Michael
> > > >>
> > > >> -----Original Message-----
> > > >> From: Eran Sevi [mailto:eransevi@gmail.com]
> > > >> Sent: Thursday, November 12, 2009 9:06 AM
> > > >> To: lucene-net-user@incubator.apache.org
> > > >> Subject: IndexWriter is slow when reader is open
> > > >>
> > > >> Hi,
> > > >> I'm using Lucene.Net 2.4 and I just noticed that when I index
> > documents
> > > >> while there's at least one IndexReader open on that index (even
> > without
> > > >> doing anything), the indexing speed is slower by a factor of 3 to
5.
> > > >> When
> > > >> closing the reader, the indexing speed goes back to normal.
> > > >> I'm not doing any deletes, only adds.
> > > >>
> > > >>  My index is going to be updated regularly and there's going to be
a
> > > >> reader/searcher in use almost all the time so this might be a big
> > > >> problem
> > > >> for me.
> > > >>
> > > >> Does anyone have a clue if this is normal behavior? why does it
> happen
> > > >> and
> > > >> how can I avoid such a big loss in performance?
> > > >>
> > > >>
> > > >> Thanks,
> > > >> Eran.
> > > >>
> > > >>
> > > >
> > >
> >
> >
> >
>
>
>

Eran,

Make no mistake, the poor performance you are experiencing is due to calling commit on every
document addition and not due to internal 'coding by exception'.  There are transactional
capabilities of Lucene that will ensure that your documents are added and persisted to disk.
 Check out the IndexWriter documentation for more information.

The only 'connection' between the reader and the writer are the files on disk.  The writer
writes them once, they are not updated, and the reader holds a reference to the file to ensure
it is not deleted out from underneath it as it still needs to read from it to perform searches.

During a commit, all of your changes are written to disk and any necessary segment merges
take place, which leaves the older segments that were merged together as 'orphans' that are
no longer referenced by the segments file and are cleaned up during the final stage of the
commit process after all of the new segments have been written.  An attempt is made to then
clean up the older segments that are no longer necessary, which will fail as your reader still
has them open.  It fails gracefully in that the file names are persisted internally to attempt
to delete again later, hopefully after the reader has been reopened and a reference to the
orphaned files is no longer being held.

I suggest you step through the commit process in a debugger or use a profiler to demonstrate
this issue. 

Michael



-----Original Message-----
From: Eran Sevi [mailto:eransevi@gmail.com]
Sent: Tue 11/17/2009 4:55 AM
To: lucene-net-user@incubator.apache.org
Subject: Re: IndexWriter is slow when reader is open
 
Michael,
Thanks for the answer.

I thought the reader was less connected to the writer. Basically what your
saying is that as long as at least one reader is open, exceptions are thrown
when trying to commit changes (or more accurately, when trying to merge
segments) ?
Can you point me to the place in the source code where that happens?

What happens to the new documents that were added? are they still saved in
another segments?

It's very important to us to make sure every document is persistent in the
index so working in batches could be a problem.
But if there's a way to save each added document to disk without merging the
segment with older segments, this can solve our problem. And since the
reader can't see the new segments anyway until it's reopened, I don't see a
problem continuing writing documents to new segments without performing a
merge. I'll try to change the merge policy/scheduler and see what happens.

Anyway, coding by exception is quite bad practice. Since we're following the
java versions I guess it'll take time to be able to change that.

Eran.

On Mon, Nov 16, 2009 at 8:56 PM, Michael Garski <mgarski@myspace-inc.com>wrote:

> Eran,
>
> The root cause of the issue is due to calling commit after every document
> addition while having a reader open.  Calls to commit should be batched up -
> we frequently use batches of 100 or 1000 between commits.
>
> This is by design within Lucene.  Adding documents will cause segments to
> merge and the writer will then delete the older segments that have been
> merged together to create a new one, however with an open reader the writer
> will not be able to delete the older segment due to a file lock held by the
> reader.  On the call to delete the file an exception is thrown and swallowed
> internally and the name of the file that the delete was attempted upon is
> added to a list of files that can be deleted on another call.
>
> I suggest you refrain from calling commit so often, as that is why you are
> experiencing performance issues.
>
> Michael
>
>
> -----Original Message-----
> From: Eran Sevi [mailto:eransevi@gmail.com]
> Sent: Mon 11/16/2009 5:07 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: IndexWriter is slow when reader is open
>
> I've tried to use it with read-only mode and it looks like it's even worse
> right now.
>
> I must admit that we're abusing the indexing a bit by commiting after each
> document addition, but still when there's no reader open, each document is
> indexed in about 30-50ms and when there's a read-only reader open then each
> document is indexed in about 150-500ms.
> Why should an open reader affect the commit process so deeply?
>
> I wonder if no one encountered this phenomena before.
>
>
> On Sat, Nov 14, 2009 at 8:27 PM, Matt Honeycutt <mbhoneycutt@gmail.com
> >wrote:
>
> > 2.4 does indeed support read-only mode. I don't know how much it will
> > help, but I would definitely try it.
> >
> > On 11/14/09, Eran Sevi <eransevi@gmail.com> wrote:
> > > I'm still using version 2.4 so I think there's still no read only mode.
> > > Is there no other way to prevent this slow down in previous versions?
> > >
> > > Eran.
> > >
> > > On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski
> > > <mgarski@myspace-inc.com>wrote:
> > >
> > >> Eran,
> > >>
> > >> What version of Lucene are you using?  Are you opening the IndexReader
> > >> in read-only mode?
> > >>
> > >> Michael
> > >>
> > >> -----Original Message-----
> > >> From: Eran Sevi [mailto:eransevi@gmail.com]
> > >> Sent: Thursday, November 12, 2009 9:06 AM
> > >> To: lucene-net-user@incubator.apache.org
> > >> Subject: IndexWriter is slow when reader is open
> > >>
> > >> Hi,
> > >> I'm using Lucene.Net 2.4 and I just noticed that when I index
> documents
> > >> while there's at least one IndexReader open on that index (even
> without
> > >> doing anything), the indexing speed is slower by a factor of 3 to 5.
> > >> When
> > >> closing the reader, the indexing speed goes back to normal.
> > >> I'm not doing any deletes, only adds.
> > >>
> > >>  My index is going to be updated regularly and there's going to be a
> > >> reader/searcher in use almost all the time so this might be a big
> > >> problem
> > >> for me.
> > >>
> > >> Does anyone have a clue if this is normal behavior? why does it happen
> > >> and
> > >> how can I avoid such a big loss in performance?
> > >>
> > >>
> > >> Thanks,
> > >> Eran.
> > >>
> > >>
> > >
> >
>
>
>

 

Michael,
Thanks for the answer.

I thought the reader was less connected to the writer. Basically what your
saying is that as long as at least one reader is open, exceptions are thrown
when trying to commit changes (or more accurately, when trying to merge
segments) ?
Can you point me to the place in the source code where that happens?

What happens to the new documents that were added? are they still saved in
another segments?

It's very important to us to make sure every document is persistent in the
index so working in batches could be a problem.
But if there's a way to save each added document to disk without merging the
segment with older segments, this can solve our problem. And since the
reader can't see the new segments anyway until it's reopened, I don't see a
problem continuing writing documents to new segments without performing a
merge. I'll try to change the merge policy/scheduler and see what happens.

Anyway, coding by exception is quite bad practice. Since we're following the
java versions I guess it'll take time to be able to change that.

Eran.

On Mon, Nov 16, 2009 at 8:56 PM, Michael Garski <mgarski@myspace-inc.com>wrote:

> Eran,
>
> The root cause of the issue is due to calling commit after every document
> addition while having a reader open.  Calls to commit should be batched up -
> we frequently use batches of 100 or 1000 between commits.
>
> This is by design within Lucene.  Adding documents will cause segments to
> merge and the writer will then delete the older segments that have been
> merged together to create a new one, however with an open reader the writer
> will not be able to delete the older segment due to a file lock held by the
> reader.  On the call to delete the file an exception is thrown and swallowed
> internally and the name of the file that the delete was attempted upon is
> added to a list of files that can be deleted on another call.
>
> I suggest you refrain from calling commit so often, as that is why you are
> experiencing performance issues.
>
> Michael
>
>
> -----Original Message-----
> From: Eran Sevi [mailto:eransevi@gmail.com]
> Sent: Mon 11/16/2009 5:07 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: IndexWriter is slow when reader is open
>
> I've tried to use it with read-only mode and it looks like it's even worse
> right now.
>
> I must admit that we're abusing the indexing a bit by commiting after each
> document addition, but still when there's no reader open, each document is
> indexed in about 30-50ms and when there's a read-only reader open then each
> document is indexed in about 150-500ms.
> Why should an open reader affect the commit process so deeply?
>
> I wonder if no one encountered this phenomena before.
>
>
> On Sat, Nov 14, 2009 at 8:27 PM, Matt Honeycutt <mbhoneycutt@gmail.com
> >wrote:
>
> > 2.4 does indeed support read-only mode. I don't know how much it will
> > help, but I would definitely try it.
> >
> > On 11/14/09, Eran Sevi <eransevi@gmail.com> wrote:
> > > I'm still using version 2.4 so I think there's still no read only mode.
> > > Is there no other way to prevent this slow down in previous versions?
> > >
> > > Eran.
> > >
> > > On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski
> > > <mgarski@myspace-inc.com>wrote:
> > >
> > >> Eran,
> > >>
> > >> What version of Lucene are you using?  Are you opening the IndexReader
> > >> in read-only mode?
> > >>
> > >> Michael
> > >>
> > >> -----Original Message-----
> > >> From: Eran Sevi [mailto:eransevi@gmail.com]
> > >> Sent: Thursday, November 12, 2009 9:06 AM
> > >> To: lucene-net-user@incubator.apache.org
> > >> Subject: IndexWriter is slow when reader is open
> > >>
> > >> Hi,
> > >> I'm using Lucene.Net 2.4 and I just noticed that when I index
> documents
> > >> while there's at least one IndexReader open on that index (even
> without
> > >> doing anything), the indexing speed is slower by a factor of 3 to 5.
> > >> When
> > >> closing the reader, the indexing speed goes back to normal.
> > >> I'm not doing any deletes, only adds.
> > >>
> > >>  My index is going to be updated regularly and there's going to be a
> > >> reader/searcher in use almost all the time so this might be a big
> > >> problem
> > >> for me.
> > >>
> > >> Does anyone have a clue if this is normal behavior? why does it happen
> > >> and
> > >> how can I avoid such a big loss in performance?
> > >>
> > >>
> > >> Thanks,
> > >> Eran.
> > >>
> > >>
> > >
> >
>
>
>

Eran,

The root cause of the issue is due to calling commit after every document addition while having
a reader open.  Calls to commit should be batched up - we frequently use batches of 100 or
1000 between commits.

This is by design within Lucene.  Adding documents will cause segments to merge and the writer
will then delete the older segments that have been merged together to create a new one, however
with an open reader the writer will not be able to delete the older segment due to a file
lock held by the reader.  On the call to delete the file an exception is thrown and swallowed
internally and the name of the file that the delete was attempted upon is added to a list
of files that can be deleted on another call.

I suggest you refrain from calling commit so often, as that is why you are experiencing performance
issues.

Michael


-----Original Message-----
From: Eran Sevi [mailto:eransevi@gmail.com]
Sent: Mon 11/16/2009 5:07 AM
To: lucene-net-user@incubator.apache.org
Subject: Re: IndexWriter is slow when reader is open
 
I've tried to use it with read-only mode and it looks like it's even worse
right now.

I must admit that we're abusing the indexing a bit by commiting after each
document addition, but still when there's no reader open, each document is
indexed in about 30-50ms and when there's a read-only reader open then each
document is indexed in about 150-500ms.
Why should an open reader affect the commit process so deeply?

I wonder if no one encountered this phenomena before.


On Sat, Nov 14, 2009 at 8:27 PM, Matt Honeycutt <mbhoneycutt@gmail.com>wrote:

> 2.4 does indeed support read-only mode. I don't know how much it will
> help, but I would definitely try it.
>
> On 11/14/09, Eran Sevi <eransevi@gmail.com> wrote:
> > I'm still using version 2.4 so I think there's still no read only mode.
> > Is there no other way to prevent this slow down in previous versions?
> >
> > Eran.
> >
> > On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski
> > <mgarski@myspace-inc.com>wrote:
> >
> >> Eran,
> >>
> >> What version of Lucene are you using?  Are you opening the IndexReader
> >> in read-only mode?
> >>
> >> Michael
> >>
> >> -----Original Message-----
> >> From: Eran Sevi [mailto:eransevi@gmail.com]
> >> Sent: Thursday, November 12, 2009 9:06 AM
> >> To: lucene-net-user@incubator.apache.org
> >> Subject: IndexWriter is slow when reader is open
> >>
> >> Hi,
> >> I'm using Lucene.Net 2.4 and I just noticed that when I index documents
> >> while there's at least one IndexReader open on that index (even without
> >> doing anything), the indexing speed is slower by a factor of 3 to 5.
> >> When
> >> closing the reader, the indexing speed goes back to normal.
> >> I'm not doing any deletes, only adds.
> >>
> >>  My index is going to be updated regularly and there's going to be a
> >> reader/searcher in use almost all the time so this might be a big
> >> problem
> >> for me.
> >>
> >> Does anyone have a clue if this is normal behavior? why does it happen
> >> and
> >> how can I avoid such a big loss in performance?
> >>
> >>
> >> Thanks,
> >> Eran.
> >>
> >>
> >
>

 

I've noticed that the slow down only happens when the reader created is
"ReadOnlyMultiSegmentReader".
When the index is fully optimized (thus the reader created is
"ReadOnlySegmentReader"), the writer that is opened afterwards still
functions at full speed.
Since most of the time the index is far from being optimized, this is still
a major problem.

I can only guess that it's because of locking issues. I'll continue to
research it and update if I find something new.

On Mon, Nov 16, 2009 at 3:07 PM, Eran Sevi <eransevi@gmail.com> wrote:

> I've tried to use it with read-only mode and it looks like it's even worse
> right now.
>
> I must admit that we're abusing the indexing a bit by commiting after each
> document addition, but still when there's no reader open, each document is
> indexed in about 30-50ms and when there's a read-only reader open then each
> document is indexed in about 150-500ms.
> Why should an open reader affect the commit process so deeply?
>
> I wonder if no one encountered this phenomena before.
>
>
> On Sat, Nov 14, 2009 at 8:27 PM, Matt Honeycutt <mbhoneycutt@gmail.com>wrote:
>
>> 2.4 does indeed support read-only mode. I don't know how much it will
>> help, but I would definitely try it.
>>
>> On 11/14/09, Eran Sevi <eransevi@gmail.com> wrote:
>> > I'm still using version 2.4 so I think there's still no read only mode.
>> > Is there no other way to prevent this slow down in previous versions?
>> >
>> > Eran.
>> >
>> > On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski
>> > <mgarski@myspace-inc.com>wrote:
>> >
>> >> Eran,
>> >>
>> >> What version of Lucene are you using?  Are you opening the IndexReader
>> >> in read-only mode?
>> >>
>> >> Michael
>> >>
>> >> -----Original Message-----
>> >> From: Eran Sevi [mailto:eransevi@gmail.com]
>> >> Sent: Thursday, November 12, 2009 9:06 AM
>> >> To: lucene-net-user@incubator.apache.org
>> >> Subject: IndexWriter is slow when reader is open
>> >>
>> >> Hi,
>> >> I'm using Lucene.Net 2.4 and I just noticed that when I index documents
>> >> while there's at least one IndexReader open on that index (even without
>> >> doing anything), the indexing speed is slower by a factor of 3 to 5.
>> >> When
>> >> closing the reader, the indexing speed goes back to normal.
>> >> I'm not doing any deletes, only adds.
>> >>
>> >>  My index is going to be updated regularly and there's going to be a
>> >> reader/searcher in use almost all the time so this might be a big
>> >> problem
>> >> for me.
>> >>
>> >> Does anyone have a clue if this is normal behavior? why does it happen
>> >> and
>> >> how can I avoid such a big loss in performance?
>> >>
>> >>
>> >> Thanks,
>> >> Eran.
>> >>
>> >>
>> >
>>
>
>

I've tried to use it with read-only mode and it looks like it's even worse
right now.

I must admit that we're abusing the indexing a bit by commiting after each
document addition, but still when there's no reader open, each document is
indexed in about 30-50ms and when there's a read-only reader open then each
document is indexed in about 150-500ms.
Why should an open reader affect the commit process so deeply?

I wonder if no one encountered this phenomena before.


On Sat, Nov 14, 2009 at 8:27 PM, Matt Honeycutt <mbhoneycutt@gmail.com>wrote:

> 2.4 does indeed support read-only mode. I don't know how much it will
> help, but I would definitely try it.
>
> On 11/14/09, Eran Sevi <eransevi@gmail.com> wrote:
> > I'm still using version 2.4 so I think there's still no read only mode.
> > Is there no other way to prevent this slow down in previous versions?
> >
> > Eran.
> >
> > On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski
> > <mgarski@myspace-inc.com>wrote:
> >
> >> Eran,
> >>
> >> What version of Lucene are you using?  Are you opening the IndexReader
> >> in read-only mode?
> >>
> >> Michael
> >>
> >> -----Original Message-----
> >> From: Eran Sevi [mailto:eransevi@gmail.com]
> >> Sent: Thursday, November 12, 2009 9:06 AM
> >> To: lucene-net-user@incubator.apache.org
> >> Subject: IndexWriter is slow when reader is open
> >>
> >> Hi,
> >> I'm using Lucene.Net 2.4 and I just noticed that when I index documents
> >> while there's at least one IndexReader open on that index (even without
> >> doing anything), the indexing speed is slower by a factor of 3 to 5.
> >> When
> >> closing the reader, the indexing speed goes back to normal.
> >> I'm not doing any deletes, only adds.
> >>
> >>  My index is going to be updated regularly and there's going to be a
> >> reader/searcher in use almost all the time so this might be a big
> >> problem
> >> for me.
> >>
> >> Does anyone have a clue if this is normal behavior? why does it happen
> >> and
> >> how can I avoid such a big loss in performance?
> >>
> >>
> >> Thanks,
> >> Eran.
> >>
> >>
> >
>

2.4 does indeed support read-only mode. I don't know how much it will
help, but I would definitely try it.

On 11/14/09, Eran Sevi <eransevi@gmail.com> wrote:
> I'm still using version 2.4 so I think there's still no read only mode.
> Is there no other way to prevent this slow down in previous versions?
>
> Eran.
>
> On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski
> <mgarski@myspace-inc.com>wrote:
>
>> Eran,
>>
>> What version of Lucene are you using?  Are you opening the IndexReader
>> in read-only mode?
>>
>> Michael
>>
>> -----Original Message-----
>> From: Eran Sevi [mailto:eransevi@gmail.com]
>> Sent: Thursday, November 12, 2009 9:06 AM
>> To: lucene-net-user@incubator.apache.org
>> Subject: IndexWriter is slow when reader is open
>>
>> Hi,
>> I'm using Lucene.Net 2.4 and I just noticed that when I index documents
>> while there's at least one IndexReader open on that index (even without
>> doing anything), the indexing speed is slower by a factor of 3 to 5.
>> When
>> closing the reader, the indexing speed goes back to normal.
>> I'm not doing any deletes, only adds.
>>
>>  My index is going to be updated regularly and there's going to be a
>> reader/searcher in use almost all the time so this might be a big
>> problem
>> for me.
>>
>> Does anyone have a clue if this is normal behavior? why does it happen
>> and
>> how can I avoid such a big loss in performance?
>>
>>
>> Thanks,
>> Eran.
>>
>>
>

I'm still using version 2.4 so I think there's still no read only mode.
Is there no other way to prevent this slow down in previous versions?

Eran.

On Thu, Nov 12, 2009 at 8:16 PM, Michael Garski <mgarski@myspace-inc.com>wrote:

> Eran,
>
> What version of Lucene are you using?  Are you opening the IndexReader
> in read-only mode?
>
> Michael
>
> -----Original Message-----
> From: Eran Sevi [mailto:eransevi@gmail.com]
> Sent: Thursday, November 12, 2009 9:06 AM
> To: lucene-net-user@incubator.apache.org
> Subject: IndexWriter is slow when reader is open
>
> Hi,
> I'm using Lucene.Net 2.4 and I just noticed that when I index documents
> while there's at least one IndexReader open on that index (even without
> doing anything), the indexing speed is slower by a factor of 3 to 5.
> When
> closing the reader, the indexing speed goes back to normal.
> I'm not doing any deletes, only adds.
>
>  My index is going to be updated regularly and there's going to be a
> reader/searcher in use almost all the time so this might be a big
> problem
> for me.
>
> Does anyone have a clue if this is normal behavior? why does it happen
> and
> how can I avoid such a big loss in performance?
>
>
> Thanks,
> Eran.
>
>

It scales well with tens of millions of documents or more with enough
RAM, provided you have a mechanism for expiring the cache when a reader
is no longer in use and pool cached items for re-use so your process
doesn't incur massive GC sweeps when the cache is expired.

Michael

-----Original Message-----
From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com] 
Sent: Thursday, November 12, 2009 12:35 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: FieldLookup for field with multiple values

I shudder to think about what the higher-ups will say about the cost of
faster storage.  They're very stingy on things that make our lives
easier.
:)

I like the idea of caching the term vectors somehow, though I wonder how
that would scale with millions of documents.  I will add that to the
list of
things to prototype for our long-term solution.

Thanks for the feedback,

---Matt

On Thu, Nov 12, 2009 at 12:14 PM, Michael Garski
<mgarski@myspace-inc.com>wrote:

> Matt,
>
> Metadata can be collected during the course of a search in a
> HitCollector (note that 2.9 will deprecate the current HitCollector in
> favor of the more flexible Collector).  One approach to collecting
> facets is to index the field with the desired facet metadata with
> TermVectors enabled.  During collection you can retrieve the
> TermFreqVector for a document with Reader.GetTermFreqVector for each
> document and collect/aggregate metadata however you like.
>
> Making the call to get the TermFreqVector for each hit will be a
> performance hit due to IO and deserialization (even though it's very
> fast, 50K times in one search is a lot).  At that point you can add
> faster IO (Fusion IO rules this space) or during an index warmup
create
> an in-memory structure with lookups to the vectors keyed off the
> document id.
>
> Hope that helps.
>
> Michael
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: Thursday, November 12, 2009 7:13 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> I can elaborate a little on what our *planned* approach for utilizing
> SQL
> Server is.  I don't know if this will work, but I've done similar
things
> with SQL CLR and haven't had it explode (yet), so I'm hopeful.
Anyway:
>
> Our system needs two types of output, basically: the full-text reports
> (I'll
> call them documents to reduce ambiguity), and then statistical reports
> built
> on those documents.  The documents can easily be retrieved from
Lucene,
> so
> the challenge is building the reports.  For that, my plan is to submit
> the
> same query string to SQL Server that I sent to Lucene.  Internally,
SQL
> Server would then pass the query back to Lucene and retrieve a list of
> document IDs that matched.  The communication may be over WCF or
> something
> similar, and will be compressed during transit to reduce IO overhead.
> Once
> SQL Server has the IDs, they will be loaded into a temporary table
(with
> indexes) or a table variable, which will then be used to filter the
> metadata
> that the statistical reports are built from.
>
> I have no idea how such a system would perform.  I do hope to do some
> feasibility tests sometime Real Soon (like in the next few weeks), and
> I'll
> post my results if I manage to get it working.
>
> If anyone has any other suggestions, please do share.
>
> On Thu, Nov 12, 2009 at 8:43 AM, Moray McConnachie <
> mmcconna@oxford-analytica.com> wrote:
>
> > >While we're discussing this, anyone have any advice or suggestions
> for
> > a better solution?  We've considered a few things for our long-term
> > solution.
> >
> > I'd be very interested to hear thoughts on intersecting SQL and
Lucene
> > too, as in our case we have very large lists of organisations which
> have
> > different permissions (stored in SQL) for different documents stored
> in
> > Lucene. Showing in the search results only those documents to which
> the
> > organisation has permission is quite expensive for queries with lots
> of
> > results. Storing it in the documents is not manageable because they
> need
> > to be updated frequently across multiple documents. Currently we
> > precompute a list for each organisation, cache that in memory, and
> > recache it every time that organisation is updated. However this too
> is
> > costly.
> >
> > Storing the Lucene document nos in SQL during indexing, and then
> passing
> > the list of Lucene document nos matching a search to SQL for
filtering
> > seems the right way to go. But Matt is right, the problem is with
> > scaling this to searches returning many thousands of documents.
> >
> > Yours,
> > M.
> > -------------------------------------
> > Moray McConnachie
> > Director of IT    +44 1865 261 600
> > Oxford Analytica  http://www.oxan.com
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: 12 November 2009 13:53
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: FieldLookup for field with multiple values
> >
> > Yeah, it is sort of like your standard faceting scenario, except
there
> > are about 20,000 facets (organizations), and there's complex
> > relationships among the facets.
> >
> > The reports we're dealing with only occasionally break the funding
up
> by
> > organization, so we decided (for now) to just store a single funding
> > value, then break it up after-the-fact by dividing it by the number
of
> > organizations.  So no, the funding is only stored once.
> >
> > While we're discussing this, anyone have any advice or suggestions
for
> a
> > better solution?  We've considered a few things for our long-term
> > solution.
> > One is to put this metadata in a SQL Server instance, and use SQL
CLR
> to
> > build a temporary table based on document IDs from a Lucene index
> > (hosted over WCF or something similar), then do the reporting within
> SQL
> > Server.  We plan to compress the list of IDs going back from Lucene
to
> > SQL Server to cut down on IO overhead, but we're still concerned
that
> > approach won't scale as we go from hundreds of thousands to millions
> of
> > reports.
> >
> > Another option we've discussed is to precompute data cubes and use
> these
> > to calculate reporting information.  The concern here is the high
> > dimensionality of the data (we have about 20,000 distinct
> organizations
> > now, but fully expect that to increase by an order of magnitude) as
> well
> > as the accuracy of the generated reports, since there's (probably)
not
> a
> > good way to divide the cube based on arbitrary Lucene queries.
> >
> > On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski
> > <mgarski@myspace-inc.com>wrote:
> >
> > > Sounds like a full-text search with the results simply being
facets
> on
> >
> > > the organizations sorted by the funding amount?
> > >
> > > You mentioned adding the org ID once for each document.  Do you do
> the
> >
> > > same for the funding, with the funding for each corresponding
> > organization?
> > >
> > > Michael
> > >
> > >
> > > -----Original Message-----
> > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > Sent: Wed 11/11/2009 10:17 PM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Re: FieldLookup for field with multiple values
> > >
> > > Well, let me prefix what I'm about to describe by saying that I
know
> > > that I'm doing something with Lucene that it wasn't meant to do.
> This
> >
> > > is for a "proof of concept" system that I'm helping put together
on
> a
> > > tight schedule with very limited resources, and we're trying to
get
> to
> >
> > > a mostly-working state as quickly as possible.
> > >
> > > That said, we are basically storing reports in Lucene.  The
reports
> > > are fairly standard documents for the most part: they have a
title,
> > > body, abstract, etc, all of which we index and search with Lucene.
> > > However, they also have a few fields that aren't standard,
including
> a
> >
> > > list of involved organizations as well as a dollar amount for each
> > > report.  The organizations are stored as IDs, and we add the org
ID
> > > field multiple times, once for each organization involved in the
> > > report.  The funding is also stored as a non-indexed field on the
> > > Lucene document.
> > >
> > > What I'm trying to do is build a quick-and-dirty org-by-dollar
> report
> > > off of the reports that match the user's query.  So, a query for
> > > "aerospace" might match 50,000 documents, and I want to show the
> user
> > > the top 5 organizations in terms of dollars.  Again, I know
> reporting
> > > like this isn't what Lucene was meant for, and we do have some
ideas
> > > on how to handle it long-term, but for now, I'm trying to get it
> > > working as well as I can using Lucene alone, and Lucene does do a
> > > great job of finding the relevant set of documents to build a
report
> > > from.
> > >
> > > On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski
> > > <mgarski@myspace-inc.com
> > > >wrote:
> > >
> > > > Matt,
> > > >
> > > > StringIndex is for use when a field has only one value in it for
> the
> >
> > > > purposes of sorting results, not for tokenized fields with
> multiple
> > > > values.  TermVectors might be a better approach, but for 50K
docs,
> > > > you'll encounter an IO hit on reading them.
> > > >
> > > > I'm curious why you are looking to grab all of the terms for a
> > > > ScoreDoc...  can you shed some light on that?
> > > >
> > > > Michael
> > > >
> > > > -----Original Message-----
> > > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > > Sent: Wednesday, November 11, 2009 4:57 PM
> > > > To: lucene-net-user@incubator.apache.org
> > > > Subject: FieldLookup for field with multiple values
> > > >
> > > > It seems that the StringIndex returned by
> > > > FieldCache.Fields.Default.GetStringIndex() only indexes one
value
> > > > for a document even when the document has multiple values for
the
> > > > field.  Is there a performant want to get all the values for a
> > > > particular field in a ScoreDoc?  I'm having to do this across
the
> > > > entire result set of ScoreDocs (up to 50,000), and retrieving
the
> > > > values through LuceneDocument.GetFields is not going to cut it.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>

I shudder to think about what the higher-ups will say about the cost of
faster storage.  They're very stingy on things that make our lives easier.
:)

I like the idea of caching the term vectors somehow, though I wonder how
that would scale with millions of documents.  I will add that to the list of
things to prototype for our long-term solution.

Thanks for the feedback,

---Matt

On Thu, Nov 12, 2009 at 12:14 PM, Michael Garski <mgarski@myspace-inc.com>wrote:

> Matt,
>
> Metadata can be collected during the course of a search in a
> HitCollector (note that 2.9 will deprecate the current HitCollector in
> favor of the more flexible Collector).  One approach to collecting
> facets is to index the field with the desired facet metadata with
> TermVectors enabled.  During collection you can retrieve the
> TermFreqVector for a document with Reader.GetTermFreqVector for each
> document and collect/aggregate metadata however you like.
>
> Making the call to get the TermFreqVector for each hit will be a
> performance hit due to IO and deserialization (even though it's very
> fast, 50K times in one search is a lot).  At that point you can add
> faster IO (Fusion IO rules this space) or during an index warmup create
> an in-memory structure with lookups to the vectors keyed off the
> document id.
>
> Hope that helps.
>
> Michael
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: Thursday, November 12, 2009 7:13 AM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> I can elaborate a little on what our *planned* approach for utilizing
> SQL
> Server is.  I don't know if this will work, but I've done similar things
> with SQL CLR and haven't had it explode (yet), so I'm hopeful.  Anyway:
>
> Our system needs two types of output, basically: the full-text reports
> (I'll
> call them documents to reduce ambiguity), and then statistical reports
> built
> on those documents.  The documents can easily be retrieved from Lucene,
> so
> the challenge is building the reports.  For that, my plan is to submit
> the
> same query string to SQL Server that I sent to Lucene.  Internally, SQL
> Server would then pass the query back to Lucene and retrieve a list of
> document IDs that matched.  The communication may be over WCF or
> something
> similar, and will be compressed during transit to reduce IO overhead.
> Once
> SQL Server has the IDs, they will be loaded into a temporary table (with
> indexes) or a table variable, which will then be used to filter the
> metadata
> that the statistical reports are built from.
>
> I have no idea how such a system would perform.  I do hope to do some
> feasibility tests sometime Real Soon (like in the next few weeks), and
> I'll
> post my results if I manage to get it working.
>
> If anyone has any other suggestions, please do share.
>
> On Thu, Nov 12, 2009 at 8:43 AM, Moray McConnachie <
> mmcconna@oxford-analytica.com> wrote:
>
> > >While we're discussing this, anyone have any advice or suggestions
> for
> > a better solution?  We've considered a few things for our long-term
> > solution.
> >
> > I'd be very interested to hear thoughts on intersecting SQL and Lucene
> > too, as in our case we have very large lists of organisations which
> have
> > different permissions (stored in SQL) for different documents stored
> in
> > Lucene. Showing in the search results only those documents to which
> the
> > organisation has permission is quite expensive for queries with lots
> of
> > results. Storing it in the documents is not manageable because they
> need
> > to be updated frequently across multiple documents. Currently we
> > precompute a list for each organisation, cache that in memory, and
> > recache it every time that organisation is updated. However this too
> is
> > costly.
> >
> > Storing the Lucene document nos in SQL during indexing, and then
> passing
> > the list of Lucene document nos matching a search to SQL for filtering
> > seems the right way to go. But Matt is right, the problem is with
> > scaling this to searches returning many thousands of documents.
> >
> > Yours,
> > M.
> > -------------------------------------
> > Moray McConnachie
> > Director of IT    +44 1865 261 600
> > Oxford Analytica  http://www.oxan.com
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: 12 November 2009 13:53
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: FieldLookup for field with multiple values
> >
> > Yeah, it is sort of like your standard faceting scenario, except there
> > are about 20,000 facets (organizations), and there's complex
> > relationships among the facets.
> >
> > The reports we're dealing with only occasionally break the funding up
> by
> > organization, so we decided (for now) to just store a single funding
> > value, then break it up after-the-fact by dividing it by the number of
> > organizations.  So no, the funding is only stored once.
> >
> > While we're discussing this, anyone have any advice or suggestions for
> a
> > better solution?  We've considered a few things for our long-term
> > solution.
> > One is to put this metadata in a SQL Server instance, and use SQL CLR
> to
> > build a temporary table based on document IDs from a Lucene index
> > (hosted over WCF or something similar), then do the reporting within
> SQL
> > Server.  We plan to compress the list of IDs going back from Lucene to
> > SQL Server to cut down on IO overhead, but we're still concerned that
> > approach won't scale as we go from hundreds of thousands to millions
> of
> > reports.
> >
> > Another option we've discussed is to precompute data cubes and use
> these
> > to calculate reporting information.  The concern here is the high
> > dimensionality of the data (we have about 20,000 distinct
> organizations
> > now, but fully expect that to increase by an order of magnitude) as
> well
> > as the accuracy of the generated reports, since there's (probably) not
> a
> > good way to divide the cube based on arbitrary Lucene queries.
> >
> > On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski
> > <mgarski@myspace-inc.com>wrote:
> >
> > > Sounds like a full-text search with the results simply being facets
> on
> >
> > > the organizations sorted by the funding amount?
> > >
> > > You mentioned adding the org ID once for each document.  Do you do
> the
> >
> > > same for the funding, with the funding for each corresponding
> > organization?
> > >
> > > Michael
> > >
> > >
> > > -----Original Message-----
> > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > Sent: Wed 11/11/2009 10:17 PM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: Re: FieldLookup for field with multiple values
> > >
> > > Well, let me prefix what I'm about to describe by saying that I know
> > > that I'm doing something with Lucene that it wasn't meant to do.
> This
> >
> > > is for a "proof of concept" system that I'm helping put together on
> a
> > > tight schedule with very limited resources, and we're trying to get
> to
> >
> > > a mostly-working state as quickly as possible.
> > >
> > > That said, we are basically storing reports in Lucene.  The reports
> > > are fairly standard documents for the most part: they have a title,
> > > body, abstract, etc, all of which we index and search with Lucene.
> > > However, they also have a few fields that aren't standard, including
> a
> >
> > > list of involved organizations as well as a dollar amount for each
> > > report.  The organizations are stored as IDs, and we add the org ID
> > > field multiple times, once for each organization involved in the
> > > report.  The funding is also stored as a non-indexed field on the
> > > Lucene document.
> > >
> > > What I'm trying to do is build a quick-and-dirty org-by-dollar
> report
> > > off of the reports that match the user's query.  So, a query for
> > > "aerospace" might match 50,000 documents, and I want to show the
> user
> > > the top 5 organizations in terms of dollars.  Again, I know
> reporting
> > > like this isn't what Lucene was meant for, and we do have some ideas
> > > on how to handle it long-term, but for now, I'm trying to get it
> > > working as well as I can using Lucene alone, and Lucene does do a
> > > great job of finding the relevant set of documents to build a report
> > > from.
> > >
> > > On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski
> > > <mgarski@myspace-inc.com
> > > >wrote:
> > >
> > > > Matt,
> > > >
> > > > StringIndex is for use when a field has only one value in it for
> the
> >
> > > > purposes of sorting results, not for tokenized fields with
> multiple
> > > > values.  TermVectors might be a better approach, but for 50K docs,
> > > > you'll encounter an IO hit on reading them.
> > > >
> > > > I'm curious why you are looking to grab all of the terms for a
> > > > ScoreDoc...  can you shed some light on that?
> > > >
> > > > Michael
> > > >
> > > > -----Original Message-----
> > > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > > Sent: Wednesday, November 11, 2009 4:57 PM
> > > > To: lucene-net-user@incubator.apache.org
> > > > Subject: FieldLookup for field with multiple values
> > > >
> > > > It seems that the StringIndex returned by
> > > > FieldCache.Fields.Default.GetStringIndex() only indexes one value
> > > > for a document even when the document has multiple values for the
> > > > field.  Is there a performant want to get all the values for a
> > > > particular field in a ScoreDoc?  I'm having to do this across the
> > > > entire result set of ScoreDocs (up to 50,000), and retrieving the
> > > > values through LuceneDocument.GetFields is not going to cut it.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>

Eran,

What version of Lucene are you using?  Are you opening the IndexReader
in read-only mode?

Michael

-----Original Message-----
From: Eran Sevi [mailto:eransevi@gmail.com] 
Sent: Thursday, November 12, 2009 9:06 AM
To: lucene-net-user@incubator.apache.org
Subject: IndexWriter is slow when reader is open

Hi,
I'm using Lucene.Net 2.4 and I just noticed that when I index documents
while there's at least one IndexReader open on that index (even without
doing anything), the indexing speed is slower by a factor of 3 to 5.
When
closing the reader, the indexing speed goes back to normal.
I'm not doing any deletes, only adds.

 My index is going to be updated regularly and there's going to be a
reader/searcher in use almost all the time so this might be a big
problem
for me.

Does anyone have a clue if this is normal behavior? why does it happen
and
how can I avoid such a big loss in performance?


Thanks,
Eran.

Matt,

Metadata can be collected during the course of a search in a
HitCollector (note that 2.9 will deprecate the current HitCollector in
favor of the more flexible Collector).  One approach to collecting
facets is to index the field with the desired facet metadata with
TermVectors enabled.  During collection you can retrieve the
TermFreqVector for a document with Reader.GetTermFreqVector for each
document and collect/aggregate metadata however you like.

Making the call to get the TermFreqVector for each hit will be a
performance hit due to IO and deserialization (even though it's very
fast, 50K times in one search is a lot).  At that point you can add
faster IO (Fusion IO rules this space) or during an index warmup create
an in-memory structure with lookups to the vectors keyed off the
document id.

Hope that helps.

Michael

-----Original Message-----
From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com] 
Sent: Thursday, November 12, 2009 7:13 AM
To: lucene-net-user@incubator.apache.org
Subject: Re: FieldLookup for field with multiple values

I can elaborate a little on what our *planned* approach for utilizing
SQL
Server is.  I don't know if this will work, but I've done similar things
with SQL CLR and haven't had it explode (yet), so I'm hopeful.  Anyway:

Our system needs two types of output, basically: the full-text reports
(I'll
call them documents to reduce ambiguity), and then statistical reports
built
on those documents.  The documents can easily be retrieved from Lucene,
so
the challenge is building the reports.  For that, my plan is to submit
the
same query string to SQL Server that I sent to Lucene.  Internally, SQL
Server would then pass the query back to Lucene and retrieve a list of
document IDs that matched.  The communication may be over WCF or
something
similar, and will be compressed during transit to reduce IO overhead.
Once
SQL Server has the IDs, they will be loaded into a temporary table (with
indexes) or a table variable, which will then be used to filter the
metadata
that the statistical reports are built from.

I have no idea how such a system would perform.  I do hope to do some
feasibility tests sometime Real Soon (like in the next few weeks), and
I'll
post my results if I manage to get it working.

If anyone has any other suggestions, please do share.

On Thu, Nov 12, 2009 at 8:43 AM, Moray McConnachie <
mmcconna@oxford-analytica.com> wrote:

> >While we're discussing this, anyone have any advice or suggestions
for
> a better solution?  We've considered a few things for our long-term
> solution.
>
> I'd be very interested to hear thoughts on intersecting SQL and Lucene
> too, as in our case we have very large lists of organisations which
have
> different permissions (stored in SQL) for different documents stored
in
> Lucene. Showing in the search results only those documents to which
the
> organisation has permission is quite expensive for queries with lots
of
> results. Storing it in the documents is not manageable because they
need
> to be updated frequently across multiple documents. Currently we
> precompute a list for each organisation, cache that in memory, and
> recache it every time that organisation is updated. However this too
is
> costly.
>
> Storing the Lucene document nos in SQL during indexing, and then
passing
> the list of Lucene document nos matching a search to SQL for filtering
> seems the right way to go. But Matt is right, the problem is with
> scaling this to searches returning many thousands of documents.
>
> Yours,
> M.
> -------------------------------------
> Moray McConnachie
> Director of IT    +44 1865 261 600
> Oxford Analytica  http://www.oxan.com
>
> -----Original Message-----
> From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> Sent: 12 November 2009 13:53
> To: lucene-net-user@incubator.apache.org
> Subject: Re: FieldLookup for field with multiple values
>
> Yeah, it is sort of like your standard faceting scenario, except there
> are about 20,000 facets (organizations), and there's complex
> relationships among the facets.
>
> The reports we're dealing with only occasionally break the funding up
by
> organization, so we decided (for now) to just store a single funding
> value, then break it up after-the-fact by dividing it by the number of
> organizations.  So no, the funding is only stored once.
>
> While we're discussing this, anyone have any advice or suggestions for
a
> better solution?  We've considered a few things for our long-term
> solution.
> One is to put this metadata in a SQL Server instance, and use SQL CLR
to
> build a temporary table based on document IDs from a Lucene index
> (hosted over WCF or something similar), then do the reporting within
SQL
> Server.  We plan to compress the list of IDs going back from Lucene to
> SQL Server to cut down on IO overhead, but we're still concerned that
> approach won't scale as we go from hundreds of thousands to millions
of
> reports.
>
> Another option we've discussed is to precompute data cubes and use
these
> to calculate reporting information.  The concern here is the high
> dimensionality of the data (we have about 20,000 distinct
organizations
> now, but fully expect that to increase by an order of magnitude) as
well
> as the accuracy of the generated reports, since there's (probably) not
a
> good way to divide the cube based on arbitrary Lucene queries.
>
> On Thu, Nov 12, 2009 at 1:03 AM, Michael Garski
> <mgarski@myspace-inc.com>wrote:
>
> > Sounds like a full-text search with the results simply being facets
on
>
> > the organizations sorted by the funding amount?
> >
> > You mentioned adding the org ID once for each document.  Do you do
the
>
> > same for the funding, with the funding for each corresponding
> organization?
> >
> > Michael
> >
> >
> > -----Original Message-----
> > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > Sent: Wed 11/11/2009 10:17 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: Re: FieldLookup for field with multiple values
> >
> > Well, let me prefix what I'm about to describe by saying that I know
> > that I'm doing something with Lucene that it wasn't meant to do.
This
>
> > is for a "proof of concept" system that I'm helping put together on
a
> > tight schedule with very limited resources, and we're trying to get
to
>
> > a mostly-working state as quickly as possible.
> >
> > That said, we are basically storing reports in Lucene.  The reports
> > are fairly standard documents for the most part: they have a title,
> > body, abstract, etc, all of which we index and search with Lucene.
> > However, they also have a few fields that aren't standard, including
a
>
> > list of involved organizations as well as a dollar amount for each
> > report.  The organizations are stored as IDs, and we add the org ID
> > field multiple times, once for each organization involved in the
> > report.  The funding is also stored as a non-indexed field on the
> > Lucene document.
> >
> > What I'm trying to do is build a quick-and-dirty org-by-dollar
report
> > off of the reports that match the user's query.  So, a query for
> > "aerospace" might match 50,000 documents, and I want to show the
user
> > the top 5 organizations in terms of dollars.  Again, I know
reporting
> > like this isn't what Lucene was meant for, and we do have some ideas
> > on how to handle it long-term, but for now, I'm trying to get it
> > working as well as I can using Lucene alone, and Lucene does do a
> > great job of finding the relevant set of documents to build a report
> > from.
> >
> > On Wed, Nov 11, 2009 at 8:56 PM, Michael Garski
> > <mgarski@myspace-inc.com
> > >wrote:
> >
> > > Matt,
> > >
> > > StringIndex is for use when a field has only one value in it for
the
>
> > > purposes of sorting results, not for tokenized fields with
multiple
> > > values.  TermVectors might be a better approach, but for 50K docs,
> > > you'll encounter an IO hit on reading them.
> > >
> > > I'm curious why you are looking to grab all of the terms for a
> > > ScoreDoc...  can you shed some light on that?
> > >
> > > Michael
> > >
> > > -----Original Message-----
> > > From: Matt Honeycutt [mailto:mbhoneycutt@gmail.com]
> > > Sent: Wednesday, November 11, 2009 4:57 PM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: FieldLookup for field with multiple values
> > >
> > > It seems that the StringIndex returned by
> > > FieldCache.Fields.Default.GetStringIndex() only indexes one value
> > > for a document even when the document has multiple values for the
> > > field.  Is there a performant want to get all the values for a
> > > particular field in a ScoreDoc?  I'm having to do this across the
> > > entire result set of ScoreDocs (up to 50,000), and retrieving the
> > > values through LuceneDocument.GetFields is not going to cut it.
> > >
> > >
> >
> >
> >
>
>