lucenenet-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koga, Diego" <dik...@gmail.com>
Subject Re: Facet Performance
Date Thu, 29 Dec 2016 16:07:55 GMT
Hi Itamar,

Is there any difference between 3.0.3 and 4.8 on this matter?



Att.,
------------------
Koga, Diego


On Thu, Dec 29, 2016 at 11:01 AM, Itamar Syn-Hershko <itamar@code972.com> wrote:
> This makes sense, yes. Faceted search is by definition a high-memory
> consumer.
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> Freelance Developer & Consultant
> Lucene.NET committer and PMC member
>
> On Thu, Dec 29, 2016 at 5:53 PM, Koga, Diego <dikoga@gmail.com> wrote:
>
>> Hi all,
>>
>> I figured something out. Once I keep the reader and searcher as
>> singleton, I also need to keep the facet engine.
>>
>> However, the SimpleFacetedSearch was consuming huge amount of memory,
>> the SparseFacetedSearcher is way more memory optimized. I am creating
>> one for each facet that I have and while searching, I pick the right
>> facet searcher to use.
>>
>> This is how my code looks like:
>>
>>
>> private void CreateReader()
>> {
>> if (_indexReader == null)
>> {
>> var dir = FSDirectory.Open(IndexDirectories.WorkDirectory);
>>
>> if (!IndexReader.IndexExists(dir))
>> {
>> dir.Dispose();
>> throw new FileNotFoundException(
>> "Index not found. Before initialize the reader, make sure to create
>> the index.");
>> }
>> _indexReader = IndexReader.Open(dir, true);
>> _indexSeacher = new IndexSearcher(_indexReader);
>> _facetsSearchers = CreateFacetsSearchers(dir);
>> }
>> }
>>
>> private Dictionary<string, SparseFacetedSearcher>
>> CreateFacetsSearchers(FSDirectory directory)
>> {
>> var facetReaders = new Dictionary<string, SparseFacetedSearcher>();
>>
>> foreach (var facet in AvailableFacets.Split(new[] { ',' },
>> StringSplitOptions.RemoveEmptyEntries))
>> {
>> var indexReader = IndexReader.Open(directory, true);
>> var facetSeacher = new SparseFacetedSearcher(indexReader, facet);
>> facetReaders.Add(facet, facetSeacher);
>> }
>>
>> return facetReaders;
>> }
>>
>>
>>
>> Att.,
>> ------------------
>> Koga, Diego
>>
>>
>> On Mon, Dec 26, 2016 at 6:05 PM, Koga, Diego <dikoga@gmail.com> wrote:
>> > Guys, Happy Holidays!!!
>> >
>> > I am tunning an existing Lucene.Net index where I am currently working.
>> >
>> > The perfomance was greatly increase when we talk about the search itself.
>> >
>> > But when it goes to facet concern, I'm getting crazy!
>> >
>> > This is the scenario:
>> >
>> > - 1.8M products on a SQL Server.
>> > - Basically we concatenate all analyzed fields into one. (made the search
>> > work way better)
>> > - Create a specific price field to set a price range in order to group
>> later
>> > in a facet. (here we have ranges configured in the system that we save
>> only
>> > the id. The ideia is to minimize the number of possible facets)
>> > - 16k categories (maybe I can do a tree here and minimize it too, need
>> > opinions)
>> > - 2k manufacturers
>> > - Facets needed: On Sale (boolean), Review (0 - 5), Price (let's say 10
>> > ranges), Free Shipping (boolean), Manufacturer (string) and Category
>> > (string).
>> >
>> > Question 1: What do you guys would improve when writing the index?
>> >
>> > The lucene is embedded in ASP.NET app where the clients make a REST http
>> > request. It's only one request that returns the result of the search and
>> > also the facets. To make it faster, I am processing the search and each
>> > facet in parallel. Which also make the search waits for the facets until
>> > it's done (I figured out that is faster process in parallel than pass an
>> > array).
>> >
>> > This is the facet search:
>> >
>> >             using (var simpleFacetedSearch = new
>> > SimpleFacetedSearch(_indexReader, facetName))
>> >             {
>> >                 var hits = simpleFacetedSearch.Search(query, 1);
>> >                 return hits;
>> >             }
>> >
>> > Question 2: What do you guys recommend to make this search faster and
>> also
>> > does not freeze the server because high memory usage?
>> >
>> >
>> > Thanks....
>> >
>> >
>> > Att.,
>> > ------------------
>> > Koga, Diego
>> >
>> >>
>> >
>>

Mime
View raw message