lucenenet-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Itamar Syn-Hershko <ita...@code972.com>
Subject Re: Facet Performance
Date Thu, 29 Dec 2016 16:01:43 GMT
This makes sense, yes. Faceted search is by definition a high-memory
consumer.

--

Itamar Syn-Hershko
http://code972.com | @synhershko <https://twitter.com/synhershko>
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Dec 29, 2016 at 5:53 PM, Koga, Diego <dikoga@gmail.com> wrote:

> Hi all,
>
> I figured something out. Once I keep the reader and searcher as
> singleton, I also need to keep the facet engine.
>
> However, the SimpleFacetedSearch was consuming huge amount of memory,
> the SparseFacetedSearcher is way more memory optimized. I am creating
> one for each facet that I have and while searching, I pick the right
> facet searcher to use.
>
> This is how my code looks like:
>
>
> private void CreateReader()
> {
> if (_indexReader == null)
> {
> var dir = FSDirectory.Open(IndexDirectories.WorkDirectory);
>
> if (!IndexReader.IndexExists(dir))
> {
> dir.Dispose();
> throw new FileNotFoundException(
> "Index not found. Before initialize the reader, make sure to create
> the index.");
> }
> _indexReader = IndexReader.Open(dir, true);
> _indexSeacher = new IndexSearcher(_indexReader);
> _facetsSearchers = CreateFacetsSearchers(dir);
> }
> }
>
> private Dictionary<string, SparseFacetedSearcher>
> CreateFacetsSearchers(FSDirectory directory)
> {
> var facetReaders = new Dictionary<string, SparseFacetedSearcher>();
>
> foreach (var facet in AvailableFacets.Split(new[] { ',' },
> StringSplitOptions.RemoveEmptyEntries))
> {
> var indexReader = IndexReader.Open(directory, true);
> var facetSeacher = new SparseFacetedSearcher(indexReader, facet);
> facetReaders.Add(facet, facetSeacher);
> }
>
> return facetReaders;
> }
>
>
>
> Att.,
> ------------------
> Koga, Diego
>
>
> On Mon, Dec 26, 2016 at 6:05 PM, Koga, Diego <dikoga@gmail.com> wrote:
> > Guys, Happy Holidays!!!
> >
> > I am tunning an existing Lucene.Net index where I am currently working.
> >
> > The perfomance was greatly increase when we talk about the search itself.
> >
> > But when it goes to facet concern, I'm getting crazy!
> >
> > This is the scenario:
> >
> > - 1.8M products on a SQL Server.
> > - Basically we concatenate all analyzed fields into one. (made the search
> > work way better)
> > - Create a specific price field to set a price range in order to group
> later
> > in a facet. (here we have ranges configured in the system that we save
> only
> > the id. The ideia is to minimize the number of possible facets)
> > - 16k categories (maybe I can do a tree here and minimize it too, need
> > opinions)
> > - 2k manufacturers
> > - Facets needed: On Sale (boolean), Review (0 - 5), Price (let's say 10
> > ranges), Free Shipping (boolean), Manufacturer (string) and Category
> > (string).
> >
> > Question 1: What do you guys would improve when writing the index?
> >
> > The lucene is embedded in ASP.NET app where the clients make a REST http
> > request. It's only one request that returns the result of the search and
> > also the facets. To make it faster, I am processing the search and each
> > facet in parallel. Which also make the search waits for the facets until
> > it's done (I figured out that is faster process in parallel than pass an
> > array).
> >
> > This is the facet search:
> >
> >             using (var simpleFacetedSearch = new
> > SimpleFacetedSearch(_indexReader, facetName))
> >             {
> >                 var hits = simpleFacetedSearch.Search(query, 1);
> >                 return hits;
> >             }
> >
> > Question 2: What do you guys recommend to make this search faster and
> also
> > does not freeze the server because high memory usage?
> >
> >
> > Thanks....
> >
> >
> > Att.,
> > ------------------
> > Koga, Diego
> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message