lucenenet-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Itamar Syn-Hershko <ita...@code972.com>
Subject Re: Facet Performance
Date Thu, 29 Dec 2016 16:27:44 GMT
I'd assume 4.8 is much better in this regard, but the design is mostly the
same (aka FieldCache). I haven't used it extensively myself - Shad may be
able to shed more light on it? (no pun intended!)

--

Itamar Syn-Hershko
http://code972.com | @synhershko <https://twitter.com/synhershko>
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Dec 29, 2016 at 6:07 PM, Koga, Diego <dikoga@gmail.com> wrote:

> Hi Itamar,
>
> Is there any difference between 3.0.3 and 4.8 on this matter?
>
>
>
> Att.,
> ------------------
> Koga, Diego
>
>
> On Thu, Dec 29, 2016 at 11:01 AM, Itamar Syn-Hershko <itamar@code972.com>
> wrote:
> > This makes sense, yes. Faceted search is by definition a high-memory
> > consumer.
> >
> > --
> >
> > Itamar Syn-Hershko
> > http://code972.com | @synhershko <https://twitter.com/synhershko>
> > Freelance Developer & Consultant
> > Lucene.NET committer and PMC member
> >
> > On Thu, Dec 29, 2016 at 5:53 PM, Koga, Diego <dikoga@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> I figured something out. Once I keep the reader and searcher as
> >> singleton, I also need to keep the facet engine.
> >>
> >> However, the SimpleFacetedSearch was consuming huge amount of memory,
> >> the SparseFacetedSearcher is way more memory optimized. I am creating
> >> one for each facet that I have and while searching, I pick the right
> >> facet searcher to use.
> >>
> >> This is how my code looks like:
> >>
> >>
> >> private void CreateReader()
> >> {
> >> if (_indexReader == null)
> >> {
> >> var dir = FSDirectory.Open(IndexDirectories.WorkDirectory);
> >>
> >> if (!IndexReader.IndexExists(dir))
> >> {
> >> dir.Dispose();
> >> throw new FileNotFoundException(
> >> "Index not found. Before initialize the reader, make sure to create
> >> the index.");
> >> }
> >> _indexReader = IndexReader.Open(dir, true);
> >> _indexSeacher = new IndexSearcher(_indexReader);
> >> _facetsSearchers = CreateFacetsSearchers(dir);
> >> }
> >> }
> >>
> >> private Dictionary<string, SparseFacetedSearcher>
> >> CreateFacetsSearchers(FSDirectory directory)
> >> {
> >> var facetReaders = new Dictionary<string, SparseFacetedSearcher>();
> >>
> >> foreach (var facet in AvailableFacets.Split(new[] { ',' },
> >> StringSplitOptions.RemoveEmptyEntries))
> >> {
> >> var indexReader = IndexReader.Open(directory, true);
> >> var facetSeacher = new SparseFacetedSearcher(indexReader, facet);
> >> facetReaders.Add(facet, facetSeacher);
> >> }
> >>
> >> return facetReaders;
> >> }
> >>
> >>
> >>
> >> Att.,
> >> ------------------
> >> Koga, Diego
> >>
> >>
> >> On Mon, Dec 26, 2016 at 6:05 PM, Koga, Diego <dikoga@gmail.com> wrote:
> >> > Guys, Happy Holidays!!!
> >> >
> >> > I am tunning an existing Lucene.Net index where I am currently
> working.
> >> >
> >> > The perfomance was greatly increase when we talk about the search
> itself.
> >> >
> >> > But when it goes to facet concern, I'm getting crazy!
> >> >
> >> > This is the scenario:
> >> >
> >> > - 1.8M products on a SQL Server.
> >> > - Basically we concatenate all analyzed fields into one. (made the
> search
> >> > work way better)
> >> > - Create a specific price field to set a price range in order to group
> >> later
> >> > in a facet. (here we have ranges configured in the system that we save
> >> only
> >> > the id. The ideia is to minimize the number of possible facets)
> >> > - 16k categories (maybe I can do a tree here and minimize it too, need
> >> > opinions)
> >> > - 2k manufacturers
> >> > - Facets needed: On Sale (boolean), Review (0 - 5), Price (let's say
> 10
> >> > ranges), Free Shipping (boolean), Manufacturer (string) and Category
> >> > (string).
> >> >
> >> > Question 1: What do you guys would improve when writing the index?
> >> >
> >> > The lucene is embedded in ASP.NET app where the clients make a REST
> http
> >> > request. It's only one request that returns the result of the search
> and
> >> > also the facets. To make it faster, I am processing the search and
> each
> >> > facet in parallel. Which also make the search waits for the facets
> until
> >> > it's done (I figured out that is faster process in parallel than pass
> an
> >> > array).
> >> >
> >> > This is the facet search:
> >> >
> >> >             using (var simpleFacetedSearch = new
> >> > SimpleFacetedSearch(_indexReader, facetName))
> >> >             {
> >> >                 var hits = simpleFacetedSearch.Search(query, 1);
> >> >                 return hits;
> >> >             }
> >> >
> >> > Question 2: What do you guys recommend to make this search faster and
> >> also
> >> > does not freeze the server because high memory usage?
> >> >
> >> >
> >> > Thanks....
> >> >
> >> >
> >> > Att.,
> >> > ------------------
> >> > Koga, Diego
> >> >
> >> >>
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message