Mailing-List: contact user-help@lucenenet.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@lucenenet.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOv9rehgJRcdb7soT6baUj4GCq14E1GN6LuSOZTn=y4ioP-dGw@mail.gmail.com>
References: <CAOv9rehgJRcdb7soT6baUj4GCq14E1GN6LuSOZTn=y4ioP-dGw@mail.gmail.com>
From: "Koga, Diego" <dikoga@gmail.com>
Date: Thu, 29 Dec 2016 10:53:54 -0500
Message-ID: <CAOv9regx_c1ooKaVuc_GjJ8J1UD9Ro3mUgz44nrFNexcwgOiGg@mail.gmail.com>
Subject: Re: Facet Performance
To: user@lucenenet.apache.org
Content-Type: text/plain; charset=UTF-8
archived-at: Thu, 29 Dec 2016 15:54:26 -0000

Hi all,

I figured something out. Once I keep the reader and searcher as
singleton, I also need to keep the facet engine.

However, the SimpleFacetedSearch was consuming huge amount of memory,
the SparseFacetedSearcher is way more memory optimized. I am creating
one for each facet that I have and while searching, I pick the right
facet searcher to use.

This is how my code looks like:


private void CreateReader()
{
if (_indexReader == null)
{
var dir = FSDirectory.Open(IndexDirectories.WorkDirectory);

if (!IndexReader.IndexExists(dir))
{
dir.Dispose();
throw new FileNotFoundException(
"Index not found. Before initialize the reader, make sure to create
the index.");
}
_indexReader = IndexReader.Open(dir, true);
_indexSeacher = new IndexSearcher(_indexReader);
_facetsSearchers = CreateFacetsSearchers(dir);
}
}

private Dictionary<string, SparseFacetedSearcher>
CreateFacetsSearchers(FSDirectory directory)
{
var facetReaders = new Dictionary<string, SparseFacetedSearcher>();

foreach (var facet in AvailableFacets.Split(new[] { ',' },
StringSplitOptions.RemoveEmptyEntries))
{
var indexReader = IndexReader.Open(directory, true);
var facetSeacher = new SparseFacetedSearcher(indexReader, facet);
facetReaders.Add(facet, facetSeacher);
}

return facetReaders;
}


Att.,
------------------
Koga, Diego


On Mon, Dec 26, 2016 at 6:05 PM, Koga, Diego <dikoga@gmail.com> wrote:
> Guys, Happy Holidays!!!
>
> I am tunning an existing Lucene.Net index where I am currently working.
>
> The perfomance was greatly increase when we talk about the search itself.
>
> But when it goes to facet concern, I'm getting crazy!
>
> This is the scenario:
>
> - 1.8M products on a SQL Server.
> - Basically we concatenate all analyzed fields into one. (made the search
> work way better)
> - Create a specific price field to set a price range in order to group later
> in a facet. (here we have ranges configured in the system that we save only
> the id. The ideia is to minimize the number of possible facets)
> - 16k categories (maybe I can do a tree here and minimize it too, need
> opinions)
> - 2k manufacturers
> - Facets needed: On Sale (boolean), Review (0 - 5), Price (let's say 10
> ranges), Free Shipping (boolean), Manufacturer (string) and Category
> (string).
>
> Question 1: What do you guys would improve when writing the index?
>
> The lucene is embedded in ASP.NET app where the clients make a REST http
> request. It's only one request that returns the result of the search and
> also the facets. To make it faster, I am processing the search and each
> facet in parallel. Which also make the search waits for the facets until
> it's done (I figured out that is faster process in parallel than pass an
> array).
>
> This is the facet search:
>
>             using (var simpleFacetedSearch = new
> SimpleFacetedSearch(_indexReader, facetName))
>             {
>                 var hits = simpleFacetedSearch.Search(query, 1);
>                 return hits;
>             }
>
> Question 2: What do you guys recommend to make this search faster and also
> does not freeze the server because high memory usage?
>
>
> Thanks....
>
>
> Att.,
> ------------------
> Koga, Diego
>
>>
>