lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Optimize FTS memory footprint
Date Wed, 13 Dec 2017 00:25:15 GMT
Try upgrading Elasticsearch -- it's up to 6.0 release just a few week ago
now -- its (and Lucene's) memory usage has decreased over time.

The _uid field in particular will always be costly, unfortunately.  Since
it's a primary key, every term will be unique, and the term index has to
work hard to store all the prefixes for those keys.

Mike McCandless

http://blog.mikemccandless.com

On Sat, Dec 2, 2017 at 9:42 PM, Bingtao Yin <ybtsdst@gmail.com> wrote:

> Thanks, mike. I'm facing a similar problem.
> I'm running a 2.0 elasticsearch cluster, and find the fst of _uid field
> takes a lot of memory. The _uid field is not analyzed and generated by
> elasticsearch, which also has high cardinality.
> Is there any ways to  reduce memory cost for _uid field? Thanks.
>
>
> 2017-11-29 5:47 GMT+08:00 elirev <elirevach12@gmail.com>:
>
> > Thanks   Mike .
> > I did not  find  any  clear  way to know it its FST or Norm , or
> something
> > else ( unless i miss something )  the fact the FST is an in memory prefix
> > index lead me to think it using most of the heap   .
> > Our  mapping is normal with around of 200 columns one of the columns is
> > nested object with limited amount of objects (up to 4 instances  )   , we
> > are using monthly base indexes  (keep 6 month open ) . In last month  i
> see
> > dramatic extra  allocation on the segment memory (around 30% where in
> > regulare    month is around 5%)  , the only change i see is that the
> nested
> > object is now include avg 8 instances  )  , this increases the amount of
> > the hidden document we have now on the  index (about more then twice) .
> > When we optimize the index the amount of allocation memory was reduced
> (we
> > see it only after rolling restart the nodes )   .
> >
> > If you don't mind  i have few question :
> > 1) Do you know about an  way  to figure   out which component is taking
> all
> > this memory .
> > 2) Do you see relation between the fact that the nested objects was
> > increases to the extra memory allocation we have ?
> > 3) Did FST memory usage is  impacted by the fact we optimize the
> > problematic
> > index  and why  we see it only after restarting ES service
> >
> > Thanks mike
> >
> > .
> >
> >
> >
> >
> >
> >
> > --
> > Sent from: http://lucene.472066.n3.nabble.com/Lucene-Java-Users-
> > f532864.html
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message