lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Duke DAI <duke.dai....@gmail.com>
Subject Re: Re: memory cost in forceMerge(1)
Date Tue, 11 Aug 2015 15:31:04 GMT
>From my experience, you must hit some system issue. You should check disk
performance at first, disk queue length on Windows. Or you can enable gc
verbose to know the gc activities in details.

I designed auto upgrade mechanism in application by calling forceMerge(1),
to eradicate hybrid index format as to solve file not found exception after
merging index directories. It handled far more than 15G indexes by various
customers, and no problem reported so far.

I recently designed special merge policy for specific application, and just
verified forceMerge(5) on 15G index taking less than 10 minute(5
minute???). The server is so common on hardware, 4G heap assigned.


Best regards,
Duke
If not now, when? If not me, who?

On Tue, Aug 11, 2015 at 7:00 PM, Phaneendra N <phaneendran.gitam@gmail.com>
wrote:

> There could be other applications running on the machine with 24 GB memory?
> Which would result in total available memory less than what is required. In
> this case there may be disk swap, which would take long time.
> In theory, if you run this test on machines with memory 50 GB and 100 GB in
> this case. You might see the same result.
>
> -Phani
>
> On Tue, Aug 11, 2015 at 2:51 PM, 丁儒 <bforevdr@126.com> wrote:
>
> >
> >
> > The index will not change oftenly, so we call forceMerge in the end. Will
> > forceMerge(1) cost too much memory? And the final size of the index is
> > 15GB. I just want to know why different machine cost different time in
> > forceMerge, them have the same cpu and disk, but different size of
> memory.
> > One has 24GB, the other has 48 GB. And i allocate -Xmx15GB for the
> > application.
> >
> >
> >
> >
> >
> >
> > At 2015-08-11 12:30:38, "Erick Erickson" <erickerickson@gmail.com>
> wrote:
> > >It is generally unnecessary to use forceMerge, that's a legacy from
> > >older versions of Lucene/Solr.
> > >Especially if the index is constantly changing, forceMerge generally
> > >is both expensive and not
> > >very useful.
> > >
> > >These indexes must be huge though if any of them are taking 8 hours.
> > >What's the background here?
> > >
> > >Best,
> > >Erick
> > >
> > >On Mon, Aug 10, 2015 at 6:28 PM, 丁儒 <bforevdr@126.com> wrote:
> > >> GreetingsNow, i'm using lucene , the version is 4.10.3. For some
> > reason, i called forceMerge(1) in the end,and the final  space of the
> Index
> > Library is 15 GB. . But i found that forceMerge(1) cost a lot of time,
> and
> > on different machines ,the time differs.  Is this caused by the different
> > size of memory of the macheines. One has 24GB memory, the other has 48GB
> > memory. And the machine with 24 GB memory cost nearly 18 hours in
> > forceMerge(1), and the machine with 48 GB memory cost nearly 8 hours in
> > forceMerge(1). I guess the forceMerge(1) needs too much memory, so it
> costs
> > more time in less memory machine. Am i right ?
> > >>
> > >>
> > >>     Also, can forceMerge(1)  speed up the search?
> > >>
> > >>
> > >> Thank you
> > >> All
> > >
> > >---------------------------------------------------------------------
> > >To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message