lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: growing segments issue in lucene
Date Thu, 29 Sep 2011 08:32:34 GMT
Directory.copy is deprecated in recent releases, but reading the
javadocs it says

"Copy contents of a directory src to a directory dest. If a file in
src already exists in dest then the one in dest will be blindly
overwritten."

Which implies that any files in dest that don't exist in src will not
be overwritten.  Sounds like you need to clear out the contents of the
dest FSDirectory before doing the copy.


--
Ian.


On Thu, Sep 29, 2011 at 6:45 AM, suman.holani <suman.holani@zapak.co.in> wrote:
> Hi Ian,
> Thanks for reply.
>
> Let me reframe the question
>
> My aim is to search fast in ever changing indexes.
> So I decided onto use RAM directory index.but since it is not persistent .I
> need to copy onto FSFirectory , which is nothing but the replica of it for
> failure scenarios.
>
> RAMDirectory is used for adding/deleting and searching.I am doing
> optimization on it and then copying .I am assuming that since RAMdirectory
> is optimized so copy of it would also be optimized. But when I look at
> FSDirectory it is showing me too many files.
>
> I am not getting why the FSdirectory is growing
> Also , How can I check if  proper optimsation is done in RAMdirectory . all
> is in memory or does it stores segments somewhere in disk also.
>
>> merge factor is10
>>
>> max merge docs is2147483647
>
> My concerned is growing segments not getting merged on optimization ( as I
> can see in my fsdirectory )irrespective of merge factor.
>
>
> Regards,
> suman
>
>
> -----Original Message-----
> From: Ian Lea [mailto:ian.lea@gmail.com]
> Sent: Wednesday, September 28, 2011 8:55 PM
> To: java-user@lucene.apache.org
> Subject: Re: growing segments issue in lucene
>
> Seems a strange way of doing things.  What do you gain by messing
> around with RAMDirectory?
>
> Anyway, you appear to just add docs to the file based index, so surely
> it will keep growing?  You could try optimizing the file based index
> but that is unlikely to gain you a huge amount unless it holds deleted
> docs.
>
>
> --
> Ian.
>
>
> On Wed, Sep 28, 2011 at 11:32 AM, suman.holani <suman.holani@zapak.co.in>
> wrote:
>> Hi ,
>>
>>
>>
>> I am using lucene 3.0.3.
>>
>> It seems like either the lucene indexes are not getting optimized properly
>> as the files created in lucene index dir is growing with every doc added
>>
>> Please let me know the way to restrict the files and segments created .
>>
>>
>>
>> merge factor is10
>>
>> max merge docs is2147483647
>>
>>
>>
>>
>>
>> I am using RAM index to add doc ,optimising it and then copying it to
>> FSDirectory.
>>
>>
>>
>> indexWriter.optimize();
>>
>>               indexWriter.commit();
>>
>>               Directory.copy(RamDir, FSDirectory.open(FILE_INDEX_DIR),
>> false);  //hoping that optimized indexes will b transferred since m
>> optimizing ramdir before copying
>>
>>
>>
>>
>>
>> Files created in FSDirectory as shown as .and these are increasing with a
>> index change.
>>
>> _0.cfs   _16.frq  _1e.fdt  _1k.prx  _1s.fnm  _1y.tis  _4.nrm  _c.fdx
>  _i.tii
>> _q.frq        segments_l  _u.tii
>>
>> _0.cfx   _16.nrm  _1e.fdx  _1k.tii  _1s.frq  _20.fdt  _4.prx  _c.fnm
>  _i.tis
>> _q.nrm        segments_m  _u.tis
>>
>> _10.fdt  _16.prx  _1e.fnm  _1k.tis  _1s.nrm  _20.fdx  _4.tii  _c.frq
>  _k.fdt
>> _q.prx        segments_n  _w.fdt
>>
>> _10.fdx  _16.tii  _1e.frq  _1m.fdt  _1s.prx  _20.fnm  _4.tis  _c.nrm
>  _k.fdx
>> _q.tii        segments_o  _w.fdx
>>
>> _10.fnm  _16.tis  _1e.nrm  _1m.fdx  _1s.tii  _20.frq  _6.fdt  _c.prx
>  _k.fnm
>> _q.tis        segments_p  _w.fnm
>>
>> _10.frq  _18.fdt  _1e.prx  _1m.fnm  _1s.tis  _20.nrm  _6.fdx  _c.tii
>  _k.frq
>> segments_10   segments_q  _w.frq
>>
>> _10.nrm  _18.fdx  _1e.tii  _1m.frq  _1u.fdt  _20.prx  _6.fnm  _c.tis
>  _k.nrm
>> segments_11   segments_r  _w.nrm
>>
>> _10.prx  _18.fnm  _1e.tis  _1m.nrm  _1u.fdx  _20.tii  _6.frq  _e.fdt
>  _k.prx
>> segments_12   segments_s  _w.prx
>>
>> _10.tii  _18.frq  _1g.fdt  _1m.prx  _1u.fnm  _20.tis  _6.nrm  _e.fdx
>  _k.tii
>> segments_13   segments_t  _w.tii
>>
>> _10.tis  _18.nrm  _1g.fdx  _1m.tii  _1u.frq  _22.fdt  _6.prx  _e.fnm
>  _k.tis
>> segments_2    segments_u  _w.tis
>>
>> _12.fdt  _18.prx  _1g.fnm  _1m.tis  _1u.nrm  _22.fdx  _6.tii  _e.frq
>  _m.fdt
>> segments_3    segments_v  _y.fdt
>>
>> _12.fdx  _18.tii  _1g.frq  _1o.fdt  _1u.prx  _22.fnm  _6.tis  _e.nrm
>  _m.fdx
>> segments_4    segments_w  _y.fdx
>>
>> _12.fnm  _18.tis  _1g.nrm  _1o.fdx  _1u.tii  _22.frq  _8.fdt  _e.prx
>  _m.fnm
>> segments_5    segments_x  _y.fnm
>>
>> _12.frq  _1a.fdt  _1g.prx  _1o.fnm  _1u.tis  _22.nrm  _8.fdx  _e.tii
>  _m.frq
>> segments_6    segments_y  _y.frq
>>
>> _12.nrm  _1a.fdx  _1g.tii  _1o.frq  _1w.fdt  _22.prx  _8.fnm  _e.tis
>  _m.nrm
>> segments_7    segments_z  _y.nrm
>>
>> _12.prx  _1a.fnm  _1g.tis  _1o.nrm  _1w.fdx  _22.tii  _8.frq  _g.fdt
>  _m.prx
>> segments_8    _s.fdt      _y.prx
>>
>> _12.tii  _1a.frq  _1i.fdt  _1o.prx  _1w.fnm  _22.tis  _8.nrm  _g.fdx
>  _m.tii
>> segments_9    _s.fdx      _y.tii
>>
>> _12.tis  _1a.nrm  _1i.fdx  _1o.tii  _1w.frq  _2.fdt   _8.prx  _g.fnm
>  _m.tis
>> segments_a    _s.fnm      _y.tis
>>
>> _14.fdt  _1a.prx  _1i.fnm  _1o.tis  _1w.nrm  _2.fdx   _8.tii  _g.frq
>  _o.fdt
>> segments_b    _s.frq
>>
>> _14.fdx  _1a.tii  _1i.frq  _1q.fdt  _1w.prx  _2.fnm   _8.tis  _g.nrm
>  _o.fdx
>> segments_c    _s.nrm
>>
>> _14.fnm  _1a.tis  _1i.nrm  _1q.fdx  _1w.tii  _2.frq   _a.fdt  _g.prx
>  _o.fnm
>> segments_d    _s.prx
>>
>> _14.frq  _1c.fdt  _1i.prx  _1q.fnm  _1w.tis  _2.nrm   _a.fdx  _g.tii
>  _o.frq
>> segments_e    _s.tii
>>
>> _14.nrm  _1c.fdx  _1i.tii  _1q.frq  _1y.fdt  _2.prx   _a.fnm  _g.tis
>  _o.nrm
>> segments_f    _s.tis
>>
>> _14.prx  _1c.fnm  _1i.tis  _1q.nrm  _1y.fdx  _2.tii   _a.frq  _i.fdt
>  _o.prx
>> segments_g    _u.fdt
>>
>> _14.tii  _1c.frq  _1k.fdt  _1q.prx  _1y.fnm  _2.tis   _a.nrm  _i.fdx
>  _o.tii
>> segments.gen  _u.fdx
>>
>> _14.tis  _1c.nrm  _1k.fdx  _1q.tii  _1y.frq  _4.fdt   _a.prx  _i.fnm
>  _o.tis
>> segments_h    _u.fnm
>>
>> _16.fdt  _1c.prx  _1k.fnm  _1q.tis  _1y.nrm  _4.fdx   _a.tii  _i.frq
>  _q.fdt
>> segments_i    _u.frq
>>
>> _16.fdx  _1c.tii  _1k.frq  _1s.fdt  _1y.prx  _4.fnm   _a.tis  _i.nrm
>  _q.fdx
>> segments_j    _u.nrm
>>
>> _16.fnm  _1c.tis  _1k.nrm  _1s.fdx  _1y.tii  _4.frq   _c.fdt  _i.prx
>  _q.fnm
>> segments_k    _u.prx
>>
>>
>>
>> Regards
>>
>> suman
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message