lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anshum <ansh...@gmail.com>
Subject Re: slow search threads during a disk copy
Date Mon, 23 Aug 2010 10:38:52 GMT
There is bound to be IO contention. I'm sure iostat will give you a much
better picture on it.

--
Anshum Gupta
http://ai-cafe.blogspot.com


On Mon, Aug 23, 2010 at 3:13 PM, <gaganb@graffiti.net> wrote:

> Yes, all version directories are on the same disk. iostat output should be
> useful. Using rsync is complicated because of legacy reasons - there's a
> change impact to manage.
>
>
> Intererstingly, the copy is quite fast (around 30s) when there are no
> searches in progress.
>
>
>
> Thanks,
> - Gagan
>
>
> -----Original Message-----
> From: Ian Lea <ian.lea@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Mon, Aug 23, 2010 2:10 pm
> Subject: Re: slow search threads during a disk copy
>
>
> Well, there still sounds like plenty of scope for IO contention.  Are
> source and dest directories on the same disk?  What does iostat show?
> Could you use rsync rather than cp -rp, or something else that only
> copies changes?
>
>
> --
> Ian.
>
>
> On Mon, Aug 23, 2010 at 9:20 AM,  <gaganb@graffiti.net> wrote:
> >
> > We suspected the same at first, and so modified the flow to do the copy
> of the
> new version "before" searching is activated on it. So, there are no
> searches on
> the new version dir at the time of copy - yet we still see slowness during
> the
> disk copy.
> >
> >
> > Thanks,
> > - Gagan
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Anshum <anshumg@gmail.com>
> > To: java-user@lucene.apache.org
> > Sent: Mon, Aug 23, 2010 1:17 pm
> > Subject: Re: slow search threads during a disk copy
> >
> >
> > Seems like a case of I/O issues. You may be reading content off the index
> > while performing searches while the I/O for copy is also happening.
> >
> > --
> > Anshum Gupta
> > http://ai-cafe.blogspot.com
> >
> >
> > On Mon, Aug 23, 2010 at 1:12 PM, <gaganb@graffiti.net> wrote:
> >
> >>
> >> Hi all,
> >>
> >>
> >> We're observing search threads slowing down during directory copies
> >> performed during updates to the index. The thread dump shows search
> threads
> >> blocked on a FSDirectory$FSIndexInput$Descriptor instance:
> >>
> >>
> >>
> >> "Worker Thread - 12" daemon prio=10 tid=0x082b2400 nid=0x4654 waiting
> for
> >> monitor entry [0x988ed000..0x988edf30]
> >>   java.lang.Thread.State: BLOCKED (on object monitor)
> >>    at
> >>
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:542)
> >>    - waiting to lock <0xc79c9900> (a
> >> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
> >>    at
> >>
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
> >>    at
> >>
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
> >>    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
> >>    at
> >> org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:133)
> >>    at
> >>
> org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.read(MultiSegmentReader.java:573)
> >>    at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
> >>    at
> >>
> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:80)
> >>    at
> >>
> org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:48)
> >>    at
> >> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:319)
> >>    at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> >>    at org.apache.lucene.search.Searcher.search(Searcher.java:118)
> >>
> >>    ...
> >>
> >>
> >> The directory copy we do is a 'cp -pr <src-dir>/* <dest-dir>' command
> prior
> >> to applying changes (addDocument calls) on the current "available"
> segment.
> >> This takes >7 mins to copy a directory of size 1.4G. During this time
> >> window, the searches are slow and the above thread stacks are observed.
> >>
> >>
> >> Could there be any system level limits we're hitting?
> >>
> >>
> >> Our test environment is:
> >> lucene-2.3.2
> >> 4x2.6 GHz, 16G memory
> >> Red Hat 3.4.6-9
> >>
> >>
> >> Thanks and regards,
> >>
> >>
> >> - Gagan
> >>
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message