lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: stucked indexing process
Date Wed, 12 Jul 2017 15:27:45 GMT
Hi Tamer,

Actually you can skip the check with a “hack”:

You can override the check by enforcing number of threads and number of merges at same time
by setting this on your own config when using ConcurrentMergeScheduler:

https://goo.gl/5QJpMh

If you override maxThreadCount and maxMergeCount in the CMS instance, the check is not executed.
You may pass the CMS config using IndexWriterConfig.

In addition to find out the real issue, we need more information: The problem I had was that
I cannot say where it exactly stopped for your case, because your “stack” trace-like output
had no line numbers. It would be better to run “jstack <pid>” on command line when
it hangs to see the line number in Lucene code. There are 2 places that might hang: Listing
the mount points on its own or listing their block device properties inside “/sys/block”
folder.

Just to be sure: I hope you don’t place your indexes on NFS mounts? Aren’t you?

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Tamer Gur [mailto:tgur@ebi.ac.uk]
> Sent: Wednesday, July 12, 2017 4:57 PM
> To: java-user@lucene.apache.org; Uwe Schindler <uwe@thetaphi.de>
> Subject: Re: stucked indexing process
> 
> thanks Uwe for reply. we are indexing data in a cluster where there are
> many mount points so it is possible that one them has issue or slowness
> when this check first tried but now when i execute "mount" it is
> responding all the mount points.
> 
> I was wondering is there any configuration to skip this SSD check?
> 
> Tamer
> 
> On 12/07/2017 14:15, Uwe Schindler wrote:
> >
> > Hi,
> >
> > to figure out if you system is using an SSD drive for the index
> > directory, the merge scheduler has to get the underlying mount point
> > of the index directory. As there is no direct lookup for that, it
> > needs to list all mount points in the system with a Java7 FS function.
> > And that seems to hang for some reason. Could it be that you have a
> > mount (like NFS or CIFS) that no longer responds?
> >
> > Just list all with “cat /proc/mounts” or the “mount” command and check
> > if any of them is stuck or no longer responding.
> >
> > Uwe
> >
> > -----
> >
> > Uwe Schindler
> >
> > Achterdiek 19, D-28357 Bremen
> >
> > http://www.thetaphi.de <http://www.thetaphi.de/>
> >
> > eMail: uwe@thetaphi.de
> >
> > *From:*Tamer Gur [mailto:tgur@ebi.ac.uk]
> > *Sent:* Wednesday, July 12, 2017 12:29 PM
> > *To:* java-user@lucene.apache.org
> > *Subject:* stucked indexing process
> >
> > Hi all,
> >
> > we are having an issue in our indexing pipeline time to time our
> > indexing process are stucked. Following text&picture is from jvisualvm
> > and it seems process is waiting at
> > sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext() method all the
> > time. we are using lucene 5.4.1 and java 1.8.0_65-b17.
> >
> > what can be the reason of this?
> >
> > Many Thanks
> >
> > Tamer
> >
> > text version
> >
> > "
> >
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init
> >()","100.0","73509067","73509067","3"
> > "
> >
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init
> >()","100.0","73509067","73509067","3"
> > "
> >
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addC
> ategory()","100.0","73509067","73509067","3"
> > "
> >
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.inter
> nalAddCategory()","100.0","73509067","73509067","3"
> > "
> >
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addC
> ategoryDocument()","100.0","73509067","73509067","3"
> > "
> >
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getT
> axoArrays()","100.0","73509067","73509067","3"
> > "
> >
> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initR
> eaderManager()","100.0","73509067","73509067","3"
> > "
> >
> org.apache.lucene.index.ReaderManager.<init>()","100.0","73509067","7350
> 9067","3"
> > "
> >
> org.apache.lucene.index.DirectoryReader.open()","100.0","73509067","7350
> 9067","3"
> > "
> >
> org.apache.lucene.index.IndexWriter.getReader()","100.0","73509067","735
> 09067","3"
> > "
> >
> org.apache.lucene.index.IndexWriter.maybeMerge()","100.0","73509067","7
> 3509067","3"
> > "
> >
> org.apache.lucene.index.ConcurrentMergeScheduler.merge()","100.0","7350
> 9067","73509067","3"
> > "
> >
> org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults()",
> "100.0","73509067","73509067","3"
> > "
> > org.apache.lucene.util.IOUtils.spins()","100.0","73509067","73509067","3"
> > "
> > org.apache.lucene.util.IOUtils.spins()","100.0","73509067","73509067","3"
> > "
> >
> org.apache.lucene.util.IOUtils.spinsLinux()","100.0","73509067","73509067",
> "3"
> > "
> >
> org.apache.lucene.util.IOUtils.getFileStore()","100.0","73509067","73509067"
> ,"3"
> > "
> >
> sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext()","100.0","73509067","
> 73509067","3"
> >
> > image version
> >



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message