lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamer Gur <t...@ebi.ac.uk>
Subject Re: stucked indexing process
Date Wed, 12 Jul 2017 16:32:17 GMT
thanks a lot for the "hack" and jstack suggestion Uwe i will try them.

Unfortunately we are in the NFS mount since we don't have other choices.

also might be related, in the cluster(computing farm) we are indexing 
parallel several size of different datasets and most them are indexed 
without problem the one that are stucking recently always the one we 
allocate them 1 cpu since they are very small and easy to index we ask 
less cpu to use our cluster efficiently. I will also increase these 
datasets cpu size to 2 and to see if it helps.

thanks again
Tamer

On 12/07/2017 16:27, Uwe Schindler wrote:
> Hi Tamer,
>
> Actually you can skip the check with a “hack”:
>
> You can override the check by enforcing number of threads and number of merges at same
time by setting this on your own config when using ConcurrentMergeScheduler:
>
> https://goo.gl/5QJpMh
>
> If you override maxThreadCount and maxMergeCount in the CMS instance, the check is not
executed. You may pass the CMS config using IndexWriterConfig.
>
> In addition to find out the real issue, we need more information: The problem I had was
that I cannot say where it exactly stopped for your case, because your “stack” trace-like
output had no line numbers. It would be better to run “jstack <pid>” on command
line when it hangs to see the line number in Lucene code. There are 2 places that might hang:
Listing the mount points on its own or listing their block device properties inside “/sys/block”
folder.
>
> Just to be sure: I hope you don’t place your indexes on NFS mounts? Aren’t you?
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>> -----Original Message-----
>> From: Tamer Gur [mailto:tgur@ebi.ac.uk]
>> Sent: Wednesday, July 12, 2017 4:57 PM
>> To: java-user@lucene.apache.org; Uwe Schindler <uwe@thetaphi.de>
>> Subject: Re: stucked indexing process
>>
>> thanks Uwe for reply. we are indexing data in a cluster where there are
>> many mount points so it is possible that one them has issue or slowness
>> when this check first tried but now when i execute "mount" it is
>> responding all the mount points.
>>
>> I was wondering is there any configuration to skip this SSD check?
>>
>> Tamer
>>
>> On 12/07/2017 14:15, Uwe Schindler wrote:
>>> Hi,
>>>
>>> to figure out if you system is using an SSD drive for the index
>>> directory, the merge scheduler has to get the underlying mount point
>>> of the index directory. As there is no direct lookup for that, it
>>> needs to list all mount points in the system with a Java7 FS function.
>>> And that seems to hang for some reason. Could it be that you have a
>>> mount (like NFS or CIFS) that no longer responds?
>>>
>>> Just list all with “cat /proc/mounts” or the “mount” command and check
>>> if any of them is stuck or no longer responding.
>>>
>>> Uwe
>>>
>>> -----
>>>
>>> Uwe Schindler
>>>
>>> Achterdiek 19, D-28357 Bremen
>>>
>>> http://www.thetaphi.de <http://www.thetaphi.de/>
>>>
>>> eMail: uwe@thetaphi.de
>>>
>>> *From:*Tamer Gur [mailto:tgur@ebi.ac.uk]
>>> *Sent:* Wednesday, July 12, 2017 12:29 PM
>>> *To:* java-user@lucene.apache.org
>>> *Subject:* stucked indexing process
>>>
>>> Hi all,
>>>
>>> we are having an issue in our indexing pipeline time to time our
>>> indexing process are stucked. Following text&picture is from jvisualvm
>>> and it seems process is waiting at
>>> sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext() method all the
>>> time. we are using lucene 5.4.1 and java 1.8.0_65-b17.
>>>
>>> what can be the reason of this?
>>>
>>> Many Thanks
>>>
>>> Tamer
>>>
>>> text version
>>>
>>> "
>>>
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init
>>> ()","100.0","73509067","73509067","3"
>>> "
>>>
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init
>>> ()","100.0","73509067","73509067","3"
>>> "
>>>
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addC
>> ategory()","100.0","73509067","73509067","3"
>>> "
>>>
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.inter
>> nalAddCategory()","100.0","73509067","73509067","3"
>>> "
>>>
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addC
>> ategoryDocument()","100.0","73509067","73509067","3"
>>> "
>>>
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getT
>> axoArrays()","100.0","73509067","73509067","3"
>>> "
>>>
>> org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initR
>> eaderManager()","100.0","73509067","73509067","3"
>>> "
>>>
>> org.apache.lucene.index.ReaderManager.<init>()","100.0","73509067","7350
>> 9067","3"
>>> "
>>>
>> org.apache.lucene.index.DirectoryReader.open()","100.0","73509067","7350
>> 9067","3"
>>> "
>>>
>> org.apache.lucene.index.IndexWriter.getReader()","100.0","73509067","735
>> 09067","3"
>>> "
>>>
>> org.apache.lucene.index.IndexWriter.maybeMerge()","100.0","73509067","7
>> 3509067","3"
>>> "
>>>
>> org.apache.lucene.index.ConcurrentMergeScheduler.merge()","100.0","7350
>> 9067","73509067","3"
>>> "
>>>
>> org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults()",
>> "100.0","73509067","73509067","3"
>>> "
>>> org.apache.lucene.util.IOUtils.spins()","100.0","73509067","73509067","3"
>>> "
>>> org.apache.lucene.util.IOUtils.spins()","100.0","73509067","73509067","3"
>>> "
>>>
>> org.apache.lucene.util.IOUtils.spinsLinux()","100.0","73509067","73509067",
>> "3"
>>> "
>>>
>> org.apache.lucene.util.IOUtils.getFileStore()","100.0","73509067","73509067"
>> ,"3"
>>> "
>>>
>> sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext()","100.0","73509067","
>> 73509067","3"
>>> image version
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message