Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id AAB10200CB5 for ; Wed, 12 Jul 2017 15:15:37 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A94F2168BF0; Wed, 12 Jul 2017 13:15:37 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A3F7B168BEF for ; Wed, 12 Jul 2017 15:15:36 +0200 (CEST) Received: (qmail 71008 invoked by uid 500); 12 Jul 2017 13:15:35 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 70993 invoked by uid 99); 12 Jul 2017 13:15:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jul 2017 13:15:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8E4AB192308 for ; Wed, 12 Jul 2017 13:15:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.318 X-Spam-Level: X-Spam-Status: No, score=0.318 tagged_above=-999 required=6.31 tests=[HTML_IMAGE_RATIO_04=0.61, HTML_MESSAGE=2, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id sWWfu4oKcWIA for ; Wed, 12 Jul 2017 13:15:30 +0000 (UTC) Received: from mail.sd-datasolutions.de (serv2.sd-datasolutions.de [78.47.65.36]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 6A4DF62677 for ; Wed, 12 Jul 2017 13:15:30 +0000 (UTC) Received: from VEGA (fw1.marum.de [134.102.234.1]) by mail.sd-datasolutions.de (Postfix) with ESMTPSA id 2944F48011C for ; Wed, 12 Jul 2017 13:15:23 +0000 (UTC) X-NSA-Greeting: Dear NSA, have fun with reading and analyzing this e-mail! From: "Uwe Schindler" To: References: <9200cc0a-698a-bd4a-61be-09cf8e08b2a6@ebi.ac.uk> In-Reply-To: <9200cc0a-698a-bd4a-61be-09cf8e08b2a6@ebi.ac.uk> Subject: RE: stucked indexing process Date: Wed, 12 Jul 2017 15:15:22 +0200 Message-ID: <062d01d2fb10$eb729810$c257c830$@thetaphi.de> MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_NextPart_000_062E_01D2FB21.AEFE2730" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQIFtpPRfkZQe2hOTj6zIPo6Vbo3/qHqs8ZQ Content-Language: de archived-at: Wed, 12 Jul 2017 13:15:37 -0000 ------=_NextPart_000_062E_01D2FB21.AEFE2730 Content-Type: multipart/alternative; boundary="----=_NextPart_001_062F_01D2FB21.AEFE2730" ------=_NextPart_001_062F_01D2FB21.AEFE2730 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, =20 to figure out if you system is using an SSD drive for the index = directory, the merge scheduler has to get the underlying mount point of = the index directory. As there is no direct lookup for that, it needs to = list all mount points in the system with a Java7 FS function. And that = seems to hang for some reason. Could it be that you have a mount (like = NFS or CIFS) that no longer responds? =20 Just list all with =E2=80=9Ccat /proc/mounts=E2=80=9D or the = =E2=80=9Cmount=E2=80=9D command and check if any of them is stuck or no = longer responding. =20 Uwe =20 ----- Uwe Schindler Achterdiek 19, D-28357 Bremen http://www.thetaphi.de eMail: uwe@thetaphi.de =20 From: Tamer Gur [mailto:tgur@ebi.ac.uk]=20 Sent: Wednesday, July 12, 2017 12:29 PM To: java-user@lucene.apache.org Subject: stucked indexing process =20 Hi all, we are having an issue in our indexing pipeline time to time our = indexing process are stucked. Following text&picture is from jvisualvm = and it seems process is waiting at = sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext() method all the = time. we are using lucene 5.4.1 and java 1.8.0_65-b17. what can be the reason of this?=20 Many Thanks Tamer text version " = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.= ()","100.0","73509067","73509067","3" " = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.= ()","100.0","73509067","73509067","3" " = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCat= egory()","100.0","73509067","73509067","3" " = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.intern= alAddCategory()","100.0","73509067","73509067","3" " = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCat= egoryDocument()","100.0","73509067","73509067","3" " = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getTax= oArrays()","100.0","73509067","73509067","3" " = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initRe= aderManager()","100.0","73509067","73509067","3" " = org.apache.lucene.index.ReaderManager.()","100.0","73509067","73509= 067","3" " = org.apache.lucene.index.DirectoryReader.open()","100.0","73509067","73509= 067","3" " = org.apache.lucene.index.IndexWriter.getReader()","100.0","73509067","7350= 9067","3" " = org.apache.lucene.index.IndexWriter.maybeMerge()","100.0","73509067","735= 09067","3" " = org.apache.lucene.index.ConcurrentMergeScheduler.merge()","100.0","735090= 67","73509067","3" " = org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults()","= 100.0","73509067","73509067","3" " = org.apache.lucene.util.IOUtils.spins()","100.0","73509067","73509067","3"= " = org.apache.lucene.util.IOUtils.spins()","100.0","73509067","73509067","3"= " = org.apache.lucene.util.IOUtils.spinsLinux()","100.0","73509067","73509067= ","3" " = org.apache.lucene.util.IOUtils.getFileStore()","100.0","73509067","735090= 67","3" " = sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext()","100.0","73509067"= ,"73509067","3" image version ------=_NextPart_001_062F_01D2FB21.AEFE2730 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable

Hi,

 

to figure out if = you system is using an SSD drive for the index directory, the merge = scheduler has to get the underlying mount point of the index directory. = As there is no direct lookup for that, it needs to list all mount points = in the system with a Java7 FS function. And that seems to hang for some = reason. Could it be that you have a mount (like NFS or CIFS) that no = longer responds?

 

Just list all with = =E2=80=9Ccat /proc/mounts=E2=80=9D or the =E2=80=9Cmount=E2=80=9D = command and check if any of them is stuck or no longer = responding.

 

Uwe

 

-----

Uwe = Schindler

Achterdiek 19, D-28357 = Bremen

http://www.thetaphi.de<= /p>

eMail: = uwe@thetaphi.de

 

From: Tamer Gur [mailto:tgur@ebi.ac.uk] =
Sent: Wednesday, July 12, 2017 12:29 PM
To: = java-user@lucene.apache.org
Subject: stucked indexing = process

 

Hi all,

we = are having an issue in our indexing pipeline time to time our indexing = process are stucked. Following text&picture is from jvisualvm and it = seems process is waiting at = sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext() method all the = time. we are using lucene 5.4.1 and java  = 1.8.0_65-b17.

what can be the reason of this? =

Many = Thanks

Tamer

text = version

"       =     = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<in= it>()","100.0","73509067","73509067"= ;,"3"
"        =     = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<in= it>()","100.0","73509067","73509067"= ;,"3"
"        =      = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCat= egory()","100.0","73509067","73509067"= ,"3"
"        &= nbsp;     = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.intern= alAddCategory()","100.0","73509067","735090= 67","3"
"       = ;        = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCat= egoryDocument()","100.0","73509067","735090= 67","3"
"       = ;         = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getTax= oArrays()","100.0","73509067","73509067&quo= t;,"3"
"        = ;         = org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initRe= aderManager()","100.0","73509067","73509067= ","3"
"       &= nbsp;          = org.apache.lucene.index.ReaderManager.<init>()","100.0&qu= ot;,"73509067","73509067","3"
"&nbs= p;            = ;      = org.apache.lucene.index.DirectoryReader.open()","100.0",&q= uot;73509067","73509067","3"
" &nbs= p;            = ;      = org.apache.lucene.index.IndexWriter.getReader()","100.0",&= quot;73509067","73509067","3"
" &nb= sp;           &nbs= p;       = org.apache.lucene.index.IndexWriter.maybeMerge()","100.0",= "73509067","73509067","3"
" &n= bsp;           &nb= sp;        = org.apache.lucene.index.ConcurrentMergeScheduler.merge()","100.= 0","73509067","73509067","3"
"=             &= nbsp;          = org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults()&qu= ot;,"100.0","73509067","73509067","3&q= uot;
"          = ;            =   = org.apache.lucene.util.IOUtils.spins()","100.0","7350= 9067","73509067","3"
"   =             &= nbsp;         = org.apache.lucene.util.IOUtils.spins()","100.0","7350= 9067","73509067","3"
"   =             &= nbsp;          = org.apache.lucene.util.IOUtils.spinsLinux()","100.0","= ;73509067","73509067","3"
"  &= nbsp;           &n= bsp;            = org.apache.lucene.util.IOUtils.getFileStore()","100.0",&qu= ot;73509067","73509067","3"
"  = ;            =             &= nbsp; = sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext()","100.0&q= uot;,"73509067","73509067","3"

image version

------=_NextPart_001_062F_01D2FB21.AEFE2730-- ------=_NextPart_000_062E_01D2FB21.AEFE2730--