From java-user-return-64328-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Fri May 3 08:06:03 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id EA8FB18064D for ; Fri, 3 May 2019 10:06:02 +0200 (CEST) Received: (qmail 6206 invoked by uid 500); 3 May 2019 08:05:59 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 6182 invoked by uid 99); 3 May 2019 08:05:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 May 2019 08:05:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8F3FE180E36 for ; Fri, 3 May 2019 08:05:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2 X-Spam-Level: ** X-Spam-Status: No, score=2 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=mikemccandless-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id ephmrLz6wd-v for ; Fri, 3 May 2019 08:05:54 +0000 (UTC) Received: from mail-vs1-f45.google.com (mail-vs1-f45.google.com [209.85.217.45]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 615675F6C6 for ; Fri, 3 May 2019 08:05:54 +0000 (UTC) Received: by mail-vs1-f45.google.com with SMTP id z145so3067187vsc.0 for ; Fri, 03 May 2019 01:05:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mikemccandless-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=M++zV+TIOF5GtmeLmJQmZEyqyXmp0Qp49EoXu6R5jP0=; b=zecYu9cpA2JzmuVZ8lObTUSOt0oWanJ15wiZZUfhEKVftJyzposC7SEW0fXmp3kiw5 xp/VHDWueN88xa4GqhYQNvn5iyLgHjBVV/7Cpl49PzvmAazSr4mJOiX81kqxdIoCUnsL vlDyDdH15KBlIIeqFshXrnWqUL9M2pQsGGjeTVR0P/UHFdlBULRhk3iK9HLr9Yk+gsEl NUEs0iGwbAEqM3pmqiylXfEQNde5aCU/ypw+ZmWzmnz34gj9lN/+xV6oKq5njbwxWZN7 SjucRv51JU+hAU4Yl8wvwgKMfOHBVBBPoH9mYD1oMvh2rf4p5I5OzYs/ak6RVSjFy/CU FHbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=M++zV+TIOF5GtmeLmJQmZEyqyXmp0Qp49EoXu6R5jP0=; b=XnQF6u5zKVWhDFARFxwXb2gwuj0TjDnosYLssnzVum0x2/UgtdrHKu4CySJlp/tpZP /YS7rHz2eKirK7CTQ89GNWmX+a96nI4cehZ4VPjLKSFtznVUzAknjPgrV1LI5hvPuMSj EfbofVCkETCjjon25zLuh7csKcEduvNjWVCNHdBntKzM/Gq2MOrHwtk+n+ZvcGfV0//k J4Wo+Qn0HiamgKmQIBCTPQtD7UnNdv13RoK+QeJvUfDXCE2SYfJOCvem/W2wGzhjOK0V RSLBwe3jcgeWV18lt6kiZFgEzcdw/h44t4jpQBqans4tgQCwWOeNS6RWtNzHqbkQAR9C KE8Q== X-Gm-Message-State: APjAAAVOIfxvHW7SMsO/i0/oGDu3ZzjmAcUDp6hplPVG8c2O0gUj0xJA dqJFikigCdZ1WYEk+CQGBxMEwAmDcB2ZyIirW15w5aMqcBU= X-Google-Smtp-Source: APXvYqzO3qZrgKZsvN73k7bRAn2TfssF7HCCVSO7nWbN9O5humTbWYgFcvhKXfC38k+Bqbd/QJUj4riF4LpQ+6iXZlI= X-Received: by 2002:a67:f105:: with SMTP id n5mr4757099vsk.181.1556870753569; Fri, 03 May 2019 01:05:53 -0700 (PDT) MIME-Version: 1.0 References: <0F811AD50020C94291196596D19310F801245E94B8@msexch01.multi-support.lan> In-Reply-To: <0F811AD50020C94291196596D19310F801245E94B8@msexch01.multi-support.lan> From: Michael McCandless Date: Fri, 3 May 2019 04:05:17 -0400 Message-ID: Subject: Re: ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter To: Lucene Users , tri@multi-support.com Content-Type: multipart/alternative; boundary="0000000000003a55ca0587f738d8" --0000000000003a55ca0587f738d8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Note that the -Xint flag will make your code run tremendously more slowly! Likely to the point of not really being usable. But it'd be interesting to see if that side-steps the bug. Is it possible to test with OpenJDK as well? The BKDWriter code is quite complex, so it is also possible there is a Lucene bug at work. Can you open an issue in Lucene's jira and we can iterate there? Thanks, Mike McCandless http://blog.mikemccandless.com On Wed, May 1, 2019 at 9:34 AM Torben Riis wrote: > Hi, > > > > I=E2=80=99m a bit stuck here and needs a clue or two in order to continue= our > investigations. Hope that someone can help. :) > > > > Periodically, around once a month, we get the below > ArrayIndexOutOfBoundsException on our system. We use multiple indexes and > the error can originate from any of them, but the error always occurs in > line 1217 in BKDWriter (during a System.arraycopy). > > > > We found a couple of issues on the net regarding JIT optimization problem > related to J9, but they all looks like they have been resolved and cannot > be reproduced anymore. But nevertheless, we have just added the -Xint fla= g > (disable JIT compiler) in order to see whether this has any impact. > Unfortunately we do not have the result of this yet, but I=E2=80=99ll of = course > post it when it is known. > > > > Are there any of you clever guys out there, that has some good ideas > further investigations? Or have seen such issue before? > > > > We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform. > > > > > > *Java version* > > java version "1.8.0_191" > > Java(TM) SE Runtime Environment (build 8.0.5.25 - > pap6480sr5fp25-20181030_01(SR5 FP25)) > > IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References > 20181029_400846 (JIT enabled, AOT enabled) > > OpenJ9 - c5c78da > > OMR - 3d5ac33 > > IBM - 8c1bdc2) > > JCL - 20181022_01 based on Oracle jdk8u191-b26 > > NOTICE: If no version information is found above, this could indicate a > corrupted Java installation! > > Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java > > -Dmultiarchive.basepath=3D/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2= 048m > > > > > > *Stacktrace* > > Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST > [Lucene Merge Thread #0] > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.ArrayIndexOutOfBoundsException > > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Con= currentMergeScheduler.java:703) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurre= ntMergeScheduler.java:683) > > Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] > java.lang.ArrayIndexOutOfBoundsException > > at > org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217= ) > > at > org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197= ) > > at > org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078) > > at > org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245) > > at > org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82) > > at > org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWrit= er.java:648) > > at > org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560) > > at > org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60Poin= tsWriter.java:212) > > at > org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173) > > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122) > > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356) > > at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) > > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeS= cheduler.java:624) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurre= ntMergeScheduler.java:661) > > Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST > [Lucene Merge Thread #0] > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.ArrayIndexOutOfBoundsException > > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Con= currentMergeScheduler.java:703) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurre= ntMergeScheduler.java:683) > > Caused by: 2019-05-01T06:10:08.076 CEST [Lucene Merge Thread #0] > java.lang.ArrayIndexOutOfBoundsException > > at > org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217= ) > > at > org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197= ) > > at > org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078) > > at > org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245) > > at > org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82) > > at > org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWrit= er.java:648) > > at > org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560) > > at > org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60Poin= tsWriter.java:212) > > at > org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173) > > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122) > > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356) > > at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) > > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeS= cheduler.java:624) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurre= ntMergeScheduler > > > > > > *BKDWriter.java* > > > > > > > Kind regards > > Torben Riis > > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94 > Torben Riis > Lead Architect > > > Multi Support =C2=B7 Making good business run better > +45 96 600 600 =C2=B7 www.multi-support.com > > > > > > > --0000000000003a55ca0587f738d8--